There isn't a specific textbook definition but in my view, it could be a similar definition to labour market policies.
Labour market institutions can be defined as the bodies and organisations established to influence the operation and outcomes in the labour market, including industrial relations policies that regulate the processes of wage regulation and conflict resolution. These include trade unions, employer associations, industrial tribunals and the government.
Put simply, labour market institutions are the institutions that affect the operation of the free market for labour.