I am working on a solution to integrate AEM with multiple Identity provider based on the type of user. Is this supported out of box or requires customization?
You can configure multiple SAML IDPs (Check the screenshot) and the order in which it would authenticate will be based on the ranking and the same can be checked in /system/console/jaas
but your usecase is different you need different provider based on the user for this you have to write custom authentication handler and OOB is not available.
Blow links will help you: Multiple Authentication handlers
AEM authentication- Custom Authentication handlers and External Login Modules