Interestingly the issue was with Authorization Server URL. In the documentation its mentioned that we need to give the complete URL with API Key but it works when we give only the domain.
Ex:
Correct config - https://ims-na1.adobelogin.com
Wrong config - https://ims-na1.adobelogin.com/c/c95###042A...