We are trying to use OAuth 2 Identity Provider with self-signed SSL certificate and it appears that this approach are not supported by the system.
OAuth authentication endpoint is not reachable from mobile application - Chrome Debugger shows "Canceled" status for HTTP call to the endpoint.
Usage of "Desktop Web Viewer" gives possibility to add security exception for the host but on the step of passing "code" parameter to Experience Manager Mobile Redirection Endpoint  the flow still breaks with "Exception in the Entitlement Service while processing the result from the identity provider" message passed as parameter in the redirect call to the Web Viewer 
There I would like to note that Entitlement is disabled for the project.
I am wondering, is the error above caused by the untrusted connection? Is that possible to use custom Identity Provider with self-signed SSL certificate?
Thank you for considering my question.
 Web Viewer