We've been troubleshooting this with Adobe support for a few weeks, and were finally able to perform the AEM/Sign integration successfully, but we had to jump through a few hoops.
For anyone encountering the 'failed to get refresh token' error, first try disabling Minification in Adobe Granite HTM...