I have seen this before where the minification is not working due to compilation errors. It might be due to the yui compiler for js. So, try using gcc for js
Also, For detailed analysis, follow the steps below:
- Setup a DEBUG logger on "com.adobe.granite.ui.clientlibs"
- Go to <host>:<port>/libs/granite/ui/content/dumplibs.rebuild.html
- Click on invalidate cache
- Now try to load the page where you are loading the custom clientlib and AEM should try to minify it.
- You should see details about why minification is failing in the DEBUG logger setup in the first step.