How do you properly allow anonymous view permissions of a clientlib located inside the components directory?

Inside /apps/myproject/components/content/mycomponent I have a mycomponent.html file and a ClientLibraryFolder. In my html file I have:

<sly data-sly-use.myComponent="myComponent.js" data-sly-use.clientLib="${'/libs/granite/sightly/templates/clientlib.html'}" id="${myComponent.id}" /> <sly data-sly-call="${clientLib.all @ categories='myproject.components.content.mycomponent'}" />

Which does transpile to importing the script from myproject/components/content/mycomponent

The issue that I'm having though is on the author and dispatcher servers. If I'm logged in as an administrator, my clientlibraries render, but if I'm not an administrator, then I don't have sufficient permissions to access that directory. I could embed these libraries to the base clientlib, but then I'd be loading every library to the client, regardless if the component that uses it is present on the page or not.