The only way I can think of is to provide a clientlib per component and let every component include its own clientlib. Which would result in lots of clientlibs. An optimization would be to combine the JS/CSS code of many components into a single clientlib and reference it from all of these components. This would give you still some overhead, but reduce the amount of files loaded.
Although, with HTTP/2 the overhead of loading lots of individual files compared to a smaller number of larger files is non-existent.
Instead , I want to add component level clientlibs using template policies as below by this on specific page load instead of loading complete clientlibs I can only get client lib for the component which are specified on the particular template.