I am not able to enable style system on custom component level . I did below steps still Style tab is not showing on click of policy on custom components under a template. I have added "cq:design_dialog" and entry is done is like below.
Delete the cq:design_dialog node you have created and copy & paste from one of the core components say - /apps/core/wcm/components/list/v2/list/cq:design_dialog and check if it works. (Just to check if it is because of human error)
If it does, retain styletab node alone and remove the rest. (Update "jcr:title" of cq:design_dialog node)