Hi,
We added custom styles tab in RTE like below and it is working as expected
<styles jcr:primaryType="nt:unstructured" features="*">
<styles jcr:primaryType="nt:unstructured">
<size1 jcr:primaryType="nt:unstructured" cssName="txt--1" text="Size 1"/>
<size2 jcr:primaryType="nt:unstructured" cssName="txt--2" text="Size 2"/>
</styles>
</styles>
Try to change from cq:WidgetCollection to nt:unstructured