Hi,
For styles to apply in the rich text editor, we have to add the styles in the clientlibs which loads for cq:dialog. You can add the styles in a stylesheet and add clientlib to "extraClientlibs" property of the dialog for styles to apply in cq:dialog