Here is an article that talks about how to validate values that an author enters in a dialog. You need to write app logic to perform validation on dialog values- as discussed in the article.
https://helpx.adobe.com/experience-manager/using/creating-touchui-validate.html
Add a listener (validator) on click of OK button of dialog checking the length of the characters entered in RTE and validate accordingly.
Reference article:- http://10codes.blogspot.in/2014/02/adding-custom-listener-on-rich-text.html
//Adding custom listener on rich text component in cq5 to impose maximum number of characters
I hope this helps.
~kautuk
Kautuk Sahni