The RTE does not support max length and required properties.
Alternatively, you can use allowBlank= false for required and for maxlength you can add a listener (validator) on click of OK button of dialog checking the length of the characters entered in RTE and validate accordingly.
Add a property "validateValue" in your RTE field and add this function as the value, This is working for me. Change the length in if condition according to your need.
this.markInvalid("Description should not exceed 500 characters");
Note:- RTE in dialog uses the concept of validation from the Form Validation in Granite UI. And, since, Inline-editing does not behave as a form, it can’t use that validation concept.