With the default implementation of the styles plugin in RTE, I am able to add multiple classes to the selected text. The recent selection gets nested to the previously selected option. Is there any configuration that can make styles plugin to not chose more than one style for any selected text?
There is no such configuration exists OOTB which will allow to prevent selection of multiple styles from style dropdown. If a style is sleected, the corresponding style will be having a checkmark in the authoring view so you should be able to identify if anything is selected.
If you really want to block it, then we might need to add a custom plugin which will be loaded while authoring and will prevent the functionality.