Which AEM version are you on ?
The popover should be closed before moving to the maximized dialog. RTE handles that by itself at least on the latest version.
Calling any event directly would only hide this problem, not really solve the issue. You could see some other issue as a side effect.
Can you try it on a latest SP/CFP installation, otherwise report it as an issue ?