you need create custom link plugin by extending /libs/clientlibs/granite/coralui2/optional/rte/js/core/commands/Link.js plug-in
and to create options pop-up you need to find the correct template and implemnt same for your new custom plugn-in
/libs/clientlibs/granite/coralui2/optional/rte/js/components/cui-rte.templates.js
ex:
locate this code this["CUI"]["rte"]["Templates"]["dlg-link"] in template js and you can see the options.
I believe you are using AEM 6.3 or above, for lower versions the libs path is different.
Thanks!