Hi Dim,
We had similar requirement and we were using AEM version 5.6.1.
These might not be the best approaches but we figured out 2 possible ways of doing it:
1. Overlaying OOTB js file
Steps:
- We created linkDialogConfig node, for e.g. /libs/foundation/components/text/dialog/items/tab1/items/text/rtePlugins/links/linkDialogConfig
- On this node we configured necessary properties (see the screenshot) which we will read in step 3.
- [img]Approach1.jpg[/img]
- Overlay LinkDialog.js which is at location - /libs/cq/ui/widgets/source/widgets/form/rte/plugins/LinkDialog.js
- we added highlighted code in constructor,
- basically we are reading the necessary properties from linkDialogConfig node configured in step 1
- [img]Approach1-JS.jpg[/img]
- Below is how you RTE link dialog should appear:
- [img]Approach1-Dialog.jpg[/img]
2. Recreating Link Dialog Config via nodes
Steps:
- Create a node of type widgetCollection having name dialogItems under /dialog/items/tab1/items/text/rtePlugins/links/linkDialogConfig/
- Under that create a node of type cq:widget having name href and xtype as pathfield. Here you can configure additional properties for your pathfield like label, description, rootpath etc.
- we are basically creating a structure using node what LinkDialog.js provides as default constructor via code (see the snapshot below)
- [img]Approach2.jpg[/img]
With Approach 2 the issue is; you will need to create entire structure same as default one created in LinkDialog.js with all listener config and other widget nodes e.g. target checkbox. Hence we resorted to Approach 1.
Hope it helps 
-Runal