We have the following requirement that at time of experience fragment creation I have to provide variation title and variation tag but I don't have those options in the dialog in the new fragment creation action, I tried to overlay the dialog but it is still loading the libs.
I am able to override the existing dialog if I change the dialog url variable in /libs/cq/experience-fragments/editor/clientlibs/experiencefragments/js/editor/actions/convert.js to point to apps instead of libs. is the above approach a good practice?
That is not a good approach and certainly not recommended. Your changes will go away on the installation of service packs (if SP has code changes related to this path).
Please review the sling Resource Merger doc  to make sure the overlay/override is implemented correctly.
I am assuming you are using 6.4 or 6.5. This path is marked as Internal (granite:InternalArea) which means it cannot be overlaid, inherited, or used directly. These nodes are meant only for the internal functionality of AEM.