Hello @Sudarshan1992
Option-1: Ideally for such scenarios, one should separate out these XF in seperate folders. And then use rootPath config, to allow user to pick only from relevant folder.
Option-2:
Value of each column in XF picker is returned from a query similar to the one below:

OOTB it is not possible to filter XF based on templates. However, there is a workaround.
There is propertyFilter. If the value of this property is false on a page, it won't be returned.
You could create your own extended component. Override "propertyFilter" as some other property. And use that property in pages to show/hide them in XF picker. Default values of this property can be copied to pages via templates.
Location of propertyFilter "/libs/core/wcm/components/experiencefragment/v2/experiencefragment/cq:dialog/content/items/tabs/items/properties/items/columns/items/column/items/fragmentVariationPath"
Aanchal Sikka