This works completely fine with AEM 6.4.4 but when I install this along with the build in AEM 6.5.5, seems like the engine gets stuck somehow and not being able to recover at all(even after AEM restart, or system restart).
I actually checked the logs, and I couldn't find anything about it(not even 54% completed or something like that messages that says that re-indexing is in progress). As soon as I comment the oak_index .content.xml and the line in filter.xml, everything works great.
Why is this happening? is this way not recommended? or am I doing something wrong?
Custom index definition you have shared has few properties that are incorrect as highlighted below.
compatVersion should be of type "Long"
excludePropertyNames is not applicable for lucene index definition (on the index node - myCustomLuceneIndex)
dexPath is not applicable. Instead property named "indexPath" should be used for mentioning the index path
includePropertyTypes is not applicable on the index node. Instead on the specific indexRules (Eg : In this case, it is on nt:base node)
Also this definition will index all nodes(nt:base) -> all props. (nt:base is the super node type for all existing node types and hence the entire nodes in repo)
Based on the repo volume it is taking quite long to index all. That's why you see message related to the progress in percent and it is still in progress.
Per 6.5.0, /oak:index/lucene this covers the indexing for nt:base node type (but is marked as deprecated with help of property). Check if this is still available in 6.5.5 or any new equivalent is available to cover nt:base node type OOB.
Remove custom definition from your code base. (Per the definition you have shared, except for the excludedPaths property which has project specific paths, everything looks similar to OOB definition)
Analyze the existing OOB definition on 6.5.5 related to nt:base and then decide to create custom only if it is really needed.