Incremental indexing in loops

[blumukesh]

Background of this issue :-

4 indexes got corrupt in Production environment. We completed reindex for all corrupt indexes , I could see indexes got restored and removed from corrupt list in JMX (async)

As we know Incremental Indexing will trigger for Delta updates but its in loops due to Excepting during commit .

Error:-

org.apache.jackrabbit.oak.api.CommitFailedException: OakMerge0004: OakMerge0004: Following exceptions occurred during the bulk update operations: [org.apache.jackrabbit.oak.plugins.document.ConflictException:

We have opened Adobe ticket . Actually ticket response is Slow .

Any pointers  for fix this issue ?

AEM Version used - AEM 6.4 
OAK version - 1.8.11

nodeStore - documentMK

[vanegi]

Can you share the complete stack trace of this exception? The document can be fixed either by using the oak-run recovery command or the oak-run console with a script.

[0]: https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run#recovery-mode

[1]: https://github.com/mreutegg/oak-console-scripts/tree/lastRevRecovery/src/main/groovy/utils#last-rev-...

[blumukesh]

Thanks for comment , below is partial traces exception :-

org.apache.jackrabbit.oak.api.CommitFailedException: OakMerge0004: OakMerge0004: Following exceptions occurred during the bulk update operations: [org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 3:/oak:index/AuthorSlingFolder/:data was changed in revision r16ef58e9319-0-a (older than base r16e4e15c74d-0-1,r16e4dec3f1a-0-2,r16e4de42aa6-0-3,r16e4dd66fe9-0-4,r159dc7b983e-0-5,r159de9fa555-0-6,r15a4b823b18-0-7,r172ff3d0eb0-0-8,r17322d46b18-3-9,r17322d2ac07-0-a,r17322d344e1-0-b,r172c8f8c65e-0-c,r172f09aee0f-0-d,r172f0a1e619-0-e,r172f111e1dd-0-f,r172f0c41900-0-10,r172f09ef09d-0-11), which was applied after the base revision r16e4e15c74d-0-1,r16e4dec3f1a-0-2,r16e4de42aa6-0-3,r16e4dd66fe9-0-4,r159dc7b983e-0-5,r159de9fa555-0-6,r15a4b823b18-0-7,r172ff3d0eb0-0-8,r17322d46b18-3-9,r17322d2ac07-0-a,r17322d344e1-0-b,r172c8f8c65e-0-c,r172f09aee0f-0-d,r172f0a1e619-0-e,r172f111e1dd-0-f,r172f0c41900-0-10,r172f09ef09d-0-11, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nw.si does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nu.si does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nv.cfs does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5ny.si does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5ns_Lucene41_0.doc does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nz.cfe does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nx.cfe does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nt.cfs does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 3:/oak:index/cqTagPmaster/:data was changed in revision r17034af4786-0-8 (older than base r16e4e15c74d-0-1,r16e4dec3f1a-0-2,r16e4de42aa6-0-3,r16e4dd66fe9-0-4,r159dc7b983e-0-5,r159de9fa555-0-6,r15a4b823b18-0-7,r172ff3d0eb0-0-8,r17322d46b18-3-9,r17322d2ac07-0-a,r17322d344e1-0-b,r172c8f8c65e-0-c,r172f09aee0f-0-d,r172f0a1e619-0-e,r172f111e1dd-0-f,r172f0c41900-0-10,r172f09ef09d-0-11), which was applied after the base revision r16e4e15c74d-0-1,r16e4dec3f1a-0-2,r16e4de42aa6-0-3,r16e4dd66fe9-0-4,r159dc7b983e-0-5,r159de9fa555-0-6,r15a4b823b18-0-7,r172ff3d0eb0-0-8,r17322d46b18-3-9,r17322d2ac07-0-a,r17322d344e1-0-b,r172c8f8c65e-0-c,r172f09aee0f-0-d,r172f0a1e619-0-e,r172f111e1dd-0-f,r172f0c41900-0-10,r172f09ef09d-0-11, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 3:/oak:index/ntBaseLucene/:data was changed in revision r1703634869e-0-8 (older than base r16e4e15c74d-0-1,r16e4dec3f1a-0-2,r16e4de42aa6-0-3,r16e4dd66fe9-0-4,r159dc7b983e-0-5,r159de9fa555-0-6,r15a4b823b18-0-7,r172ff3d0eb0-0-8,r17322d46b18-3-9,r17322d2ac07-0-a,r17322d344e1-0-b,r172c8f8c65e-0-c,r172f09aee0f-0-d,r172f0a1e619-0-e,r172f111e1dd-0-f,r172f0c41900-0-10,r172f09ef09d-0-11), which was applied after the base revision

[blumukesh]

org.apache.jackrabbit.oak.api.CommitFailedException: OakMerge0004: OakMerge0004: Following exceptions occurred during the bulk update operations: [org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 3:/oak:index/AuthorSlingFolder/:data was changed in revision r16ef58e9319-0-a (older than base r16e4e15c74d-0-1,r16e4dec3f1a-0-2,r16e4de42aa6-0-3,r16e4dd66fe9-0-4,r159dc7b983e-0-5,r159de9fa555-0-6,r15a4b823b18-0-7,r172ff3d0eb0-0-8,r17322d46b18-3-9,r17322d2ac07-0-a,r17322d344e1-0-b,r172c8f8c65e-0-c,r172f09aee0f-0-d,r172f0a1e619-0-e,r172f111e1dd-0-f,r172f0c41900-0-10,r172f09ef09d-0-11), which was applied after the base revision r16e4e15c74d-0-1,r16e4dec3f1a-0-2,r16e4de42aa6-0-3,r16e4dd66fe9-0-4,r159dc7b983e-0-5,r159de9fa555-0-6,r15a4b823b18-0-7,r172ff3d0eb0-0-8,r17322d46b18-3-9,r17322d2ac07-0-a,r17322d344e1-0-b,r172c8f8c65e-0-c,r172f09aee0f-0-d,r172f0a1e619-0-e,r172f111e1dd-0-f,r172f0c41900-0-10,r172f09ef09d-0-11, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nw.si does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nu.si does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nv.cfs does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5ny.si does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5ns_Lucene41_0.doc does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nz.cfe does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nx.cfe does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 4:/oak:index/AuthorSlingFolder/:data/_5nt.cfs does not exist or is already deleted, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 3:/oak:index/cqTagPmaster/:data was changed in revision r17034af4786-0-8 (older than base r16e4e15c74d-0-1,r16e4dec3f1a-0-2,r16e4de42aa6-0-3,r16e4dd66fe9-0-4,r159dc7b983e-0-5,r159de9fa555-0-6,r15a4b823b18-0-7,r172ff3d0eb0-0-8,r17322d46b18-3-9,r17322d2ac07-0-a,r17322d344e1-0-b,r172c8f8c65e-0-c,r172f09aee0f-0-d,r172f0a1e619-0-e,r172f111e1dd-0-f,r172f0c41900-0-10,r172f09ef09d-0-11), which was applied after the base revision r16e4e15c74d-0-1,r16e4dec3f1a-0-2,r16e4de42aa6-0-3,r16e4dd66fe9-0-4,r159dc7b983e-0-5,r159de9fa555-0-6,r15a4b823b18-0-7,r172ff3d0eb0-0-8,r17322d46b18-3-9,r17322d2ac07-0-a,r17322d344e1-0-b,r172c8f8c65e-0-c,r172f09aee0f-0-d,r172f0a1e619-0-e,r172f111e1dd-0-f,r172f0c41900-0-10,r172f09ef09d-0-11, before r17322d46f54-0-9, org.apache.jackrabbit.oak.plugins.document.ConflictException: The node 3:/oak:index/ntBaseLucene/:data was changed in revision r1703634869e-0-8 (older than base r16e4e15c74d-0-1,r16e4dec3f1a-0-2,r16e4de42aa6-0-3,r16e4dd66fe9-0-4,r159dc7b983e-0-5,r159de9fa555-0-6,r15a4b823b18-0-7,r172ff3d0eb0-0-8,r17322d46b18-3-9,r17322d2ac07-0-a,r17322d344e1-0-b,r172c8f8c65e-0-c,r172f09aee0f-0-d,r172f0a1e619-0-e,r172f111e1dd-0-f,r172f0c41900-0-10,r172f09ef09d-0-11), which was applied after the base revision

[blumukesh · Accepted solution]

Root cause of this problem was OAK-8591 (bug in OAK) and the incremental indexing problem was resolved by upgrading the OAK version 1.8.20 .

As per OAK-8591 :-

• Affects Version/s:1.8.0, 1.10.0, 1.12.0, 1.14.0, 1.16.0
• Fix Version/s:1.18.0, 1.10.5, 1.8.17

@kautuk_sahni @vanegi

[kautuk_sahni]

Thank you for sharing this with community. This would help in posterity.