org.apache.jackrabbit.oak.segment.SegmentNotFoundException -in my local

[rajat168]

Hi Team,

      My local AEM(p-4502, aem-sdk v2024.01) instance was getting frequently crashed with below error, could you please let me know, is there any solution for this,

I followed the below steps but no luck.

Solved: SegmentNotFoundException: Segment - Adobe Experience League Community - 416293

SegmentNotFoundException and IllegalArgumentException | Adobe Experience Manager

appreciate your response? Thanks!!!

PFB logs

27.03.2024 16:42:17.511 *ERROR* [oak-lucene-5] org.apache.jackrabbit.oak.segment.SegmentNotFoundExceptionListener Segment not found: 9d6f32c8-1437-46a3-a3ad-c730d600339f. SegmentId age=854029ms
org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 9d6f32c8-1437-46a3-a3ad-c730d600339f not found
	at org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readSegmentUncached(AbstractFileStore.java:295) [org.apache.jackrabbit.oak-segment-tar:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.segment.file.FileStore.lambda$readSegment$10(FileStore.java:508) [org.apache.jackrabbit.oak-segment-tar:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.lambda$getSegment$0(SegmentCache.java:163) [org.apache.jackrabbit.oak-segment-tar:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) [org.apache.jackrabbit.guava:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) [org.apache.jackrabbit.guava:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) [org.apache.jackrabbit.guava:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) [org.apache.jackrabbit.guava:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.get(LocalCache.java:2081) [org.apache.jackrabbit.guava:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.guava.common.cache.LocalCache.get(LocalCache.java:4019) [org.apache.jackrabbit.guava:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933) [org.apache.jackrabbit.guava:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.getSegment(SegmentCache.java:160) [org.apache.jackrabbit.oak-segment-tar:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:508) [org.apache.jackrabbit.oak-segment-tar:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:153) [org.apache.jackrabbit.oak-segment-tar:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:71) [org.apache.jackrabbit.oak-segment-tar:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.segment.SegmentPropertyState.getValue(SegmentPropertyState.java:143) [org.apache.jackrabbit.oak-segment-tar:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.plugins.index.AsyncIndexInfoServiceImpl.getDateAsMillis(AsyncIndexInfoServiceImpl.java:142) [org.apache.jackrabbit.oak-core:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.plugins.index.AsyncIndexInfoServiceImpl.getLastIndexedTo(AsyncIndexInfoServiceImpl.java:135) [org.apache.jackrabbit.oak-core:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.plugins.index.AsyncIndexInfoServiceImpl.getIndexedUptoPerLane(AsyncIndexInfoServiceImpl.java:109) [org.apache.jackrabbit.oak-core:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.plugins.index.AsyncIndexInfoService.hasIndexerUpdatedForAnyLane(AsyncIndexInfoService.java:83) [org.apache.jackrabbit.oak-core:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.diffAndUpdate(IndexTracker.java:153) [org.apache.jackrabbit.oak-lucene:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(IndexTracker.java:135) [org.apache.jackrabbit.oak-lucene:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProvider.contentChanged(LuceneIndexProvider.java:69) [org.apache.jackrabbit.oak-lucene:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:127) [org.apache.jackrabbit.oak-store-spi:1.58.0.T20231123092841-619e1bd]
	at org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:121) [org.apache.jackrabbit.oak-store-spi:1.58.0.T20231123092841-619e1bd]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

[ksh_ingole7 · Accepted solution]

Hi @rajat168 

Please try performing below steps. 

1. Stop AEM.

2. Use the Oak run console and load the childCount groovy script to identify the corrupt nodes in the segment store:

   Load the oak-run console shell:

   java -jar oak-run-*.jar console crx-quickstart/repository/segmentstore

   Run the two commands below in the shell to load the script and run it:

   :load https://gist.githubusercontent.com/stillalex/e7067bcb86c89bef66c8/raw/d7a5a9b839c3bb0ae5840252022f87...

   countNodes(session.workingNode)

   This results in the following output indicating the path to the corrupted node(s):

   21:21:42.029 [main] ERROR o.a.j.o.p.segment.SegmentTracker - Segment not found: 63ae05a4-b506-445c-baa2-cfa1b13b6e2f. Creation date delta is 3 ms.
   warning unable to read node /content/dam/test.txt/jcr:content/renditions/original/jcr:content

   In some cases the issue is linked to binary properties and the childCount groovy script is unable to locate any corrupted nodes.  In these cases you can use the following command instead which will read the first 1024 bytes for every binary encountered during the traversal (Please note this command will be slower and should only be used when the one above doesn't return the expected results):

   countNodes(session.workingNode,true)

3. Remove all the identified corrupted nodes listed in the output of the last command using rmNodes.groovy

   Load the oak-run console shell:

   java -jar oak-run-*.jar console crx-quickstart/repository/segmentstore

   Load the groovy script:

   :load https://gist.githubusercontent.com/stillalex/43c49af065e3dd1fd5bf/raw/9e726a59f75b46e7b474f7ac763b08...

   Run the rmNode command to remove the corrupt node, replace /path/to/corrupt/node with the path to the corrupt node you need to remove.

   rmNode(session, "/path/to/corrupt/node")

   Where the corrupt node path is the path obtained in step 2, for example: "/content/dam/test.txt/jcr:content/renditions/original/jcr:content/"

    

   Note: 

   When using oak-run.jar version 1.6.13 and up, set --read-write JVM parameter if you run into an error such as:

   /> rmNode(session,"/path/to/corrupt/node")
   Removing node /path/to/corrupt/node
   ERROR java.lang.UnsupportedOperationException:
   Cannot write to read-only store
           at org.apache.jackrabbit.oak.segment.SegmentWriterBuilder$1.execute (SegmentWriterBuilder.java:171)
           at org.apache.jackrabbit.oak.segment.SegmentWriter.writeNode (SegmentWriter.java:318)
           at org.apache.jackrabbit.oak.segment.SegmentNodeBuilder.getNodeState (SegmentNodeBuilder.java:111)
           at org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.<init> (SegmentNodeStore.java:581)
           at org.apache.jackrabbit.oak.segment.SegmentNodeStore.merge (SegmentNodeStore.java:333)
           at org.apache.jackrabbit.oak.spi.state.NodeStore$merge.call (Unknown Source)
           at groovysh_evaluate.rmNode (groovysh_evaluate:11)

4. Repeat step 3 for all nodes found in step 2.

   This above rmNode command should return true for the corrupt path, which means it deleted it. Make sure these found three corrupt paths get deleted by re-running the rmNode command on those paths. Next run it should return false.

   If still you see that the same paths are there in repository, then use the patched version of oak-run jar i.e. oak-run-1.2.18-NPR-17596

   What does patched version of Oak run Jar Do?

   This version of jar skips unreadable binaries on compaction replacing them with 0-byte binaries and logging the exception and the path to syserr. The thus compacted repository should then pass oak-run check, the node count script and you should also be able to compact it again using a non- patched oak-run.

5. Perform a checkpoint cleanup by listing checkpoints using below. If there are more than one checkpoint, then clean them up:

   nohup java -Xmx4096m -jar oak-run-1.2.18.jar checkpoints /app/AEM6/author/crx-quickstart/repository/segmentstore rm-all>>nohup.out &

6. Run an offline compaction.  If you do not know how to run offline compaction then see here.

7. Start the server wait for indexing completion.

   If this does not work you can also try deleting the crx-quickstart folder from your local and try re deploying your code repo. Let me know if the issue persists. 

   Thanks.