segment.SegmentNotFoundException: Error at beginning of oak-run check cmd

[manu-gupta]

Hi folks

we are getting segmentnotfound exception at every beginning of oakrun jar check cmd and not getting any good revision of aem repo.

And not sure from which line we need to clean journal.log file.

Please help

AEM 6.5.8 Linux JAVA 8

bash-4.2$ java -Xmx6000m -jar /opt/aem/author/backupmodule/oak-run-1.38.0.jar check --bin=-1 /opt/aem/author/crx-quickstart/repository/segmentstore
Apache Jackrabbit Oak 1.38.0
07:27:59.349 [main] WARN  o.a.j.o.s.file.tar.SegmentTarReader - Unable to load index of file data00183a.tar: Unrecognized magic number
07:27:59.352 [main] WARN  o.a.j.oak.segment.file.tar.TarReader - Could not find a valid tar index in data00183a.tar, recovering read-only
org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment c180a07a-60e6-4dc1-ab02-845f24da9b46 not found
        at org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readSegmentUncached(AbstractFileStore.java:288)
        at org.apache.jackrabbit.oak.segment.file.ReadOnlyFileStore$1.call(ReadOnlyFileStore.java:125)
        at org.apache.jackrabbit.oak.segment.file.ReadOnlyFileStore$1.call(ReadOnlyFileStore.java:122)
        at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.lambda$getSegment$0(SegmentCache.java:163)
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4724)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3932)
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4721)
        at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.getSegment(SegmentCache.java:160)
        at org.apache.jackrabbit.oak.segment.file.ReadOnlyFileStore.readSegment(ReadOnlyFileStore.java:122)
        at org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:153)
        at org.apache.jackrabbit.oak.segment.CachingSegmentReader$1.apply(CachingSegmentReader.java:105)
        at org.apache.jackrabbit.oak.segment.CachingSegmentReader$1.apply(CachingSegmentReader.java:101)
        at org.apache.jackrabbit.oak.segment.ReaderCache.get(ReaderCache.java:117)
        at org.apache.jackrabbit.oak.segment.CachingSegmentReader.readString(CachingSegmentReader.java:101)
        at org.apache.jackrabbit.oak.segment.MapRecord.getKeys(MapRecord.java:348)
        at org.apache.jackrabbit.oak.segment.SegmentNodeState.getChildNodeNames(SegmentNodeState.java:472)
        at org.apache.jackrabbit.oak.segment.SegmentNodeStore.checkpoints(SegmentNodeStore.java:298)
        at org.apache.jackrabbit.oak.segment.tool.Check.run(Check.java:404)
        at org.apache.jackrabbit.oak.segment.tool.Check.run(Check.java:380)
        at org.apache.jackrabbit.oak.run.CheckCommand.execute(CheckCommand.java:97)
        at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)

[Vikram_Gaur_]

Hi Manu,

Welcome to Adobe Community !!

Have you tried removing corrupted nodes manually?

Please follow Step "B" from this document to remove corrupted nodes manually : https://experienceleague.adobe.com/docs/experience-cloud-kcs/kbarticles/KA-16457.html

Thanks,

Vikram Gaur

[manu-gupta]

server is within firewall and cannot access internet, to mentioned https://gist.githubusercontent.com/ url. We cannot allow internet access as it is non prod and within internal nw.

Is there any other way to fix the segment nodes

[Raja-kp]

@manu-gupta Based on the log you have provided, you are using the oak run version of 1.38.0.

1.Make sure online compaction is disabled to avoid the nodes getting corrupted or removed by data store garbage collection.

2.Above AEM6.3 version, you must use oak-run-1.6.x.jar. Download the oak-run-1.6.x.jar version from the link :https://repo1.maven.org/maven2/org/apache/jackrabbit/oak-run/

3. After updating the oak run jar - perform the offline compaction again and let us know the result.

Regards,

Raja