AEM Compaction getting failure.

[ankitksri]

Hi Team, We are facing one issue AEM compaction, we tried to run the consistency check also but it is showing the good revision number which is already in last. the logs are mention below AEM version is 6.3 and oak version is 1.2.28. Logs are

24.09.2018 04:15:24.256 [main] *INFO*  org.apache.jackrabbit.oak.plugins.segment.file.FileSto

re - TarMK opened: /crx-quickstart/repository/segmentstore (mmap=true)

24.09.2018 04:15:24.259 [main] *INFO*  org.apache.jackrabbit.oak.plugins.segment.file.FileSto

re - TarMK compaction running, strategy=DefaultCompactionStrategy [pauseCompaction=false, clo

neBinaries=false, cleanup=CLEAN_ALL, olderThan=0]

24.09.2018 04:46:30.022 [main] *ERROR*  org.apache.jackrabbit.oak.plugins.segment.SegmentTrac

ker - Segment not found: 862486b5-41bd-43e4-bedb-cb0c8bbc9af6. Creation date delta is 11 ms.

org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment 862486b5-41bd-43e

4-bedb-cb0c8bbc9af6 not found

        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.jav

a:869) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker

.java:179) ~[oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:108)

[oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82) [oak-r

un-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.java:55) [o

ak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream.java:17

1) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.commons.IOUtils.readFully(IOUtils.java:53) [oak-run-1.2.

28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor.getBlobKey(Compactor.java:326)

[oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:291) [o

ak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:251) [o

ak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor.access$400(Compactor.java:51)

[oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAdded(Comp

actor.java:164) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:156) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

  at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState

(SegmentNodeState.java:434) [oak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java:114) [o

ak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:120) [o

ak-run-1.2.28.jar:1.2.28]

        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.compact(FileStore.java:70

8) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.run.Main.compact(Main.java:539) [oak-run-1.2.28.jar:1.2.

28]

        at org.apache.jackrabbit.oak.run.Main.main(Main.java:192) [oak-run-1.2.28.jar:1.2.28]

24.09.2018 04:46:30.509 [main] *INFO*  org.apache.jackrabbit.oak.plugins.segment.file.FileSto

re - TarMK closed: /crx-quickstart/repository/segmentstore

Compaction failure stack trace:

org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment 862486b5-41bd-43e

4-bedb-cb0c8bbc9af6 not found

        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.jav

a:869)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker

.java:179)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:108)

        at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)

        at org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.java:55)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream.java:17

1)

        at org.apache.jackrabbit.oak.commons.IOUtils.readFully(IOUtils.java:53)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor.getBlobKey(Compactor.java:326)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:291)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:251)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor.access$400(Compactor.java:51)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAdded(Comp

actor.java:164)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:156)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

  at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

   at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

        at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState

(SegmentNodeState.java:434)

       at org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java:114)

        at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:120)

        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.compact(FileStore.java:70

8)

        at org.apache.jackrabbit.oak.run.Main.compact(Main.java:539)

        at org.apache.jackrabbit.oak.run.Main.main(Main.java:192)

Compaction failed in 31.16 min (1869s).

[vaibhavs7024655]

ankits48379454 The description provided is not very clear, however taking into consideration that you are using AEM 6.3, the oak version cannot be 1.2.28, it will be something like 1.6.x .

As per the log traces, it looks like the OnRC is running but reporting an inconsistent segment store because I can see SNF errors in the log snippet provided.

Please try steps below and check if it repairs your segment store:

1) Download the oak-run-1.6.x.jar version matching the oak version of your instance.

2) Now run the consistency check,

java -jar oak-run-1.6.x.jar check /path/to/crx-quickstart/repository/segmentstore

which will search backward through the revisions until it finds a consistent on

3) Revert the repository to this revision by editing ./crx-quickstart/repository/segmentstore/journal.log and deleting all lines after the line containing the latest good revision.

4) Remove all ./crx-quickstart/repository/segmentstore/*.bak files.

5) Run checkpoint clean-up to remove orphaned checkpoints: java -jar oak-run-1.6.x.jar checkpoints /path/to/crx-quickstart/repository/segmentstore rm-unreferenced

6) Finally compact the repository: java -jar oak-run-1.6.x.jar compact /path/to/crx-quickstart/repository/segmentstore

[ankitksri]

Sorry vaibhav, it is AEM 6.1 Version.

[vaibhavs7024655]

Thanks for confirming!

In that case you can use the steps mentioned in the doc Fix Inconsistencies in the repository when SegmentNotFound Issue is reported in AEM 6.x  to repair your segment store. You can download the oak-run parallel to your oak-core from here: Maven Repository: org.apache.jackrabbit » oak-run

[ankitksri]

tried already, but all the time it shows the last revision number as a Good revision number only.

[smacdonald2008]

If you have tried this:  

Fix Inconsistencies in the repository when SegmentNotFound Issue is reported in AEM 6.x

then there could be something that would require an investigation to solve. I would open a support ticket so a customer care agent can investigate.