AEM Compaction getting failure.

Forum|Forum|7 years ago
September 24, 2018
2 replies
4033 views

Hi Team, We are facing one issue AEM compaction, we tried to run the consistency check also but it is showing the good revision number which is already in last. the logs are mention below AEM version is 6.3 and oak version is 1.2.28. Logs are

24.09.2018 04:15:24.256 [main] *INFO* org.apache.jackrabbit.oak.plugins.segment.file.FileSto

re - TarMK opened: /crx-quickstart/repository/segmentstore (mmap=true)

24.09.2018 04:15:24.259 [main] *INFO* org.apache.jackrabbit.oak.plugins.segment.file.FileSto

re - TarMK compaction running, strategy=DefaultCompactionStrategy [pauseCompaction=false, clo

neBinaries=false, cleanup=CLEAN_ALL, olderThan=0]

24.09.2018 04:46:30.022 [main] *ERROR* org.apache.jackrabbit.oak.plugins.segment.SegmentTrac

ker - Segment not found: 862486b5-41bd-43e4-bedb-cb0c8bbc9af6. Creation date delta is 11 ms.

org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment 862486b5-41bd-43e

4-bedb-cb0c8bbc9af6 not found

at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.jav

a:869) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker

.java:179) ~[oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:108)

[oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82) [oak-r

un-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.java:55) [o

ak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream.java:17

1) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.commons.IOUtils.readFully(IOUtils.java:53) [oak-run-1.2.

28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor.getBlobKey(Compactor.java:326)

[oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:291) [o

ak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:251) [o

ak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor.access$400(Compactor.java:51)

[oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAdded(Comp

actor.java:164) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:156) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState

(SegmentNodeState.java:434) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java:114) [o

ak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:120) [o

ak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.compact(FileStore.java:70

8) [oak-run-1.2.28.jar:1.2.28]

at org.apache.jackrabbit.oak.run.Main.compact(Main.java:539) [oak-run-1.2.28.jar:1.2.

28]

at org.apache.jackrabbit.oak.run.Main.main(Main.java:192) [oak-run-1.2.28.jar:1.2.28]

24.09.2018 04:46:30.509 [main] *INFO* org.apache.jackrabbit.oak.plugins.segment.file.FileSto

re - TarMK closed: /crx-quickstart/repository/segmentstore

Compaction failure stack trace:

org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment 862486b5-41bd-43e

4-bedb-cb0c8bbc9af6 not found

at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.jav

a:869)

at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker

.java:179)

at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:108)

at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)

at org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.java:55)

at org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream.java:17

1)

at org.apache.jackrabbit.oak.commons.IOUtils.readFully(IOUtils.java:53)

at org.apache.jackrabbit.oak.plugins.segment.Compactor.getBlobKey(Compactor.java:326)

at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:291)

at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:251)

at org.apache.jackrabbit.oak.plugins.segment.Compactor.access$400(Compactor.java:51)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAdded(Comp

actor.java:164)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:156)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Com

pactor.java:198)

at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(E

mptyNodeState.java:161)

at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState

(SegmentNodeState.java:434)

at org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java:114)

at org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:120)

at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.compact(FileStore.java:70

8)

at org.apache.jackrabbit.oak.run.Main.compact(Main.java:539)

at org.apache.jackrabbit.oak.run.Main.main(Main.java:192)

Compaction failed in 31.16 min (1869s).

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

vaibhavs7024655

ankits48379454 The description provided is not very clear, however taking into consideration that you are using AEM 6.3, the oak version cannot be 1.2.28, it will be something like 1.6.x .

As per the log traces, it looks like the OnRC is running but reporting an inconsistent segment store because I can see SNF errors in the log snippet provided.

Please try steps below and check if it repairs your segment store:

1) Download the oak-run-1.6.x.jar version matching the oak version of your instance.

2) Now run the consistency check,

java -jar oak-run-1.6.x.jar check /path/to/crx-quickstart/repository/segmentstore

which will search backward through the revisions until it finds a consistent on

3) Revert the repository to this revision by editing ./crx-quickstart/repository/segmentstore/journal.log and deleting all lines after the line containing the latest good revision.

4) Remove all ./crx-quickstart/repository/segmentstore/*.bak files.

5) Run checkpoint clean-up to remove orphaned checkpoints: java -jar oak-run-1.6.x.jar checkpoints /path/to/crx-quickstart/repository/segmentstore rm-unreferenced

6) Finally compact the repository: java -jar oak-run-1.6.x.jar compact /path/to/crx-quickstart/repository/segmentstore

ankitksriAuthor

Sorry vaibhav, it is AEM 6.1 Version.

vaibhavs7024655

Thanks for confirming!

In that case you can use the steps mentioned in the doc Fix Inconsistencies in the repository when SegmentNotFound Issue is reported in AEM 6.x to repair your segment store. You can download the oak-run parallel to your oak-core from here: Maven Repository: org.apache.jackrabbit » oak-run

ankitksriAuthor

tried already, but all the time it shows the last revision number as a Good revision number only.

smacdonald2008

If you have tried this:

Fix Inconsistencies in the repository when SegmentNotFound Issue is reported in AEM 6.x

then there could be something that would require an investigation to solve. I would open a support ticket so a customer care agent can investigate.

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded