AEM6.2 Instance throwing error after restart, and won't start fully

[kimhat1230]

When trying to start our new AEM 6.2 Instance we are seeing the attached exception.  It appears to be related our link checker settings.  Attaching those also.

We are not able to reproduce this in a local version, it's only happening on our prod server.

This instance is freshly created and we ran the jcr to oak migration tool to get our content in.

[kautuk_sahni]

Hi 

It seems that there is some inconsistency in these segment store.

Try 1:-

Reverting the Oak repository to fix "SegmentNotFoundException"

The check run-mode [2] of oak-run can be used to determine the last known good revision of a segment store, which can subequently be restored manually. This utility can be used to revert a corrupt segment store to its latest good revision change into Oak's working directory (the one containing the crx-quickstart folder) and backup all files in ./crx-quickstart/repository/segmentstore/.

1. Run the consistency check,
java -jar oak-run-*.jar check -d1 --bin=-1 -p
crx-quickstart/repository/segmentstore/

it will search backwards through the revisions until it finds a consistent
one:
14:00:30.783 [main] INFO  o.a.j.o.p.s.f.t.ConsistencyChecker - Found
latest good revision afdb922d-ba53-4a1b-aa1b-1cb044b535cf:234880

2. Revert the repository to this revision by editing
./crx-quickstart/repository/segmentstore/journal.log and deleting all
lines after the line containing the latest good revision.

3. Remove all ./crx-quickstart/repository/segmentstore/*.bak files.
4. Run checkpoint clean-up to remove orphaned checkpoints:
java -jar oak-run-*.jar checkpoints
./crx-quickstart/repository/segmentstore rm-unreferenced

5. Finally compact the repository:
java -jar oak-run-*.jar compact ./crx-quickstart/repository/segmentstore/

[2] 
https://github.com/apache/jackrabbit-oak/blob/trunk/oak-run/README.md#check

Answer Source :- https://mail-archives.apache.org/mod_mbox/jackrabbit-users/201511.mbox/%3CD26B66B4.A6602%25akhoury@a...mod_mbox/jackrabbit-users/201511.mbox/%3CD26B66B4.A6602%25akhoury@adobe.com%3E

[phanirajkumarb7]

Hello,

I have tried the above steps but getting the following error. Please help.

org.apache.jackrabbit.oak.segment.file.InvalidFileStoreVersionException: Using oak-segment-tar, but oak-segment should be used

        at org.apache.jackrabbit.oak.segment.file.AbstractFileStore.checkManifest(AbstractFileStore.java:153)
        at org.apache.jackrabbit.oak.segment.file.ReadOnlyFileStore.<init>(ReadOnlyFileStore.java:79)
        at org.apache.jackrabbit.oak.segment.file.FileStoreBuilder.buildReadOnly(FileStoreBuilder.java:331)
        at org.apache.jackrabbit.oak.segment.file.tooling.ConsistencyChecker.<init>(ConsistencyChecker.java:122)
        at org.apache.jackrabbit.oak.segment.file.tooling.ConsistencyChecker.checkConsistency(ConsistencyChecker.java:83)
        at org.apache.jackrabbit.oak.segment.tool.Check.run(Check.java:159)

[tiagon95643440]

That means that your oak version in aem does not match the oak-run you're using to find the last revision.

In my case I had to use the 1.2.18.

full instructions.

1. Download oak-run jar file from here http://repo1.maven.org/maven2/org/apache/jackrabbit/oak-run/

   * AEM6.1 - Download the oak-run 1.2.x version matching what is installed (check via /system/console/bundles UI)

2. Stop AEM/Oak

3. Run this command:

java -jar oak-run-*.jar check -d1 --bin=-1 -p crx-quickstart/repository/segmentstore/

it which will search backwards through the revisions until it finds a consistent one:

14:00:30.783 [main] INFO  o.a.j.o.p.s.f.t.ConsistencyChecker - Found latest good revision afdb922d-ba53-4a1b-aa1b-1cb044b535cf:234880

4. Revert the repository to this revision by editing ./crx-quickstart/repository/segmentstore/journal.log. Delete all lines after the line containing the latest good revision.

If you would like to find out what date and time you are reverting the repository to then run this command in the segmentstore folder (replace afdb922d-ba53-4a1b-aa1b-1cb044b535cf with the latest good revision in your journal.log):

find . -type f -name "data*.tar" -exec sh -c "tar -tvf {} |grep afdb922d-ba53-4a1b-aa1b-1cb044b535cf" \; -print

The output would show you an approximate date and time of that revision.

5. Remove all ./crx-quickstart/repository/segmentstore/*.bak files.

6. Run checkpoint clean-up to remove orphaned checkpoints:

java -jar oak-run-*.jar checkpoints ./crx-quickstart/repository/segmentstore rm-unreferenced

7. Finally compact the repository:

java -jar oak-run-*.jar compact ./crx-quickstart/repository/segmentstore/

Best regards,

Tiago

[radhikak8817019]

Hi Tiago,

We are also facing issues while compaction and facing issue with segmentstore. We are using oak version 1.2.16.jar .

When we executed below step :

3. Run this command:

java -jar oak-run-*.jar check -d1 --bin=-1 -p crx-quickstart/repository/segmentstore/

The search ended and it gave "no good revision found".

We tried to execute "Data store consistency check" from /crx/explorer/config/check.jsp but it took a little time and return - Traversed 8349436 nodes , 0 error found.

Could anyone please suggest next steps to recover segmentstore and execute offline compaction.

Thanks,

Radhika

[MC_Stuff]

Hi Radhika,

You are lucky that Error coming from link checker & can be other easy fix. Delete the node[1] and save the changes, should fix your issue.   Also at [2] uncheck "Save external link references".   Alternatively   If you are familiar with tar storage format & some advace topic on [3] can get it fixed from backend. It might be challege for me to guide in this open community such sensitive backend operations. Engage with official support team if below does not help.

[1]

/var/linkchecker/*

[2]

http://host:port/system/console/configMgr/com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl

[3]     https://docs.adobe.com/ddc/en/gems/Into-the-tar-pit---a-TarMK-deep-dive.html

Thanks,

[ravi_p_pandey]

While running consistency check my command execution process stuck at 'o.a.j.o.p.s.f.t.ConsistencyChecker - Checking...'. Any suggestions how I can complete the execution of check command. Also, what we need to do if 'no good revision found' result comes?

[Umesh_Chand]

Hi Ravi,

Did you solve the problem?, I am also getting no good revision found. I am taking the repository backup from staging and trying to mount to dev environment to save the content migration.

OAK jar is - oak-run-1.4.6.jar

Thanks,

Umesh