AEM 6.3 (SP2+CFP1+CFP2) Instance is not starting up throwing exceptions

[DeepakJangra]

Hi Experts,

We have AEM 6.3 with SP2+CFP1+CFP2 installed, Few days back some data was migrated using CRX2OAK tool but the system was working perfectly fine. Seems like the server got rebooted without AEM shutdown and when we are starting the AEM server back again, server is not starting properly. I am able to access system/console but crx/de or any other console is not accessible.

Below exception are the exceptions from error.log -

ERROR [FelixStartLevel] c.a.c.d.i.DeserializationFirewallImpl (DeserializationFirewallImpl.java:346) - Unable to load Attach API. Unable to dynamically load deserialization protection agent. This usually happens when running a JRE (instead of a JDK) or a JVM that doesn't provide the Attach API. Please review the AEM Java Deserialization Firewall documentation for how to load the deserialization agent via JVM command line arguments.

ERROR [Apache Sling Repository Startup Thread] o.a.j.o.s.SegmentNotFoundExceptionListener (SegmentNotFoundExceptionListener.java:48) - Segment not found: 559ffe71-f9f2-4771-a6d7-334b866f17ed. SegmentId age=233ms

• org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 559ffe71-f9f2-4771-a6d7-334b866f17ed not found

at org.apache.sling.jcr.base.AbstractSlingRepositoryManager$4.run(AbstractSlingRepositoryManager.java:455)

ERROR [Apache Sling Repository Startup Thread] c.a.g.r.impl.SlingRepositoryManager (AbstractSlingRepositoryManager.java:499) - start: Uncaught Throwable trying to access Repository, calling stopRepository()

• org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 559ffe71-f9f2-4771-a6d7-334b866f17ed not found

at org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:602)

at org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:542)

ERROR [qtp3107287-158] o.a.s.engine.impl.SlingHttpContext (SlingHttpContext.java:136) - handleSecurity: AuthenticationSupport service missing. Cannot authenticate request.

Any pointer will be great help, thanks!

[vaibhavs7024655]

DeepakJanga To start with, you seem to have a segment store inconsistency because I can see SNF errors[1] in the log snippet provided.

Please refer the doc Fix Inconsistencies in the repository when SegmentNotFound Issue is reported in AEM 6.x  and follow the steps in order to fix the same. That should help to bring your instance up once the segment inconsistencies are fixed.

[1]

ERROR [Apache Sling Repository Startup Thread] o.a.j.o.s.SegmentNotFoundExceptionListener (SegmentNotFoundExceptionListener.java:48) - Segment not found: 559ffe71-f9f2-4771-a6d7-334b866f17ed. SegmentId age=233ms

[DeepakJangra]

Yes, we tried this helpx article but it is NOT applicable for AEM 6.3. Oak 1.6.1 doesn’t support “-d1 -p” parameters.

Do you have it worked for AEM 6.3 before, might be we are missing something?

[vaibhavs7024655 · Accepted solution]

DeepakJanga The parameters "d1 -p" is no more a part of the code base of the latest oak run jar.

Below is the set of commands that you can use with AEM 6.3:

1) Download the oak-run-1.6.x.jar version matching the oak version of your instance.

2) Now run the consistency check,

java -jar oak-run-1.6.x.jar check /path/to/crx-quickstart/repository/segmentstore

which will search backwards through the revisions until it finds a consistent on

3) Revert the repository to this revision by editing ./crx-quickstart/repository/segmentstore/journal.log and deleting all lines after the line containing the latest good revision.

4) Remove all ./crx-quickstart/repository/segmentstore/*.bak files.

5) Run checkpoint clean-up to remove orphaned checkpoints: java -jar oak-run-1.6.x.jar checkpoints /path/to/crx-quickstart/repository/segmentstore rm-unreferenced

6) Finally compact the repository: java -Xmx6000m -jar oak-run-1.6.x.jar compact /path/to/crx-quickstart/repository/segmentstore

[DeepakJangra]

Thanks Vaibhav, let me try out these steps, I'll get back to you with the results, thanks!

[DeepakJangra]

Thank you Vaibhav. We followed the AEM 6.3 steps shared by you. Though it took couple of hours as repository size was huge but finally, it fixed the issue.

But still there are some exceptions in logs, I'll check if those are application specific or content. I'll get back to you if any more help is needed. Thanks!