AEM instance corrupted on startup

[abhishsh]

Hello everyone, 

I am using AEM 6.5. I have a really weird problem on startup of the instance. Every time, when I shut off my instance and start it again, my AEM instance wouldn't work properly. The image on the splash screen as well as the icons in the touch UI wouldn't load. I had to delete the crx-quickstart to reinstall the instance. Then it will be fine. But next time, when I shut off the instance and start it again, same problem happens. I can't really delete the crx-quickstart every time since I have installed some packages and I had to redo them again. I was wondering if anyone know how to resolve this issue.

Thanks in advance for your help. 

Lucy 

[abhishsh]

Hi @lwang, Do you see if there are any bundle in resolved or installed state at system console, after restart of instance and what's the error or exception you see in error log ?

[#3 · Accepted solution]

Hey @lwang,

Are you migrating from old AEM version to new one? Also, please let me know if I've understood incorrectly here:
--> when you have a fresh instance(just after you unpack the jar) things work quite well.

--> as soon as you install some packages and the the issue starts to happen(after you shut down the instance and restart)

Is there is a possibility that you have custom oak:index definitions in your package? if that's the case please remove those paths(custom index definitions) from filter(while creating the package itself) and then try installing it.(check if you have queryPaths and includedPaths in your custom index definition)

Also, if you leave your instance for a long while(2hours or so), does that resolve the issue on its own? please try that.

Thanks,

Bilal.

[#4]

Error[1] means something wrong with your datastore. Either the user running AEM does not have permissions to the crx-quickstart/repository/datastore folder or something else going on with the datastore.

[1] error: java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId [3e15c88da9b46267141b60826c0734007c3e5a5f804c953edb0280eda680a9be#174188]

[#5]

What error do you see in the log files? I would recommend checking log files for any errors.

[lwang]

@Hi @lwang 

Can you confirm if you are using AEM 6.5 with JDK 8 or JDK 11.

If you are using with JDK 11 you will need to ensure to add the below lines in the sling.properties file on your instance for sling.bootdelegation.sun property.

File before update: sling.bootdelegation.sun=sun.*,com.sun.*

After update: sling.bootdelegation.sun=sun.*,com.sun.*,jdk.internal.reflect,jdk.internal.reflect.*

 Also you can include the below snippet to your JVM parameters in the start.bat(Windows) or start.sh(Mac OS/Linux) file based on the OS.

-XX:+UseParallelGC --add-opens=java.desktop/com.sun.imageio.plugins.jpeg=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.jrt=ALL-UNNAMED --add-opens=java.naming/javax.naming.spi=ALL-UNNAMED --add-opens=java.xml/com.sun.org.apache.xerces.internal.dom=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED -Dnashorn.args=--no-deprecation-warning

Also allow enough heap memory something like 2048 MB or 4096 MB.

It will become something like:

CQ_JVM_OPTS='-server -Xmx4096m -XX:MaxPermSize=256M -Djava.awt.headless=true -XX:+UseParallelGC --add-opens=java.desktop/com.sun.imageio.plugins.jpeg=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.jrt=ALL-UNNAMED --add-opens=java.naming/javax.naming.spi=ALL-UNNAMED --add-opens=java.xml/com.sun.org.apache.xerces.internal.dom=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED -Dnashorn.args=--no-deprecation-warning'

Please refer the Java 11 consideration section in the below article:

https://experienceleague.adobe.com/docs/experience-manager-65/deploying/deploying/custom-standalone-...

Hope this will help you!

Thanks!

[lwang]

Hi @asutosh_jena, I am using JDK 8 - should I upgrade?

[asutosh_jena]

@lwang No. You do not have to upgrade it.
When you restart the instance can you confirm what is the error you are getting in the error.log file?
Also make sure all the bundles are in active state.

Before that to narrow down the issue, we will need to ensure that the same issue does not exist in the plain vanilla instance which means you will need to install a fresh copy of AEM 6.5 and check with the we-retail package if all the pages are loading fine. Now if everything works well you need to shutdown the instance and restart in the same way how you were performing earlier. If you instance is back and running then we will be sure the issue is with your code/content package that you are deploying.

Also make sure to use 6.5.8.0 service pack as it is the latest one, else you can go with AEM 6.5.5.0 minimum.

Can you please perform the above and confirm?

Thanks!

[lwang]

Hi @abhishsh - I don't see any errors in the error.log

[lwang]

Performing data store consistency check... error: java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId [3e15c88da9b46267141b60826c0734007c3e5a5f804c953edb0280eda680a9be#174188]

[asutosh_jena]

@lwang Did you try the above steps and able to reproduce the issue?