AEM 6.5 HTTP 503 Error AuthenticationSupport service...

[aemuser001]

Hello Members,

I have a AEM 6.5 OSGi framework on-premise Author and Publisher instances running in Windows OS. It was working fine for a long time. We took a snapshot of the Windows server and we used the Window's Services to restarted the Author and Publisher instances. The Publisher instance is running fine, but the Author instance is not. The error.log in Author instance shows the following error:

19.03.2021 15:27:41.963 *ERROR* [qtp1031644809-61] org.apache.sling.engine.impl.SlingHttpContext handleSecurity: AuthenticationSupport service missing. Cannot authenticate request. 19.03.2021 15:27:41.964 *ERROR* [qtp1031644809-61] org.apache.sling.engine.impl.SlingHttpContext handleSecurity: Possible reason is missing Repository service. Check AuthenticationSupport dependencies.

The URL to server:port/aem/start.html displays the following error message in the browser:

HTTP ERROR 503 Problem accessing /aem/start.html. Reason: AuthenticationSupport service missing. Cannot authenticate request.

The link to crx  -  http://server:port/crx/de/index.jsp is displaying the following error:

HTTP ERROR 503 Problem accessing /crx/de/index.jsp. Reason: Service Unavailable

http://server:port/crx/packmgr/index.jsp gives the following error:

Problem accessing /crx/packmgr/index.jsp. Reason:

AuthenticationSupport service missing. Cannot authenticate request.

I do see that the Author's crx-quickstart/repository only has cache, diff-cache and index folders. It does not have the segmentstore, datastore and blobids folders, but these 3 folders exist in the Publisher instance.

I did try to rename the index folder and restarted the Author instance, but it did not create a new index folder. The user has full permissions to the folder.

Our team tried to revert back to previous state by using the snapshot and server backup from last month, but the issue is the same.

I can access the /system/console/memoryusage and it uses only 10% of memory.

The below links work in Author and all the bundles are active. The login is like a pop-up near the browser tab, and it is not the usual login page that is used to access these links.

 http://server:port/system/console/slinglog

 http://server:port/system/console/bundles

1. Is there any suggestion on what to try next to recover from this issue.

2. Are the segmentstore, datastore and blobids folders needed in Author's crx-quickstart/repository? If yes, is it possible to upload specific packages or drop it in a specific folder structure

3. Are there any specific steps to be taken to backup crx repository when server is backed up, or a snapshot is taken?

Thanks in advance for the response,

Leena

[Asutosh_Jena_]

Hi @aemuser001 

Please follow the below steps:

1. Shutdown the instance.
2. Delete all index from the index folder.(\crx-quickstart\repository\index)
3. Restart the AEM instance.

This should resolve the issue.

Thanks!

[aemuser001]

Hi Asutosh, I tried your suggestion to stop the instance, removed the folders within index folder and start it again. Now the following error is displayed when accessing server:port/aem/start.html :

HTTP ERROR 404

Problem accessing /aem/start.html. Reason:

Not Found

Any suggestion?

Thanks,

Leena

[Asutosh_Jena_]

Hi @aemuser001 

Which service pack is currently installed on AEM 6.5? Also can you mention if you are using JDK 8 or 11?

Thanks!

[aemuser001]

Using JDK 1.8 and Adobe Experience Manager 6.5.4.0

[aemuser001]

I also need to add that after refreshing the browser I am getting the original error again:

AEM 6.5 HTTP 503 Error AuthenticationSupport service missing. Cannot authenticate request

[Asutosh_Jena_]

Hi @aemuser001 

Can you upgrade it to latest service pack i.e. 6.5.8 and try it again. Also you must be getting some error in the log when you get 503 error on screen. What is that error?

Thanks!

[aemuser001]

Hi Asutosh,

Thank you for your response and suggestions. The Author instance worked fine in using JDK 1.8 and Adobe Experience Manager 6.5.4.0 for nearly a year. Due to business considerations I will not be able to upgrade to service pack i.e. 6.5.8 now. 

Currently the the Author's crx-quickstart/repository only has cache, diff-cache and index folders. It does not have the segmentstore, datastore and blobids folders, but these 3 folders exist in the Publisher instance. Can you please confirm if these 3 subfolders are needed in the Author instance too?

The following error is seen in the startup.log when Author instance was restarted:

2021-04-05 08:40:13 Commons Daemon procrun stdout initialized

2021-04-05 08:40:13 Commons Daemon procrun stderr initialized
05.04.2021 08:40:13.426 *INFO * [main] Setting sling.home=D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\\..\.. (command line)
05.04.2021 08:40:14.473 *INFO * [main] No Apache Sling running at /127.0.0.1:63000
05.04.2021 08:40:14.473 *INFO * [main] Starting Apache Sling in D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\\..\..
05.04.2021 08:40:14.473 *INFO * [main] Sling Extension Lib Home : D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext
05.04.2021 08:40:14.504 *INFO * [main] Checking launcher JAR in folder D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad
05.04.2021 08:40:14.535 *INFO * [main] Existing launcher is up to date, using it: 6.0.2.2_6_36 (org.apache.sling.launchpad.base.jar)
05.04.2021 08:40:14.551 *INFO * [main] Loading launcher class org.apache.sling.launchpad.base.app.MainDelegate from org.apache.sling.launchpad.base.jar
05.04.2021 08:40:14.551 *INFO * [main] Sling Extension jars found = [ D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\bcmail-1.51.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\bcpkix-1.51.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\bcprov-1.51.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\certjFIPS-2.1.2.0.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\cryptojce-6.0.0.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\cryptojcommon-6.0.0.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\jcmFIPS-6.0.0.jar, ]
05.04.2021 08:40:14.660 *INFO * [main] Setting sling.properties=D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\\..\..\conf\sling.properties
05.04.2021 08:40:14.660 *INFO * [main] Setting sling.home=D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\\..\..
05.04.2021 08:40:14.660 *INFO * [main] Setting sling.launchpad=D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad
05.04.2021 08:40:14.660 *INFO * [main] Setting org.osgi.service.http.port=4502
05.04.2021 08:40:14.660 *INFO * [main] Starting launcher ...
05.04.2021 08:40:14.660 *INFO * [Apache Sling Control Listener@/127.0.0.1:63000] Apache Sling Control Listener started
05.04.2021 08:40:14.676 *INFO * [main] HTTP server port: 4502
RAWProcessor succesfully installed
05.04.2021 08:40:54.076 *INFO * [main] Startup completed
java.lang.NullPointerException
at com.day.crx.explorer.impl.j2ee.JCRExplorerServlet.getDefaultSession(JCRExplorerServlet.java:283)
at com.day.crx.explorer.impl.j2ee.CRXContext.getSession(CRXContext.java:245)
at com.day.crx.explorer.impl.j2ee.CRXHttpServletRequest.getJcrSession(CRXHttpServletRequest.java:39)
at com.day.crx.explorer.impl.j2ee.LoginServlet.redirectToLogin(LoginServlet.java:332)
at com.day.crx.explorer.impl.jsp.index_jsp._jspService(index_jsp.java:87)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at com.day.crx.explorer.impl.MainServlet.doService(MainServlet.java:171)
at com.day.crx.explorer.impl.MainServlet.service(MainServlet.java:89)
at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:123)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:86)
at com.adobe.granite.license.impl.LicenseCheckFilter.doFilter(LicenseCheckFilter.java:312)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81)
at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:131)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81)
at org.apache.felix.http.base.internal.dispatch.Dispatcher$1.doFilter(Dispatcher.java:146)
at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager$2.doFilter(WhiteboardManager.java:1002)
at org.apache.sling.security.impl.ReferrerFilter.doFilter(ReferrerFilter.java:326)
at org.apache.felix.http.base.internal.handler.PreprocessorHandler.handle(PreprocessorHandler.java:136)
at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager$2.doFilter(WhiteboardManager.java:1008)
at org.apache.felix.http.sslfilter.internal.SslFilter.doFilter(SslFilter.java:97)
at org.apache.felix.http.base.internal.handler.PreprocessorHandler.handle(PreprocessorHandler.java:136)
at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager$2.doFilter(WhiteboardManager.java:1008)
at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager.invokePreprocessors(WhiteboardManager.java:1012)
at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:91)
at org.apache.felix.http.base.internal.dispatch.DispatcherServlet.service(DispatcherServlet.java:49)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:544)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1581)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1307)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:482)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1549)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1204)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:494)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:374)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:268)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
at java.lang.Thread.run(Thread.java:748)

Thanks,

Leena

[Asutosh_Jena_]

Hi @aemuser001 

Yes, you will need the below folders to be available within the repository folder.

1. blobids
2. datastore
3. segmentstore
4. index

All the repository related data will be stored under datastore folder and segment files are stored under the segment folder.

If you have any backup running on the system, you might be able to restore those. If not, you will need to rebuild the instance.

You can get more information about the folders here:

https://experienceleague.adobe.com/docs/experience-manager-65/deploying/deploying/data-store-config....

Thanks!

Asutosh

[aemuser001]

Hi Asutosh,

Thanks for confirming. We will try to restore from our last December backup of the server. Hopefully the 3 subfolders are present in that backup.

1. What could have caused the 3 subfolders to be missing in the crx-quickstart/respository folder?

2. To avoid this issue in the future, do I need to manually copy these folders to a different location before taking a snapshot or backup of the server? The reason I ask this is that even after restoring the server from few weeks ago (when we know the Author was up and running at that time), and now when we check the crx-quickstart/respository folder the 3 subfolders are missing even in the recently restored server. I am assuming the 3 folders were missing for sometime, and the reason the Author instance was still running was because the application was using the resources in the JVM. Once the Author instance was stopped and restarted, it was then that Author instance was looking for these subfolders and it failed to start.

Is this assumption right? Is there any other take on it?

Thanks,

Leena

[Asutosh_Jena_]

Hi Leena,

These folders contains huge amount of data and it will never be deleted automatically. I doubt if something happened on this instance which deleted the file/it might be accidentally deleted by someone while performing any action on the file system.

If you want to take the offline backup of only your AEM instance, then you can schedule some time (depending upon the usage schedule) and schedule the backup process as it will take less than a few minutes, but the application needs to be stopped.

To create a offline backup perform these steps:

1. Stop the application
2. Make a SNAPSHOT backup
3. Start the application

Read more details here:

https://experienceleague.adobe.com/docs/experience-manager-65/administering/operations/backup-and-re...

You can take the SNAPSHOT backup of the whole instance i.e. if you are using this on an EC2 instance, you can take the complete volume backup where AEM is installed. Let's say you have /data as the mount point where you have the AEM instance available then you can take a weekly backup of the instance so if anything goes wrong at any point of time you still will have 1 week prior data which can be restored. This can be automated using AWS SNAPSHOT backup and there will not be any downtime as this can be taken when the system is running.

Please refer the link below for more info:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSSnapshots.html

Thanks,

Asutosh

[aemuser001]

Hi Asutosh,

Thank you for all the useful information.

We have on-premise Windows OS with Author and Publisher instances. We are not using the Cloud yet. 

We know for sure that both instances were running fine in the first week of March. When we reverted back to that week's server backup, and looked at the crx-quickstart/repository folder, we noticed that the 3 subfolders were still missing. So how was Author running in the first week of March when the 3 subfolders were missing?

Thanks,

Leena

[Asutosh_Jena_]

Hi Leena,

I doubt if the backup is running correctly. If those folders were not available in the system, system will never run without any error.

Can you restore it to somewhere 15-20 prior backup and see if it has the folders or not? Without those folders in repository node AEM will not start.

Thanks!

[aemuser001]

Hi Asutosh,

Thank you for all the information. We will try to restore from earlier backup to retrieve the folders in crx-quickstart/repository. I will need some time to provide an update to this post with that information.

Thanks again,

Leena

[aemuser001]

Hi Asutosh,

We did find an old backup copy of the 4 folders from many months ago. We have this new information that the Author instance is currently using Oracle DB repository and not the default TARMK file system repository. I may be wrong, but I think that this old backup is from the initial Author installation, and then we had configured it to use the Oracle database. We can successfully connect to the DB, and it has datastore, journal, nodes, settings and clusternodes as tables.

Do we still need the 4 folders in crx-quickstart/respository? If yes, will it impact or overwrite any data in the database once we copy these 4 old folders and restart the author instance?

In case there is no need to copy these 4 folders, is there any DB or AEM Author error log that will show any specific errors?

Thanks,

Leena

[Asutosh_Jena_]

Hi @aemuser001 

Please refer the below article:

https://experienceleague.adobe.com/docs/experience-manager-65/deploying/deploying/rdbms-support-in-a...

If still the issue persist, you might need to go with a fresh installation.

Thanks!

[aemuser001]

Hi Asutosh,

Thanks I will follow up on this suggestion.

Thanks,

Leena

[aemuser001]

Hi Asutosh,

I did not do any fresh install of the Author. It started to work after some kind of maintenance work was done by the DB team this weekend. It is not clear what was done.

Thank you for all the help. I learnt a lot from troubleshooting AEM from this discussion.

Thanks,

Leena

[Asutosh_Jena_]

Hi @aemuser001 

That's really great the issue has been resolved. It must something to do with DB config and they would have fixed it as part of the maintenance work.

Thanks

Asutosh

[Love_Sharma]

For future visitors, Last time when I saw this error was when I was trying to switch between AEM 6.3 and AEM 6.5 instance. Deleting repoinit folder from my AEM instance fixed this issue for me. Either you can run below command

 rm -rf crx-quickstart/launchpad/config/org/apache/sling/jcr/repoinit OR delete the folder manually. 

[chandracq]

This may or may not be relevant but might help somebody.

We cloned AEM from Prod 6.4.8 2012  Win 2012 to lower environment  Win 2019.  upgraded from 6.4.8 to 6.5.15 and also upgraded Oracle Java 8.21 to Zulu 11 64 Bit.

Everything worked fine with java 8 but as soon as sa i switched to Java 11.. it was giving an HTTP 503/500 error.

After several hours of investigation, i found there was an unused JAAS Authentication configuration sitting in OSGI for many years, was not getting executed in Java 8 but it was executed in JAVA 11. 

I switched back to Java 8 and cleaned that and switched back to 11 and all works consistently 

Regards,

Chandra