Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
Bedrock Mission!

Learn more

View all

Sign in to view all badges

AEM 6.5 HTTP 503 Error AuthenticationSupport service missing. Cannot authenticate request

Leena_1
Level 4
Level 4

Hello Members,

I have a AEM 6.5 OSGi framework on-premise Author and Publisher instances running in Windows OS. It was working fine for a long time. We took a snapshot of the Windows server and we used the Window's Services to restarted the Author and Publisher instances. The Publisher instance is running fine, but the Author instance is not. The error.log in Author instance shows the following error:

19.03.2021 15:27:41.963 *ERROR* [qtp1031644809-61] org.apache.sling.engine.impl.SlingHttpContext handleSecurity: AuthenticationSupport service missing. Cannot authenticate request. 19.03.2021 15:27:41.964 *ERROR* [qtp1031644809-61] org.apache.sling.engine.impl.SlingHttpContext handleSecurity: Possible reason is missing Repository service. Check AuthenticationSupport dependencies.

The URL to server:port/aem/start.html displays the following error message in the browser:

HTTP ERROR 503 Problem accessing /aem/start.html. Reason: AuthenticationSupport service missing. Cannot authenticate request.

The link to crx  -  http://server:port/crx/de/index.jsp is displaying the following error:

HTTP ERROR 503 Problem accessing /crx/de/index.jsp. Reason: Service Unavailable

http://server:port/crx/packmgr/index.jsp gives the following error:

Problem accessing /crx/packmgr/index.jsp. Reason:

AuthenticationSupport service missing. Cannot authenticate request.

I do see that the Author's crx-quickstart/repository only has cache, diff-cache and index folders. It does not have the segmentstore, datastore and blobids folders, but these 3 folders exist in the Publisher instance.

I did try to rename the index folder and restarted the Author instance, but it did not create a new index folder. The user has full permissions to the folder.

Our team tried to revert back to previous state by using the snapshot and server backup from last month, but the issue is the same.

I can access the /system/console/memoryusage and it uses only 10% of memory.

The below links work in Author and all the bundles are active. The login is like a pop-up near the browser tab, and it is not the usual login page that is used to access these links.

 http://server:port/system/console/slinglog

 http://server:port/system/console/bundles

1. Is there any suggestion on what to try next to recover from this issue.

2. Are the segmentstore, datastore and blobids folders needed in Author's crx-quickstart/repository? If yes, is it possible to upload specific packages or drop it in a specific folder structure

3. Are there any specific steps to be taken to backup crx repository when server is backed up, or a snapshot is taken?

Thanks in advance for the response,

Leena

18 Replies
Asutosh_Jena_
Community Advisor
Community Advisor

Hi @Leena_1 

Please follow the below steps:

  1. Shutdown the instance.
  2. Delete all index from the index folder.(\crx-quickstart\repository\index)
  3. Restart the AEM instance.

This should resolve the issue.

 

Thanks!

Leena_1
Level 4
Level 4

Hi Asutosh, I tried your suggestion to stop the instance, removed the folders within index folder and start it again. Now the following error is displayed when accessing server:port/aem/start.html :

HTTP ERROR 404

Problem accessing /aem/start.html. Reason:

Not Found

 

Any suggestion?

Thanks,

Leena

Asutosh_Jena_
Community Advisor
Community Advisor

Hi @Leena_1 

Which service pack is currently installed on AEM 6.5? Also can you mention if you are using JDK 8 or 11?

 

Thanks!

Leena_1
Level 4
Level 4

I also need to add that after refreshing the browser I am getting the original error again:

AEM 6.5 HTTP 503 Error AuthenticationSupport service missing. Cannot authenticate request

Asutosh_Jena_
Community Advisor
Community Advisor

Hi @Leena_1 

Can you upgrade it to latest service pack i.e. 6.5.8 and try it again. Also you must be getting some error in the log when you get 503 error on screen. What is that error?

 

Thanks!

Leena_1
Level 4
Level 4

Hi Asutosh,

Thank you for your response and suggestions. The Author instance worked fine in using JDK 1.8 and Adobe Experience Manager 6.5.4.0 for nearly a year. Due to business considerations I will not be able to upgrade to service pack i.e. 6.5.8 now. 

Currently the the Author's crx-quickstart/repository only has cache, diff-cache and index folders. It does not have the segmentstore, datastore and blobids folders, but these 3 folders exist in the Publisher instance. Can you please confirm if these 3 subfolders are needed in the Author instance too?

The following error is seen in the startup.log when Author instance was restarted:

2021-04-05 08:40:13 Commons Daemon procrun stdout initialized

2021-04-05 08:40:13 Commons Daemon procrun stderr initialized
05.04.2021 08:40:13.426 *INFO * [main] Setting sling.home=D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\\..\.. (command line)
05.04.2021 08:40:14.473 *INFO * [main] No Apache Sling running at /127.0.0.1:63000
05.04.2021 08:40:14.473 *INFO * [main] Starting Apache Sling in D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\\..\..
05.04.2021 08:40:14.473 *INFO * [main] Sling Extension Lib Home : D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext
05.04.2021 08:40:14.504 *INFO * [main] Checking launcher JAR in folder D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad
05.04.2021 08:40:14.535 *INFO * [main] Existing launcher is up to date, using it: 6.0.2.2_6_36 (org.apache.sling.launchpad.base.jar)
05.04.2021 08:40:14.551 *INFO * [main] Loading launcher class org.apache.sling.launchpad.base.app.MainDelegate from org.apache.sling.launchpad.base.jar
05.04.2021 08:40:14.551 *INFO * [main] Sling Extension jars found = [ D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\bcmail-1.51.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\bcpkix-1.51.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\bcprov-1.51.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\certjFIPS-2.1.2.0.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\cryptojce-6.0.0.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\cryptojcommon-6.0.0.jar,D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad\ext\jcmFIPS-6.0.0.jar, ]
05.04.2021 08:40:14.660 *INFO * [main] Setting sling.properties=D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\\..\..\conf\sling.properties
05.04.2021 08:40:14.660 *INFO * [main] Setting sling.home=D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\\..\..
05.04.2021 08:40:14.660 *INFO * [main] Setting sling.launchpad=D:\Adobe_Experience_Manager_Development\crx-quickstart\opt\helpers\..\..\launchpad
05.04.2021 08:40:14.660 *INFO * [main] Setting org.osgi.service.http.port=4502
05.04.2021 08:40:14.660 *INFO * [main] Starting launcher ...
05.04.2021 08:40:14.660 *INFO * [Apache Sling Control Listener@/127.0.0.1:63000] Apache Sling Control Listener started
05.04.2021 08:40:14.676 *INFO * [main] HTTP server port: 4502
RAWProcessor succesfully installed
05.04.2021 08:40:54.076 *INFO * [main] Startup completed
java.lang.NullPointerException
at com.day.crx.explorer.impl.j2ee.JCRExplorerServlet.getDefaultSession(JCRExplorerServlet.java:283)
at com.day.crx.explorer.impl.j2ee.CRXContext.getSession(CRXContext.java:245)
at com.day.crx.explorer.impl.j2ee.CRXHttpServletRequest.getJcrSession(CRXHttpServletRequest.java:39)
at com.day.crx.explorer.impl.j2ee.LoginServlet.redirectToLogin(LoginServlet.java:332)
at com.day.crx.explorer.impl.jsp.index_jsp._jspService(index_jsp.java:87)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at com.day.crx.explorer.impl.MainServlet.doService(MainServlet.java:171)
at com.day.crx.explorer.impl.MainServlet.service(MainServlet.java:89)
at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:123)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:86)
at com.adobe.granite.license.impl.LicenseCheckFilter.doFilter(LicenseCheckFilter.java:312)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81)
at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:131)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81)
at org.apache.felix.http.base.internal.dispatch.Dispatcher$1.doFilter(Dispatcher.java:146)
at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager$2.doFilter(WhiteboardManager.java:1002)
at org.apache.sling.security.impl.ReferrerFilter.doFilter(ReferrerFilter.java:326)
at org.apache.felix.http.base.internal.handler.PreprocessorHandler.handle(PreprocessorHandler.java:136)
at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager$2.doFilter(WhiteboardManager.java:1008)
at org.apache.felix.http.sslfilter.internal.SslFilter.doFilter(SslFilter.java:97)
at org.apache.felix.http.base.internal.handler.PreprocessorHandler.handle(PreprocessorHandler.java:136)
at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager$2.doFilter(WhiteboardManager.java:1008)
at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager.invokePreprocessors(WhiteboardManager.java:1012)
at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:91)
at org.apache.felix.http.base.internal.dispatch.DispatcherServlet.service(DispatcherServlet.java:49)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:544)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1581)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1307)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:482)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1549)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1204)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:494)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:374)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:268)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)
at java.lang.Thread.run(Thread.java:748)

Thanks,

Leena

Asutosh_Jena_
Community Advisor
Community Advisor

Hi @Leena_1 

Yes, you will need the below folders to be available within the repository folder.

  1. blobids
  2. datastore
  3. segmentstore
  4. index

All the repository related data will be stored under datastore folder and segment files are stored under the segment folder.

If you have any backup running on the system, you might be able to restore those. If not, you will need to rebuild the instance.

 

You can get more information about the folders here:

https://experienceleague.adobe.com/docs/experience-manager-65/deploying/deploying/data-store-config....

 

 

Thanks!

Asutosh

 

Leena_1
Level 4
Level 4

Hi Asutosh,

Thanks for confirming. We will try to restore from our last December backup of the server. Hopefully the 3 subfolders are present in that backup.

1. What could have caused the 3 subfolders to be missing in the crx-quickstart/respository folder?

2. To avoid this issue in the future, do I need to manually copy these folders to a different location before taking a snapshot or backup of the server? The reason I ask this is that even after restoring the server from few weeks ago (when we know the Author was up and running at that time), and now when we check the crx-quickstart/respository folder the 3 subfolders are missing even in the recently restored server. I am assuming the 3 folders were missing for sometime, and the reason the Author instance was still running was because the application was using the resources in the JVM. Once the Author instance was stopped and restarted, it was then that Author instance was looking for these subfolders and it failed to start.

Is this assumption right? Is there any other take on it?

Thanks,

Leena

Asutosh_Jena_
Community Advisor
Community Advisor

Hi Leena,

 

These folders contains huge amount of data and it will never be deleted automatically. I doubt if something happened on this instance which deleted the file/it might be accidentally deleted by someone while performing any action on the file system.

 

If you want to take the offline backup of only your AEM instance, then you can schedule some time (depending upon the usage schedule) and schedule the backup process as it will take less than a few minutes, but the application needs to be stopped.

To create a offline backup perform these steps:

  1. Stop the application
  2. Make a SNAPSHOT backup
  3. Start the application

Read more details here:

https://experienceleague.adobe.com/docs/experience-manager-65/administering/operations/backup-and-re...

 

You can take the SNAPSHOT backup of the whole instance i.e. if you are using this on an EC2 instance, you can take the complete volume backup where AEM is installed. Let's say you have /data as the mount point where you have the AEM instance available then you can take a weekly backup of the instance so if anything goes wrong at any point of time you still will have 1 week prior data which can be restored. This can be automated using AWS SNAPSHOT backup and there will not be any downtime as this can be taken when the system is running.

 

Please refer the link below for more info:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSSnapshots.html

 

Thanks,

Asutosh

Leena_1
Level 4
Level 4

Hi Asutosh,

 

Thank you for all the useful information.

We have on-premise Windows OS with Author and Publisher instances. We are not using the Cloud yet. 

We know for sure that both instances were running fine in the first week of March. When we reverted back to that week's server backup, and looked at the crx-quickstart/repository folder, we noticed that the 3 subfolders were still missing. So how was Author running in the first week of March when the 3 subfolders were missing?

Thanks,

Leena

Asutosh_Jena_
Community Advisor
Community Advisor

Hi Leena,

 

I doubt if the backup is running correctly. If those folders were not available in the system, system will never run without any error.

Can you restore it to somewhere 15-20 prior backup and see if it has the folders or not? Without those folders in repository node AEM will not start.

 

Thanks!

Leena_1
Level 4
Level 4

Hi Asutosh,

Thank you for all the information. We will try to restore from earlier backup to retrieve the folders in crx-quickstart/repository. I will need some time to provide an update to this post with that information.

Thanks again,

Leena

Leena_1
Level 4
Level 4

Hi Asutosh,

We did find an old backup copy of the 4 folders from many months ago. We have this new information that the Author instance is currently using Oracle DB repository and not the default TARMK file system repository. I may be wrong, but I think that this old backup is from the initial Author installation, and then we had configured it to use the Oracle database. We can successfully connect to the DB, and it has datastore, journal, nodes, settings and clusternodes as tables.

Do we still need the 4 folders in crx-quickstart/respository? If yes, will it impact or overwrite any data in the database once we copy these 4 old folders and restart the author instance?

In case there is no need to copy these 4 folders, is there any DB or AEM Author error log that will show any specific errors?

Thanks,

Leena

Leena_1
Level 4
Level 4

Hi Asutosh,

Thanks I will follow up on this suggestion.

Thanks,

Leena

Leena_1
Level 4
Level 4

Hi Asutosh,

I did not do any fresh install of the Author. It started to work after some kind of maintenance work was done by the DB team this weekend. It is not clear what was done.

Thank you for all the help. I learnt a lot from troubleshooting AEM from this discussion.

Thanks,

Leena

Asutosh_Jena_
Community Advisor
Community Advisor

Hi @Leena_1 

 

That's really great the issue has been resolved. It must something to do with DB config and they would have fixed it as part of the maintenance work.

 

Thanks

Asutosh