An instance of AEM 6.5.4 is running on Windows OS. The crx respository is in Oracle database, and the following message is seen in the error.log
12.01.2023 00:00:00.375 *ERROR* [sling-default-3-AdobeSignRefreshTokenScheduleJob] com.adobe.forms.foundation.oauth.model.OAuthConfigSlingModel Refresh Token not present for: /conf/global/settings/cloudconfigs/fdm/ms-dynamics-odata-cloud-service/jcr:content
[sling-default-3-health-org.apache.sling.discovery.oak.SynchronizedClocksHealthCheck] org.apache.sling.discovery.oak.SynchronizedClocksHealthCheck execute: clocks in local cluster out of sync by 1000msms which is equal or higher than the low-water mark of 1000ms.
Earlier in the last few months, the HTTP 503 error was displayed when accessing the admin console from /aem/start.html. On restarting AEM instance the issue will go away. Now receiving the error 'Server side error encountered. Please contact AEM administrator', and restart of AEM instance is not resolving this issue. The issue appears to be with the UI display. I can access crx/explorer/index.jsp.
One suggestion given to us earlier was to try to sync the time between the oracle DB and AEM server clock. My DB team informed that the clocks are in sync.
Any help on how to resolve it will be great.
Solved! Go to Solution.
As per the sling docs, SynchronizedClocksHealthCheck class checks how much the local time differs from the DocumentStore's time. It then applies low- and high-water marks to that time difference:
Based on the error msg, you're seeing a warning. Can you check if the time difference (if any) between the local server and DB time is lesser/greater than 1 sec?
Do you see this message whenever you try to render the AEM home page or after a periodic interval? Any other msgs when you try to render the page?
I am informed that the time difference between the AEM server and DB server is less than 100 ms.
The message entry is periodic every minute in the error.log. This may not be the cause for the Server side error.
Tried to troubleshoot by stopping the AEM instance and removing the index folder
On restart of the AEM instance, the /aem/start.html shows as 'startup in progress', and on refreshing the page after few minutes it displays the Server side error. The error.log has this entry:
18.01.2023 16:48:40.647 *ERROR* [10.3.248.58  GET /libs/granite/core/content/login.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Uncaught SlingException
java.io.IOException: Error while reading cached library: javax.jcr.RepositoryException: java.io.IOException: java.lang.RuntimeException: failed to read block from backend, id 1a62282beed54f062f8202305d4e3af4e9d1cb7a7fe5727a4506afb34d06822d@0
Is there a way to recover this specific id?
Can't access /aem/start.html, /system/console/ but can access crx/de/index.jsp, and /crx/explorer/index.jsp
On stopping the AEM instance, removing the Index folder and restarting it again, the text in the following links is now displayed, but not the GUI. This happens after logging to AEM from /crx/de/index.jsp The screenshot for /aem/start.html is attached. Earlier the server side error was seen before removing the index folder and restarting the AEM instance.
The /system/console is now displayed normally and can access /bundles, /configMgr etc., The error.log shows that few blog ids are not found. Similar to the one below.
Caused by: java.lang.RuntimeException: failed to read block from backend, id 1a62282beed54f062f8202305d4e3af4e9d1cb7a7fe5727a4506afb34d06822d@0 …
Caused by: java.io.IOException: Datastore block 1a62282beed54f062f8202305d4e3af4e9d1cb7a7fe5727a4506afb34d06822d not found
1. The blob ids are available in a backup copy of DATASTORE_DATA and DATASTORE_META tables. Any suggestion on how to restore these ids? Is there any way to restore/populate the missing ids from the existing GUI? The /crx/de/index.jsp, /system/console/configMgr, /system/console and /libs/cq/core/content/welcome.html are accessible, or from any other URL?
2. Is the id with ...@0 populated from 1a62282beed54f062f8202305d4e3af4e9d1cb7a7fe5727a4506afb34d06822d? Meaning if 1a62282beed54f062f8202305d4e3af4e9d1cb7a7fe5727a4506afb34d06822d had existed then the ...@0 would have been created from it.
3. Any suggestion on how to restore the missing IDs?
I have reviewed the logs shared and could see the posted error msgs.
These two tables: DATASTORE_DATA and DATASTORE_META contains information for a blobId reference so you can restore the database state to check if the issue is resolved.
I recall now, we have had a similar discussion sometime back, why are you seeing such issues on this setup so frequently- Could be a disk space outage, or network share instability?
19.01.2023 07:54:38.485 *ERROR* [10.3.248.58  GET /libs/granite/core/content/login.html HTTP/1.1] com.adobe.granite.ui.clientlibs.impl.HtmlLibraryManagerImpl Cannot read input stream javax.jcr.RepositoryException: java.io.IOException: java.lang.RuntimeException: failed to read block from backend, id 1a62282beed54f062f8202305d4e3af4e9d1cb7a7fe5727a4506afb34d06822d@0