Unable to load DAM PDfs on author instance on stage e...

[samikshaa223429]

hi, 

I am unable to load DAM PDfs on author instance on stage environment.getting the below error in logs :

23.07.2020 18:39:16.794 *ERROR* [10.43.32.115 [1595529556752] GET /content/dam/project1/taxonomy1/project2/pdf-link-final.pdf HTTP/1.1] org.apache.jackrabbit.oak.plugins.blob.AbstractSharedCachingDataStore Error retrieving record [51b4de9db39bc1a42525a9c557251b9e1f2ea52a]
org.apache.jackrabbit.core.data.DataStoreException: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 36B10B67290D09C9; S3 Extended Request ID: UrjJi0GLuGgbvOCwYx3qYArjL87Pmu1pFTlQw1uk2Q/FrecF9rQkNmgNlP/ynkBXLd8QmQuZ5ZY=), S3 Extended Request ID: UrjJi0GLuGgbvOCwYx3qYArjL87Pmu1pFTlQw1uk2Q/FrecF9rQkNmgNlP/ynkBXLd8QmQuZ5ZY=
at org.apache.jackrabbit.oak.blob.cloud.s3.S3Backend.getRecord(S3Backend.java:638)
at org.apache.jackrabbit.oak.plugins.blob.AbstractSharedCachingDataStore.getRecordIfStored(AbstractSharedCachingDataStore.java:210)
at org.apache.jackrabbit.oak.plugins.blob.AbstractSharedCachingDataStore.getRecord(AbstractSharedCachingDataStore.java:188)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:622)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:607)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:345)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:341)
at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1013)
at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:974)
at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:285)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:341)
at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:46)
at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:247)
at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:83)
at org.apache.jackrabbit.oak.plugins.value.jcr.BinaryImpl.getStream(BinaryImpl.java:59)
at org.apache.sling.jcr.resource.internal.helper.jcr.JcrNodeResource.getInputStream(JcrNodeResource.java:214)
at org.apache.sling.jcr.resource.internal.helper.jcr.JcrNodeResource.adaptTo(JcrNodeResource.java:133)
at org.apache.sling.servlets.get.impl.helpers.StreamRenderer.render(StreamRenderer.java:152)
at org.apache.sling.servlets.get.impl.DefaultGetServlet.doGet(DefaultGetServlet.java:316)
at org.apache.sling.api.servlets.SlingSafeMethodsServlet.mayService(SlingSafeMethodsServlet.java:266)
at org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:342)
at org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:374)
at org.apache.sling.engine.impl.request.RequestData.service(RequestData.java:552)
at org.apache.sling.engine.impl.filter.SlingComponentFilterChain.render(SlingComponentFilterChain.java:44)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:82)
at com.day.cq.dam.core.impl.servlet.DamContentDispositionFilter.doFilter(DamContentDispositionFilter.java:104)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.granite.rest.assets.impl.AssetContentDispositionFilter.doFilter(AssetContentDispositionFilter.java:96)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.wcm.core.impl.WCMDebugFilter.doFilter(WCMDebugFilter.java:138)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.wcm.core.impl.WCMComponentFilter.filterRootInclude(WCMComponentFilter.java:375)
at com.day.cq.wcm.core.impl.WCMComponentFilter.doFilter(WCMComponentFilter.java:190)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.wcm.core.impl.page.PageLockFilter.doFilter(PageLockFilter.java:91)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:131)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.personalization.impl.TargetComponentFilter.doFilter(TargetComponentFilter.java:94)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at org.apache.sling.security.impl.ContentDispositionFilter.doFilter(ContentDispositionFilter.java:152)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.granite.csrf.impl.CSRFFilter.doFilter(CSRFFilter.java:217)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at org.apache.sling.engine.impl.SlingRequestProcessorImpl.processComponent(SlingRequestProcessorImpl.java:282)
at org.apache.sling.engine.impl.SlingRequestProcessorImpl.dispatchRequest(SlingRequestProcessorImpl.java:322)
at org.apache.sling.engine.impl.request.SlingRequestDispatcher.dispatch(SlingRequestDispatcher.java:211)
at org.apache.sling.engine.impl.request.SlingRequestDispatcher.forward(SlingRequestDispatcher.java:146)
at com.day.cq.wcm.core.impl.WCMComponentFilter$ForwardRequestDispatcher.forward(WCMComponentFilter.java:503)
at com.day.cq.dam.core.impl.servlet.BinaryProviderServlet.serveTheBits(BinaryProviderServlet.java:245)
at com.day.cq.dam.core.impl.servlet.BinaryProviderServlet.doGet(BinaryProviderServlet.java:134)
at org.apache.sling.api.servlets.SlingSafeMethodsServlet.mayService(SlingSafeMethodsServlet.java:266)
at org.apache.sling.api.servlets.SlingAllMethodsServlet.mayService(SlingAllMethodsServlet.java:137)
at org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:342)
at org.apache.sling.api.servlets.SlingSafeMethodsServlet.service(SlingSafeMethodsServlet.java:374)
at org.apache.sling.engine.impl.request.RequestData.service(RequestData.java:552)
at org.apache.sling.engine.impl.filter.SlingComponentFilterChain.render(SlingComponentFilterChain.java:44)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:82)
at com.day.cq.wcm.core.impl.WCMDebugFilter.doFilter(WCMDebugFilter.java:138)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.wcm.core.impl.WCMComponentFilter.filterRootInclude(WCMComponentFilter.java:375)
at com.day.cq.wcm.core.impl.WCMComponentFilter.doFilter(WCMComponentFilter.java:190)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.wcm.core.impl.page.PageLockFilter.doFilter(PageLockFilter.java:91)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.personalization.impl.TargetComponentFilter.doFilter(TargetComponentFilter.java:94)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at org.apache.sling.engine.impl.SlingRequestProcessorImpl.processComponent(SlingRequestProcessorImpl.java:282)
at org.apache.sling.engine.impl.filter.RequestSlingFilterChain.render(RequestSlingFilterChain.java:49)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:82)
at com.day.cq.dam.core.impl.servlet.DamContentDispositionFilter.doFilter(DamContentDispositionFilter.java:104)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.dam.core.impl.servlet.ActivityRecordHandler.doFilter(ActivityRecordHandler.java:141)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.dam.core.impl.assetlinkshare.AdhocAssetShareAuthHandler.doFilter(AdhocAssetShareAuthHandler.java:434)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.cq.dam.webdav.impl.io.DamWebdavRequestFilter.doFilter(DamWebdavRequestFilter.java:132)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:78)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:78)
at com.day.cq.wcm.core.impl.warp.TimeWarpFilter.doFilter(TimeWarpFilter.java:109)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.cq.social.ugcbase.security.impl.SaferSlingPostServlet.doFilter(SaferSlingPostServlet.java:114)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.granite.rest.impl.servlet.ApiResourceFilter.doFilter(ApiResourceFilter.java:70)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.granite.requests.logging.impl.RequestLoggerImpl.doFilter(RequestLoggerImpl.java:134)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.granite.rest.assets.impl.AssetContentDispositionFilter.doFilter(AssetContentDispositionFilter.java:96)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.wcm.core.impl.AuthoringUIModeServiceImpl.doFilter(AuthoringUIModeServiceImpl.java:372)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.wcm.mobile.core.impl.redirect.RedirectFilter.doFilter(RedirectFilter.java:248)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter.doFilter(RequestProgressTrackerLogFilter.java:107)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.cq.social.commons.cors.CORSAuthenticationFilter.doFilter(CORSAuthenticationFilter.java:91)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.granite.optout.impl.OptOutFilter.doFilter(OptOutFilter.java:76)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.wcm.foundation.forms.FormsHandlingServletHelper.handleFilter(FormsHandlingServletHelper.java:226)
at com.day.cq.wcm.foundation.forms.impl.FormsHandlingServlet.doFilter(FormsHandlingServlet.java:138)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.wcm.foundation.forms.FormsHandlingServletHelper.handleFilter(FormsHandlingServletHelper.java:226)
at com.adobe.cq.wcm.core.components.internal.servlets.CoreFormHandlingServlet.doFilter(CoreFormHandlingServlet.java:131)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.cq.history.impl.HistoryRequestFilter.doFilter(HistoryRequestFilter.java:122)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.day.cq.wcm.core.impl.WCMRequestFilter.doFilter(WCMRequestFilter.java:90)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.cq.mcm.campaign.servlets.CampaignCopyTracker.doFilter(CampaignCopyTracker.java:100)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at org.apache.sling.rewriter.impl.RewriterFilter.doFilter(RewriterFilter.java:87)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.granite.httpcache.impl.InnerCacheFilter.doFilter(InnerCacheFilter.java:81)
at com.adobe.granite.httpcache.impl.InnerCacheFilter.doFilter(InnerCacheFilter.java:60)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:131)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at org.apache.sling.security.impl.ContentDispositionFilter.doFilter(ContentDispositionFilter.java:152)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.granite.csrf.impl.CSRFFilter.doFilter(CSRFFilter.java:217)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at com.adobe.granite.resourceresolverhelper.impl.ResourceResolverHelperImpl.doFilter(ResourceResolverHelperImpl.java:83)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:72)
at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:78)
at org.apache.sling.engine.impl.SlingRequestProcessorImpl.doProcessRequest(SlingRequestProcessorImpl.java:151)
at org.apache.sling.engine.impl.SlingMainServlet.service(SlingMainServlet.java:244)
at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:123)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:86)
at com.adobe.granite.license.impl.LicenseCheckFilter.doFilter(LicenseCheckFilter.java:308)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81)
at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:131)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81)
at org.apache.sling.featureflags.impl.FeatureManager.doFilter(FeatureManager.java:116)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81)
at org.apache.sling.engine.impl.log.RequestLoggerFilter.doFilter(RequestLoggerFilter.java:72)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81)
at org.apache.sling.engine.impl.parameters.RequestParameterSupportConfigurer.doFilter(RequestParameterSupportConfigurer.java:63)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:142)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:81)
at org.apache.felix.http.base.internal.dispatch.Dispatcher$1.doFilter(Dispatcher.java:146)
at org.apache.felix.http.base.internal.whiteboard.WhiteboardManager$2.doFilter(WhiteboardManager.java:1002)
at org.apache.sling.security.impl.ReferrerFilter.doFilter(ReferrerFilter.java:326)
at org.apache.felix.http.base.internal.handler.PreprocessorHandler.handle(PreprocessorHandler.java:136)

Could it be related to segment store corruption?

thanks,

Samiksha

[VeenaVikraman]

Are you sure this PDF is available in S3 ?

what I understand from this error in the method getRecordIfStored() https://github.com/apache/jackrabbit-oak/blob/trunk/oak-blob-plugins/src/main/java/org/apache/jackra... , It is throwing an error since the file is not cached and trying to fetch the record is throwing error. (see line 210-214)

Could you please also check if any of the below will help ?

https://helpx.adobe.com/in/experience-manager/kb/missing-s3-datastore-blobs.html

https://helpx.adobe.com/in/experience-manager/kb/aem-s3datastore-mongo-cluster-encounter-s3-404-.htm...

Thanks

Veena

[vanegi · Accepted solution]

Hi @samikshaa223429,

This error message [0] means segment store is trying to fetch asset which does not exist in S3. Which can be either due to content or index?

The index can be restored. Index Consistency Check:
java -jar oak-run*.jar index --fds-path=/path/to/datastore /path/to/segmentstore/ --index-consistency-check
--index-consistency-checkoperation performs index consistency check against various indexes. It supports 2 level
• Level 1 - Specified as--index-consistency-check=1. Performs a basic check to determine if all blobs referred in index are valid
• Level 2 - Specified as--index-consistency-check=2. Performs a more through check to determine if all index files are valid and no corruption has happened. This check is slower, it would generate a report in<output dir>/index-consistency-check-report.txt and supported for only Lucene indexes.

For content, there is no guarantee.
To confirm if the content blob is missing, you can tell that by running the consistency check at - ```system/console/repositorycheck```
just the datastore one, that ONLY shows missing content blobs.

~ if there are missing content blobs, there is trouble (assuming they are not running with S3 versioning enabled)
~ if this shows no missing blobs, you know the missing blobs are indexes
that's easy to fix:

- use Oak Run Console to delete all indexed content from repo (there is a Lucene remove command)
- use Oak Run to remove all checkpoints
- Start AEM and let it reindex
that will reindex and recreate the 'missing' blobs

In case of missing blobs, you can either delete these nodes from the dam and then trigger compaction to delete all the references in the segment store or restore the blob id. Please check if they are present in S3 or not. If they are present in S3 and not on AEM then follow these steps:

• Stop AEM

• Remove all FILES beneath S3 download cache (but leave folders)

• Chowned folders to be owned by root • Start AEM

[0]:

*ERROR* [10.43.32.115 [1595529556752] GET /content/dam/project1/taxonomy1/project2/pdf-link-final.pdf HTTP/1.1] org.apache.jackrabbit.oak.plugins.blob.AbstractSharedCachingDataStore Error retrieving record [51b4de9db39bc1a42525a9c557251b9e1f2ea52a]
org.apache.jackrabbit.core.data.DataStoreException: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 36B10B67290D09C9; S3 Extended Request ID: UrjJi0GLuGgbvOCwYx3qYArjL87Pmu1pFTlQw1uk2Q/FrecF9rQkNmgNlP/ynkBXLd8QmQuZ5ZY=), S3 Extended Request ID: UrjJi0GLuGgbvOCwYx3qYArjL87Pmu1pFTlQw1uk2Q/FrecF9rQkNmgNlP/ynkBXLd8QmQuZ5ZY=
at org.apache.jackrabbit.oak.blob.cloud.s3.S3Backend.getRecord(S3Backend.java:638)

.

.

Thanks!!