Segment Not Found Issue on PROD

[Madhu_Mohan_Jel]

Hi Team,

Your help here is hightly appreciated. We are facing this issue lot of times on PROD instances and we could not find the root cause of this issue yet. Please help

com.adobe.acs.commons.rewriter.impl.VersionedClientlibsTransformerFactory Attempting to get a versioned path for [ /etc/designs/xxx/clientlib-css.css ] but could not because of: Segment e7348625-7546-45ee-b16c-9e9b8608cef5 not found

com.adobe.acs.commons.rewriter.impl.VersionedClientlibsTransformerFactory Attempting to get a versioned path for [ /etc/designs/xxx/clientlib-js.js ] but could not because of: Segment 3b1bd1be-48a1-4f91-b10e-2c956d8916a8 not found

Please find the complete log below

org.apache.jackrabbit.oak.plugins.segment.SegmentTracker Segment not found: 3b1bd1be-48a1-4f91-b10e-2c956d8916a8. Creation date delta is 18305953 ms.
org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment 3b1bd1be-48a1-4f91-b10e-2c956d8916a8 not found
    at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:855)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:134)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:101)
    at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
    at org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.java:52)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream.java:168)
    at org.apache.commons.codec.digest.DigestUtils.digest(DigestUtils.java:51)
    at org.apache.commons.codec.digest.DigestUtils.md5(DigestUtils.java:179)
    at org.apache.commons.codec.digest.DigestUtils.md5Hex(DigestUtils.java:215)
    at com.adobe.acs.commons.rewriter.impl.VersionedClientlibsTransformerFactory.getVersionedPath(VersionedClientlibsTransformerFactory.java:140)
    at com.adobe.acs.commons.rewriter.impl.VersionedClientlibsTransformerFactory.rebuildAttributes(VersionedClientlibsTransformerFactory.java:81)
    at com.adobe.acs.commons.rewriter.impl.VersionedClientlibsTransformerFactory.versionClientLibs(VersionedClientlibsTransformerFactory.java:71)
    at com.adobe.acs.commons.rewriter.impl.VersionedClientlibsTransformerFactory.access$100(VersionedClientlibsTransformerFactory.java:49)
    at com.adobe.acs.commons.rewriter.impl.VersionedClientlibsTransformerFactory$VersionableClientlibsTransformer.startElement(VersionedClientlibsTransformerFactory.java:160)
    at org.apache.cocoon.xml.sax.AbstractSAXPipe.startElement(AbstractSAXPipe.java:97)
    at com.day.cq.rewriter.linkchecker.impl.LinkCheckerTransformer.startElement(LinkCheckerTransformer.java:341)
    at org.apache.cocoon.xml.sax.AbstractSAXPipe.startElement(AbstractSAXPipe.java:97)
    at com.day.cq.mcm.core.newsletter.NewsletterTransformerFactory$NewsletterTransformer.startElement(NewsletterTransformerFactory.java:132)
    at com.day.cq.rewriter.htmlparser.DocumentHandlerToSAXAdapter.onStartElement(DocumentHandlerToSAXAdapter.java:105)
    at com.day.cq.rewriter.htmlparser.HtmlParser.processTag(HtmlParser.java:621)
    at com.day.cq.rewriter.htmlparser.HtmlParser.update(HtmlParser.java:326)
    at com.day.cq.rewriter.htmlparser.HtmlParser.write(HtmlParser.java:179)
    at java.io.Writer.write(Writer.java:192)
    at java.io.PrintWriter.write(PrintWriter.java:456)
    at org.apache.sling.scripting.core.impl.helper.OnDemandWriter.write(OnDemandWriter.java:75)
    at java.io.PrintWriter.write(PrintWriter.java:456)
    at org.apache.sling.scripting.core.impl.helper.OnDemandWriter.write(OnDemandWriter.java:75)
    at java.io.PrintWriter.write(PrintWriter.java:456)
    at java.io.PrintWriter.write(PrintWriter.java:473)
    at io.sightly.java.api.StackedWriter.write(StackedWriter.java:47)
    at apps.xxx.components.page.basepage.SightlyJava_basepage.render(SightlyJava_basepage.java:79)
    at io.sightly.java.api.BaseRenderUnit.render(BaseRenderUnit.java:39)
    at io.sightly.java.engine.SightlyScriptEngine.evaluateScript(SightlyScriptEngine.java:109)
    at io.sightly.java.engine.SightlyScriptEngine.eval(SightlyScriptEngine.java:93)
    at org.apache.sling.scripting.core.impl.DefaultSlingScript.call(DefaultSlingScript.java:361)
    at org.apache.sling.scripting.core.impl.DefaultSlingScript.eval(DefaultSlingScript.java:171)
    at org.apache.sling.scripting.core.impl.DefaultSlingScript.service(DefaultSlingScript.java:463)
    at org.apache.sling.engine.impl.request.RequestData.service(RequestData.java:533)
    at org.apache.sling.engine.impl.filter.SlingComponentFilterChain.render(SlingComponentFilterChain.java:45)
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:64)
    at com.day.cq.personalization.impl.TargetComponentFilter.doFilter(TargetComponentFilter.java:96)
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
    at com.day.cq.wcm.core.impl.WCMDebugFilter.doFilter(WCMDebugFilter.java:133)
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
    at com.day.cq.wcm.core.impl.WCMComponentFilter.doFilter(WCMComponentFilter.java:252)
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
    at com.day.cq.wcm.core.impl.WCMDeveloperModeFilter.doFilter(WCMDeveloperModeFilter.java:136)
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
    at org.apache.sling.engine.impl.SlingRequestProcessorImpl.processComponent(SlingRequestProcessorImpl.java:282)
    at org.apache.sling.engine.impl.SlingRequestProcessorImpl.dispatchRequest(SlingRequestProcessorImpl.java:322)
    at org.apache.sling.engine.impl.request.SlingRequestDispatcher.dispatch(SlingRequestDispatcher.java:216)
    at org.apache.sling.engine.impl.request.SlingRequestDispatcher.include(SlingRequestDispatcher.java:103)
    at com.day.cq.wcm.core.impl.WCMComponentFilter$ForwardRequestDispatcher.include(WCMComponentFilter.java:431)
    at org.apache.jsp.libs.foundation.components.primary.cq.Page.Page_jsp._jspService(Page_jsp.java:106)
    at org.apache.sling.scripting.jsp.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.apache.sling.scripting.jsp.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:502)
    at org.apache.sling.scripting.jsp.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:449)
    at org.apache.sling.scripting.jsp.JspScriptEngineFactory.callJsp(JspScriptEngineFactory.java:284)
    at org.apache.sling.scripting.jsp.JspScriptEngineFactory.access$100(JspScriptEngineFactory.java:102)
    at org.apache.sling.scripting.jsp.JspScriptEngineFactory$JspScriptEngine.eval(JspScriptEngineFactory.java:536)
    at org.apache.sling.scripting.core.impl.DefaultSlingScript.call(DefaultSlingScript.java:361)
    at org.apache.sling.scripting.core.impl.DefaultSlingScript.eval(DefaultSlingScript.java:171)
    at org.apache.sling.scripting.core.impl.DefaultSlingScript.service(DefaultSlingScript.java:463)
    at org.apache.sling.engine.impl.request.RequestData.service(RequestData.java:533)
    at org.apache.sling.engine.impl.filter.SlingComponentFilterChain.render(SlingComponentFilterChain.java:45)
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:64)
    at com.day.cq.personalization.impl.TargetComponentFilter.doFilter(TargetComponentFilter.java:96)
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
    at com.day.cq.wcm.core.impl.WCMDebugFilter.doFilter(WCMDebugFilter.java:133)
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
    at com.day.cq.wcm.core.impl.WCMComponentFilter.filterRootInclude(WCMComponentFilter.java:357)
    at com.day.cq.wcm.core.impl.WCMComponentFilter.doFilter(WCMComponentFilter.java:166)
    at org.apache.sling.engine.impl.filter.AbstractSlingFilterChain.doFilter(AbstractSlingFilterChain.java:60)
    at org.apache.sling.engine.impl.SlingRequestProcessorImpl.processComponent(SlingRequestProcessorImpl.java:282)
    at org.apache.sling.engine.impl.filter.RequestSlingFilterChain.render(RequestSlingFilterChain.java:49)
   

[ankurk67503819]

        Due to some reason youe repository got corrupted. Kindly run oak utility to fix segment store issue. You need to download oak jar file according to your Aem instance and then you need to run few commands.

[piotrkot · Accepted solution]

Try this:

Fix Inconsistencies in the repository when SegmentNotFound Issue is reported in AEM 6.x

[Opkar_Gill]

Hi,

You need to find out the root cause, so a couple of things to check:

Does the disk run out of space?

When running offline tar compaction, are you running it as the same user that the AEM process is running as? You can check this by making sure all the files in the segmentstore are owned by the same user.

Regards,

Opkar

[apolu]

Hi,

i followed the mentioned steps in your link, any idea how much time will take to find the last good state?

if this is  depends on segment store size  any idea approx how much time will take to finish?

is there a way to check the progress ? ex: logs,any tools ?

And one more question ,after recovery can we get AEM instance back to normal state with content  as looks previous with old data or it is just fresh AEM instance?

Note: We have good network bandwidth

Thanks,

Amarnath

[Chaitanya96]

any idea how much time will take to find the last good state? and

if this is depends on segment store size any idea approx how much time will take to finish? --> in the journal.log you can see found last good segment already mentioned when you would run consistency check command (e.g. aprox around 1 hour if you have 100gb)

after done, AEM works fine and restored.

I am still looking for the root cause of this issue, as Opkar mentioned below points but this not the case for me.

Does the disk run out of space? --> No

When running offline tar compaction, are you running it as the same user that the AEM process is running as? --> Yes

Thanks,

Chaitanya

[venkateshs73486]

Team,

We also face the same issue in 6.4.5 . That too, on a newly setup server.

There is a suspicion that such issue crop up when huge set of assets are uploaded.

Has anyone faced this issue?

Please suggest.

We would like to find the rootcause of SNP issue.