Expand my Community achievements bar.

SOLVED

Purge Versions tool timeout and OutOfMemory

Avatar

Level 4

Hi,

I am trying to purge versions on my AEM6.0+SP3 instance. When I run the tool even an a very small portion of the content tree I get this error from Web Logic. Somehow the process times out or fails and I am not able to purge versions. Second time I tried to run the version purge I have an OutOfMemory.

Do you know how I can troubleshoot this problem?

####<Nov 23, 2015 11:59:02 AM CET> <Info> <Health> <aem_ms1> <weblogic.GCMonitor> <<anonymous>> <> <> <1448276342926> <BEA-310002> <14% of the total memory in the server is free.>
####<Nov 23, 2015 12:13:42 PM CET> <Error> <WebLogicServer> <aem_ms1> <[STANDBY] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1448277222014> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "609" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 609916 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
        org.apache.jackrabbit.oak.plugins.segment.Segment.internalReadRecordId(Segment.java:326)
        org.apache.jackrabbit.oak.plugins.segment.Segment.readRecordId(Segment.java:322)
        org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getTemplateId(SegmentNodeState.java:70)
        org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getTemplate(SegmentNodeState.java:79)
        org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getProperty(SegmentNodeState.java:123)
        org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.getProperty(MemoryNodeBuilder.java:464)
        org.apache.jackrabbit.oak.core.SecureNodeBuilder.getProperty(SecureNodeBuilder.java:184)
        org.apache.jackrabbit.oak.plugins.tree.AbstractTree.getProperty(AbstractTree.java:226)
        org.apache.jackrabbit.oak.core.MutableTree.getProperty(MutableTree.java:116)
        org.apache.jackrabbit.oak.query.ast.SelectorImpl.currentOakProperty(SelectorImpl.java:633)
        org.apache.jackrabbit.oak.query.ast.SelectorImpl.currentOakProperty(SelectorImpl.java:604)
        org.apache.jackrabbit.oak.query.ast.SelectorImpl.currentOakProperty(SelectorImpl.java:537)
        org.apache.jackrabbit.oak.query.ast.SelectorImpl.currentProperty(SelectorImpl.java:513)
        org.apache.jackrabbit.oak.query.ast.PropertyValueImpl.currentProperty(PropertyValueImpl.java:106)
        org.apache.jackrabbit.oak.query.ast.ComparisonImpl.evaluate(ComparisonImpl.java:88)
        org.apache.jackrabbit.oak.query.ast.AndImpl.evaluate(AndImpl.java:123)
        org.apache.jackrabbit.oak.query.ast.SelectorImpl.next(SelectorImpl.java:431)
        org.apache.jackrabbit.oak.query.QueryImpl$RowIterator.fetchNext(QueryImpl.java:659)
        org.apache.jackrabbit.oak.query.QueryImpl$RowIterator.hasNext(QueryImpl.java:679)
        com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
        org.apache.jackrabbit.oak.query.FilterIterators$DistinctIterator.fetchNext(FilterIterators.java:137)
        org.apache.jackrabbit.oak.query.FilterIterators$DistinctIterator.hasNext(FilterIterators.java:151)
        org.apache.jackrabbit.oak.jcr.query.QueryResultImpl$3.fetch(QueryResultImpl.java:181)
        org.apache.jackrabbit.oak.jcr.query.QueryResultImpl$3.next(QueryResultImpl.java:207)
        org.apache.jackrabbit.oak.jcr.query.QueryResultImpl$3.next(QueryResultImpl.java:170)
        org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate$SynchronizedIterator.next(SessionDelegate.java:697)
        org.apache.jackrabbit.oak.jcr.query.PrefetchIterator.next(PrefetchIterator.java:97)
        org.apache.jackrabbit.commons.iterator.RangeIteratorAdapter.next(RangeIteratorAdapter.java:152)
        org.apache.jackrabbit.commons.iterator.RangeIteratorDecorator.next(RangeIteratorDecorator.java:92)
        org.apache.jackrabbit.commons.iterator.NodeIteratorAdapter.nextNode(NodeIteratorAdapter.java:80)
        com.day.cq.wcm.core.impl.VersionManagerImpl.getTreePurgeList(VersionManagerImpl.java:549)
        com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:442)
        org.apache.jsp.libs.wcm.core.components.versionpurge.POST_jsp$Processor.internalProcess(POST_jsp.java:154)
        org.apache.jsp.libs.wcm.core.components.versionpurge.POST_jsp$Processor.process(POST_jsp.java:130)
        org.apache.jsp.libs.wcm.core.components.versionpurge.POST_jsp._jspService(POST_jsp.java:409)

GC overhead limit exceeded

Cannot serve request to /etc/versioning/purge.html in /libs/wcm/core/components/versionpurge/POST.jsp

Exception:

java.lang.OutOfMemoryError: GC overhead limit exceeded
    at sun.nio.cs.UTF_8.newDecoder(UTF_8.java:68)
    at java.lang.StringCoding.decode(StringCoding.java:213)
    at java.lang.String.<init>(String.java:451)
    at java.lang.String.<init>(String.java:503)
    at org.apache.jackrabbit.oak.plugins.segment.Segment.loadString(Segment.java:352)
    at org.apache.jackrabbit.oak.plugins.segment.Segment.readString(Segment.java:338)
    at org.apache.jackrabbit.oak.plugins.segment.Segment.readString(Segment.java:332)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.getValue(SegmentPropertyState.java:167)
    at org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.getValue(SegmentPropertyState.java:137)
    at org.apache.jackrabbit.oak.spi.query.PropertyStateValue.getValue(PropertyStateValue.java:61)

1 Accepted Solution

Avatar

Correct answer by
Employee

How much memory have you allocated?[0]

Regards,

Opkar

[0]https://docs.adobe.com/docs/en/aem/6-0/deploy/application-server-install.html#Oracle WebLogic 10.3.5 / 10.3.6 / 12.1

View solution in original post

3 Replies

Avatar

Correct answer by
Employee

How much memory have you allocated?[0]

Regards,

Opkar

[0]https://docs.adobe.com/docs/en/aem/6-0/deploy/application-server-install.html#Oracle WebLogic 10.3.5 / 10.3.6 / 12.1

Avatar

Level 4

-Xms5120m

-Xmx5120m

-XX:MaxDirectMemorySize=5120m

-XX:PermSize=512m

-XX:MaxPermSize=512m 

Total memory is 12G.

After re-executing the version purge I have no OutOfMemory anymore but I have an infinite loop when purging home page versions...It just cycle on the same N versions, so WebLogic reports that thread as STUCK.

Avatar

Level 4

This is the STUCK thread:

####<Nov 23, 2015 2:51:57 PM CET> <Error> <WebLogicServer> <aem_ms1> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1448286717050> <BEA-000337> <[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "685" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 685521 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
        org.apache.jackrabbit.oak.plugins.segment.file.TarReader.findEntry(TarReader.java:585)
        org.apache.jackrabbit.oak.plugins.segment.file.TarReader.readEntry(TarReader.java:537)
        org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:822)
        org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:134)
        org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:101)
        org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
        org.apache.jackrabbit.oak.plugins.segment.MapRecord.size(MapRecord.java:128)
        org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:461)
        org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:459)
        org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:459)
        org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMap(SegmentWriter.java:711)
        org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1089)
        org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1091)
        org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1072)
        org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:396)
        org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1063)
        org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1072)
        org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:396)
        org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1063)
        org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:91)
        org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:26)
        org.apache.jackrabbit.oak.spi.commit.EditorHook.processCommit(EditorHook.java:57)
        org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:60)
        org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:60)
        org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:352)
        org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:383)
        org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:439)
        org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore.merge(SegmentNodeStore.java:151)
        org.apache.jackrabbit.oak.spi.state.ProxyNodeStore.merge(ProxyNodeStore.java:42)
        org.apache.jackrabbit.oak.core.MutableRoot.commit(MutableRoot.java:247)
        org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:341)
        org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:366)
        org.apache.jackrabbit.oak.jcr.version.ReadWriteVersionManager.removeVersion(ReadWriteVersionManager.java:253)
        org.apache.jackrabbit.oak.jcr.delegate.VersionManagerDelegate.removeVersion(VersionManagerDelegate.java:225)
        org.apache.jackrabbit.oak.jcr.delegate.VersionHistoryDelegate.removeVersion(VersionHistoryDelegate.java:211)
        org.apache.jackrabbit.oak.jcr.version.VersionHistoryImpl$11.performVoid(VersionHistoryImpl.java:240)
        org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performVoid(SessionDelegate.java:268)
        org.apache.jackrabbit.oak.jcr.version.VersionHistoryImpl.removeVersion(VersionHistoryImpl.java:236)
        com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:452)
        org.apache.jsp.libs.wcm.core.components.versionpurge.POST_jsp$Processor.internalProcess(POST_jsp.java:154)
        org.apache.jsp.libs.wcm.core.components.versionpurge.POST_jsp$Processor.process(POST_jsp.java:130)
        org.apache.jsp.libs.wcm.core.components.versionpurge.POST_jsp._jspService(POST_jsp.java:409)

 

Thank you!