Expand my Community achievements bar.

Dive into Adobe Summit 2024! Explore curated list of AEM sessions & labs, register, connect with experts, ask questions, engage, and share insights. Don't miss the excitement.

High load due to ContentMirrorStoreStrategy?

Avatar

Level 5

We are experiencing extremely high load on our AEM Author (6.1, Oak 1.2.7) Our logs get filled by the following ContentMirrorStoreStrategy messages. I am not sure what they mean and if they are the cause of the load, any help would be greatly appreciated. NOTE: I only get this log for our 'Publisher A'

During this phenomenon we are unable to rollout any pages to their livecopy.

Extra info about our setup:

  • 1 author
  • 3 publishers (A, B, C)
  • 8+ sites

01.08.2017 09:53:54.630 *WARN* [10.1.106.97 [1501581233267] GET /libs/wcm/core/content/pageinfo.json HTTP/1.1] org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy Traversed 70000 nodes (678 index entries) using index event.job.topic with

filter Filter(query=select [jcr:path], [jcr:score], * from [slingevent:Job] as a where [event.job.topic] = 'com/day/cq/replication/job/publisha' and [slingevent:finishedState] is null order by [slingevent:created] /* xpath: //element(*,slingevent:Job)[@event.job.topic = 'com/day/cq/replication/job/publisha'

and not(@slingevent:finishedState)] order by @slingevent:created ascending */, path=*, property=[slingevent:finishedState=[is null], event.job.topic=[com/day/cq/replication/job/publisha]])

Greetings

Jeroen

12 Replies

Avatar

Level 10

We did a weinar with top community members that discussed (and with performance in mind)--

This session of Ask the AEM Community Experts will share large scale architectures from the author's experiences with various companies like Cisco and Symantec and compare and contrast the architecture across: Infrastructure Architecture Scaling

Ecommerce integrations and migration approach from legacy into Adobe Experience Manager, Digital Marketing Cloud Integrations such as personalization, analytics, and DMP.

This may help you -- Scott's Digital Community: Comparative Architecture Analysis of large scale Experience Manager Insta...

Avatar

Employee Advisor

Hi,

this is caused by an inperformant index; in your case I would

1) update to a later version of Oak (which comes with some improvements also in terms of performance)

2) raise a Daycare ticket and ask if there are recommendations how you can tune your indexes.

Jörg

Avatar

Employee

Can you install CFP1 or later and retest?

Avatar

Level 5

I installed Cumulative Fix Pack 10 pack but I am now seeing the following error messages in the error.log:

02.08.2017 13:48:53.146 *ERROR* [IdMapService-initializer] org.apache.sling.discovery.commons.providers.spi.base.IdMapService initializer: could not init due to java.lang.NullPointerException

java.lang.NullPointerException: null

at org.apache.sling.discovery.commons.providers.spi.base.IdMapService.init(IdMapService.java:187)

at org.apache.sling.discovery.commons.providers.spi.base.IdMapService.access$000(IdMapService.java:59)

at org.apache.sling.discovery.commons.providers.spi.base.IdMapService$1.check(IdMapService.java:108)

at org.apache.sling.discovery.commons.providers.spi.base.AbstractServiceWithBackgroundCheck$BackgroundCheckRunnable.run(AbstractServiceWithBackgroundCheck.java:77)

at java.lang.Thread.run(Thread.java:745)

I found this ticket: [SLING-5592] rapid IdMapService activation/deactivation leaves background thread in limbo - ASF JIRA , it seems to be fixed in version 1.0.12. AEM 6.1 seems to be using 1.0.4:

Screen Shot 2017-08-02 at 13.50.52.png

Is there a way to fix this?

Avatar

Employee

Hi,

I would recommend you to log daycare tickets for both of these issues and get official support. I think, this needs investigation and analysis on your setup not possible for forum discussion.

Regards,

Varun

Avatar

Level 4

Hi Varun,

We have made this property index change to lucene index change.  You mention a new element to remove slingevent:job from the /oak:index/nodetype declaringNodeTypes list.  Can you please explain a little further?  ie. does the lucene index replace the declaringNodeType entry?  Is it noticeable or harmful in anyway to NOT remove the declaringNodeType when the property index is disabled?  Does it reduce the nodetype index size (by not having)?

Thanks in advance for any further information on this...

Avatar

Employee

Use Lucene Sling Event index instead of using event.job.topic property index. Event.job.topic property index is very slow.

It seems that you still have event.job.topic property index in place. Your query is traversing even when it uses event.job.topic index since it has way too many entries. Property indexes are not very performant if you are indexing a lot of nodes.

When you have slingeventJob lucene index, then you would need to remove that property slingevent:job from nodetype index.

You would have slingevent:job as property in declared nodetype index. Also, you would have event.job.topic property index in place.

The recommendation from engineering has been to convert it to lucene based index definition.

So, you could have a lucene based index for it which should help the cause.

<slingeventJob

            jcr:primaryType="oak:QueryIndexDefinition"

            async="async"

            compatVersion="{Long}2"

            name="slingevent:Job"

            reindex="{Boolean}true"

            indexPath="/oak:index/slingeventJob"

            includedPaths="[/var/eventing]"

            type="lucene">

        <indexRules jcr:primaryType="nt:unstructured">

            <slingevent:Job jcr:primaryType="nt:unstructured">

                <properties jcr:primaryType="nt:unstructured">

                    <eventJobTopic

                            jcr:primaryType="nt:unstructured"

                            name="event.job.topic"

                            ordered="{Boolean}false"

                            propertyIndex="{Boolean}true"

                            type="String"/>

                    <slingeventCreated

                            jcr:primaryType="nt:unstructured"

                            name="slingevent:created"

                            ordered="{Boolean}true"

                            propertyIndex="{Boolean}true"

                            type="Date"/>

                    <finishedState

                            jcr:primaryType="nt:unstructured"

                            name="slingevent:finishedState"

                            ordered="{Boolean}false"

                            propertyIndex="{Boolean}true"

                            nullCheckEnabled="{Boolean}true"

                            type="String"/>

                </properties>

            </slingevent:Job>

        </indexRules>

    </slingeventJob>

and once this gets reindexed

Then you can add a property to the ootb property index event.job.topic by performing below:

Add below property to event.job.topic index so that it doesn't get picked

b) entryCount

Long

99999999

Further, also remove the slingevent:Job from the declaringNodetypes property in nodetype index

"nodetype": {

    "reindexCount": 1,

    "reindex": false,

    "nodeTypeListDefined": true,

    "type": "property",

    "propertyNames": [

      "jcr:primaryType",

      "jcr:mixinTypes"

    ],

    "jcr:primaryType": "oak:QueryIndexDefinition",

    "declaringNodeTypes": [

      "cq:PollConfigFolder",

      "sling:MessageEntry",

      "cq:ExporterConfigFolder",

      "cq:BlueprintSyncConfig",

      "cq:ClientLibraryFolder",

      "granite:Task",

      "cq:Template",

      "slingevent:Job",

      "granite:CloudsettingsConfigType",

      "oak:QueryIndexDefinition",

      "rep:User",

      "rep:Authorizable",

      "cq:PollConfig",

      "sling:bgJobData",

      "cq:ExporterConfig",

      "sling:VanityPath",

      "sling:chunks",

      "mix:language",

      "cq:Component",

      "cq:Console",

      "slingevent:TimedEvent",

      "cq:ContentSyncConfig"

    ]

Avatar

Level 5

I am going to try this, really hoping this will work vmehrotr​, you said 'and once this gets reindexed', how do I know it is reindexed?

Avatar

Employee

Once you create the new index, you would see message like below in logs.

*INFO* [pool-9-thread-1] org.apache.jackrabbit.oak.plugins.index.IndexUpdate

Reindexing would be performed for following indexes [/oak:index/slingeventJob]

..

*INFO* [pool-9-thread-1] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate

Reindexing completed for indexes: [/oak:index/slingeventJob] in x.xxxx h

Avatar

Employee Advisor

Hi,

please be advised, that this process can take some hours (maybe even days)! So this is not an operation you should do without testing.

I also advised you to ask Daycare support regarding this, because this index definition is not the only one which you should replace. Support has a number of recommendations regarding it.

Jörg