Lots of content - AEM Author performance issues

Avatar

Avatar

surenk

Avatar

surenk

surenk

21-10-2019

One of our customer has migrated a lot of content (assets and pages)... they have about 1.5 million assets and about 4 million site pages. Part of the sites are generated by XML Add-on.

Problem:

With this much load obviously Author (AEM 6.4) is slow for authoring. And sometimes crashes.

Solutions:

  • The server where AEM author is running has been upgraded to have a lot of resources (94 core CPU and 384GB memory).. this maybe not be best solution but did not help a lot.
  • Going through all the AEM systems configuration (workflows, any orphan process, etc.)
    • Using ACS Commons' Bulk Workflow tool.
  • Review an Elastic search index configured that runs weekly
  • Following offline-compaction and other suggested per the AEM 6.x Maintenance Guide

Usually what can be done with repository of such huge sizes?

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Avatar

Jörg_Hoh

Employee

Total Posts

3.1K

Likes

1.1K

Correct Reply

1.1K

Avatar

Jörg_Hoh

Employee

Total Posts

3.1K

Likes

1.1K

Correct Reply

1.1K
Jörg_Hoh
Employee

30-10-2019

Hi Surenk,

would you mind to post a small summary about how you were able to solve your issue after you completed the work with support? I am quite sure others are interested as well.

Thanks,

Jörg

Answers (9)

Answers (9)

Avatar

Avatar

surenk

Avatar

surenk

surenk

30-10-2019

We are working with them as well. There are some known issues!!

Avatar

Avatar

Jörg_Hoh

Employee

Total Posts

3.1K

Likes

1.1K

Correct Reply

1.1K

Avatar

Jörg_Hoh

Employee

Total Posts

3.1K

Likes

1.1K

Correct Reply

1.1K
Jörg_Hoh
Employee

26-10-2019

Ah, XML addon. But unfortunately I cannot help you with it.

In any way, I would get in contact with Adobe support and tell them about your case. And ask them for suggestions to help you here.

Avatar

Avatar

surenk

Avatar

surenk

surenk

23-10-2019

Thanks for chiming in Jorg.

The answer to most questions is "XML AddOn"

  1. From a handling point-of-view, does it make sense to have them all in AEM? [ Authors are copying the dita content into Assets from external source and after editing the content, generating the output. The generated output goes under "/sites" which is then published to Publisher. ]
  2. How are your authors interacting them? Are they at all? [ If there are any updates.modifications to original dita content, they can go back to the assets and re-generate the output/sites-pages. ]
  3. Or are just automatic processes working with them all the time (regenerating, publishing, deactivating)? [ Manual ]

So this process (along with some workflows that run by the xml add-on) is creating a lot of nodes/content on Author and is bloating it up. Maybe it's more of a process that needs to be updated than any issues on AEM

Avatar

Avatar

Jörg_Hoh

Employee

Total Posts

3.1K

Likes

1.1K

Correct Reply

1.1K

Avatar

Jörg_Hoh

Employee

Total Posts

3.1K

Likes

1.1K

Correct Reply

1.1K
Jörg_Hoh
Employee

22-10-2019

Hi,

I wonder what value comes with lots of pages, which are auto-generated and probably also auto-maintained in AEM. From a handling point-of-view, does it make sense to have them all in AEM? How are your authors interacting them? Are they at all? Or are just automatic processes working with them all the time (regenerating, publishing, deactivating)?

Same with assets.

If you cannot answer these questions, I would try to find a way to maintain all this content outside of AEM. If you manage this content only by automation, what's the value of having them in AEM?

Avatar

Avatar

sunjot16

Employee

Avatar

sunjot16

Employee

sunjot16
Employee

22-10-2019

Perform Revision Cleanups, Version Purge, Workflow Purge, etc.

The following could also help:

Monitoring and Maintaining Your AEM instance

AEM 6.x Maintenance Guide

Avatar

Avatar

surenk

Avatar

surenk

surenk

22-10-2019

Forgot to mention.. the data is on external AWS S3 drive and referred by AEM.

Avatar

Avatar

Santhi_Swaroop

Avatar

Santhi_Swaroop

Santhi_Swaroop

22-10-2019

Try to shard the content/sites to two different authors and publish to same publish instances so that you can minimize the load by 50%

Avatar

Avatar

rachanam1547401

Avatar

rachanam1547401

rachanam1547401

22-10-2019

Hi,

I think your AEM instance might not be tuned properly. I have written short summary of "Sling Memory Deep Dive" here https://medium.com/the-telegraph-engineering/four-highlights-from-adaptto-2018-3781782a6b7a   and a detailed session video available here adaptTo() 2018   I found it very interesting in terms of memory usage, you may want to check it.

Running compaction will obviously help and 6.4 does online compaction already so make sure it's happening.

Creating indexes based on your queries will help too.

Avatar

Avatar

BrijeshYadav

Avatar

BrijeshYadav

BrijeshYadav

22-10-2019

It's really a very huge data.

I have few things in mind which could be help full?

  1. External data store: When dealing with large number of binaries, it is recommended that an external data store be used instead of the default node stores in order to maximize performance  Configuring node stores and data stores in AEM 6
  2. Query Optimazation: Use http://localhost:4502/libs/granite/operations/content/diagnosistools/requestPerformance.html  to anaylize what are the most time taking queries which might cause of aem performance.
  3. Indexing: Check error log and search "consider creating an index or changing the query", which can give you idea if any missing index. Oak Queries and Indexing