Impact on Dispatcher cache when 10k pages are deleted

[ashish_mishra1]

Hi,
As part of optimizing authoring activity , business is looking to deactivate and delete 10k pages on prod author using groovy script.
We think this might impact cache flush on dispatcher.
As 10k cache flush request will be created.
Is there any suggestion to better handle this part ?
Thanks,

[TarunKumar]

Hi @ashish_mishra1 ,

You can try to implement a Custom Flushing Strategy. For a large volume of deactivations, avoid a full cache flush.
Instead, use the CQ:Path header in the replication agent's configuration to specify the exact paths that need to be invalidated.
This targets only the affected content and minimizes impact on the rest of the cache. 

Adjust statfileslevel:Configure the /statfileslevel property in the dispatcher.any file.
This setting determines the number of parent nodes for which .stat files are checked.
A lower level reduces the impact of flush requests on the dispatcher by ensuring that only necessary files are invalidated.

Hope it helps!

-Tarun

[giuseppebag · Accepted solution]

You could split the Operation into Batches: Instead of processing all 10,000 pages at once, break the operation into smaller batches (e.g., 100 or 500 pages per batch): it reduces the number of simultaneous cache flush requests and minimizes the risk of overwhelming the dispatcher or author instance. Besides you can add delays, ie introducing a small delay between batches to allow the system to process cache flushes and other background tasks.

Points of attention:

• It is crucial to run the script during off-peak hours to minimise the impact on end-users and reduce the load on the dispatcher.
• Keep an eye on system performance (CPU, memory, response times) during the script execution. If the system shows signs of stress, pause the script and resume later.
• Before running the script in production, test it in a staging environment with a similar number of pages. This helps you gauge the impact and fine-tune batch sizes and delays.

[arunpatidar]

I agree with @giuseppebag 
I would block the end user access to those pages first from CDN and then delete the pages in batches after business hours.

[ashish_mishra1]

Thanks, @Shashi_Mulugu @arunpatidar @Rohan_Garg @TarunKumar @giuseppebag @Jörg_Hoh 

Thanks for your replies, this monday we did tried to delete in 1k batches, but we could see the AEM Author CPU usage was quite high > 60% , and it took us 1 hour to delete 1k pages, which is quite slow. in comparison to stage where we tested 2k page deletion in 10 minutes. 

One of the cause we think is the number of invalidation request going to publishers 1 and 2 and it slowing down the AEM author processing. This we performed in off business hours with no business activity.

Code used -

                        def deactivated = false
                        try {
                            // First Deactivate the page
                            replicator.replicate(session, ReplicationActionType.DEACTIVATE, page.path)
                            deactivated = true
                            println("Page Deactivated :: " + page.path)

                        } catch(Exception e) {
                            println("Failed to deactivate: " + page.path + ", error: " + e.message)
                        }
                        if (deactivated) {
                            // Then Delete the page if deactivation was successful
                            pageManager.delete(page, false)
                            deletedPages++
                            println("Page Deleted :: " + page.path)

                            if(deletedPages % 1024 == 0) {
                                println("Committing the changes at deleted count :: "+ deletedPages)
                                session.save()
                            }
                        }else{
                            println("Skipping deletion as deactivation failed for :: " + page.path)
                        }

Now, here we were using the PageManager API and then deleting it.

Any idea what going wrong? 

Other option we could think of using JCR API to delete the page as Node object. This will not trigger the replication action, but we have to run it 1 author, 2 publishers separately and then flush cache.

we were wondering will there be any adverse impact of this on the AEM author internal references or indexes ?

I have raised this with Adobe support too, and waiting for their reply 

[giuseppebag]

You might want to consider using the ACS AEM Commons Fast Action Manager, which allows for parallel execution and intelligent throttling of Synthetic Workflows and Replication. For instance, you could first deactivate a page and then delete it through the workflow. 

Additionally, you can configure the OSGI service with your preferred CPU, heap size, and number of threads, providing you with plenty of options for optimization.

[Rohan_Garg]

Hi @ashish_mishra1,
 

Divide and conquer - Break the pages into batches (say 500) and schedule them during the off hours.
This should help in spreading out the flush events at Dispatcher.

Besides the stat file strategy mentioned by @TarunKumar, you can also use CDN purge if you have a CDN (Akamai, CloudFront etc.) you can flush cache at folder level. Targeting high level purges which would be handful would be better than having 10k small invalidations.
Are you having OOTB Fastly in AEM Cloud or some other CDN in your setup?

Hope this helps!

Best Regards,

Rohan Garg

[Shashi_Mulugu]

@ashish_mishra1 it all depends on your current statfilelevel settings and at what levels your 10k pages are.

Please take others suggestions into considerations like deactivating during off business hours,  stop live traffic to those pages first, deactivating in batches etc

My two cents:

If the impact of flushing is huge, we can do two things 

1. Configure a agent user ID in publisher dispatcher agent, and add a deny read  permissions for those 10k pages or if they are under some high level roots at those root levels, so that when you activate/deactivate those pages dispatcher agent wont pickup those events.

2. You can do similar in your dispatcher configuration,  publish farm cache in validation rules on which paths or extensions to be allowed to get flushed. This is one step delay stop than #1 where 10k requests comes to dispatcher but dispatcher wont act as per rules.

I would go with #1.