Rapid data*tar file growth

[Tom_Fought]

Hello Team.

Recently we had a publisher that started experienced rapid data*tar file file growth.

It just happened suddenly. Running with Java11.

Has anyone had this happen to them? and identified the root cause?

There is nothing in the logs that indicates why this happened.

Also, are there any tools that would allow inspecting the data*tar files with aem shutdown?

Thanks.

[Tom_Fought]

Need to add, the repository has valid consistency checki.

[narendiran_ravi · Accepted solution]

Hi Tom,

I assume you are using AEM 6.5 version. Please follow below steps to identify the issue.

• Check if the revision cleanup task is running successfully. Refer to the Adobe documentation on monitoring online revision cleanup.

• Check your AEM publisher's thread dumps to identify any long-running threads that may increase the segment store size. Refer to the Adobe documentation on thread dump analysis.

• The segment store size may increase if there are frequent content updates. Verify if any recent bulk content has been activated to the publisher, such as a new site or product catalog data.

• If your publisher setup has multiple farms, check if the same segment store size issue is observed in other publishers.

[narendiran_ravi]

This is another article which you can check to debug your issue -https://experienceleague.adobe.com/en/docs/experience-cloud-kcs/kbarticles/ka-17496#aif-aem-is-runni... 

[Tethich]

Hi @Tom_Fought 

All the indications from @narendiran_ravi are very good and are first things to go to in such cases. In addition I would add that when I experienced such issues, for me it was because two things:

1. I had long running jobs that were continuously writing data. Maybe you have some custom integrations which pulls data in. You can check them in /var/eventing .
2. Sometimes assets processing for Dynamic Media was also rapidly increasing disk usage for me.

Maybe is not your case, but worth thinking about these two leads as well. Hope you can find your root cause and share it with us.

[Tom_Fought]

Thanks for suggestions Team.

What I have found is we recently turned on the link checker on our production publishers. Along with that, the TarMK GC night job is failing due to insufficient memory for compaction. 

Once I turned off the link checker space usage has returned to normal. 

My current problem is why the servers are generating output like below.

I haven't found a solution for the PS Survivor Space issue. Suggestions?

6.12.2024 02:00:00.074 *INFO* [TarMK revision gc [/publish/publish65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #2: started
16.12.2024 02:00:00.074 *INFO* [TarMK revision gc [/publish/publish65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #2: estimation started
16.12.2024 02:00:00.079 *INFO* [TarMK revision gc [/publish/publish65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #2: estimation completed in 4.592 ms (4 ms). Segmentstore size has increased since the last tail garbage collection from 5.7 GB (5733877760 bytes) to 11.7 GB (11650875904 bytes), an increase of 5.9 GB (5916998144 bytes) or 103%. This is greater than sizeDeltaEstimation=1.1 GB (1073741824 bytes), so running garbage collection
16.12.2024 02:00:00.079 *INFO* [TarMK revision gc [/publish/publish65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #2: setting up a listener to cancel compaction if available memory on pool 'PS Survivor Space' drops below 5.6 MB (5583667 bytes) / 15%.
16.12.2024 02:00:00.080 *WARN* [TarMK revision gc [/publish/publish65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #2: canceling compaction because available memory level 117.5 kB (117480 bytes) is too low, expecting at least 5.6 MB (5583667 bytes)
16.12.2024 02:00:00.080 *INFO* [TarMK revision gc [/publish/publish65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #2: compaction skipped. Not enough memory

[narendiran_ravi]

Analyze Heap Dump: To identify memory issues, capture and analyze the heap dump using tools like Eclipse MAT 
Increase Heap Size: If necessary, increase the JVM heap size based on the analysis.
Verify Disk Space: Ensure the disk is at least 1.5 times the size of the repository to support TarMK compaction and other operations.