FileStore TarMK GC: compaction skipped. Not enough memory

[Tom_Fought]

Team.

Author and publishers are randomly failing TarMK GC failing due to lack of memory since we upgraded to Java11.

Anyone else have this issue? any suggestions?

Thanks for your assistance.

Last FileStore TarMK GC Results
13.12.2024 02:00:00.077 *INFO* [TarMK revision gc [/author/author65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #22: started
13.12.2024 02:00:00.078 *INFO* [TarMK revision gc [/author/author65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #22: estimation started
13.12.2024 02:00:00.156 *INFO* [TarMK revision gc [/author/author65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #22: estimation completed in 78.88 ms (78 ms). Segmentstore size has increased since the last tail garbage collection from 37.6 GB (37552579584 bytes) to 51.0 GB (50996005888 bytes), an increase of 13.4 GB (13443426304 bytes) or 35%. This is greater than sizeDeltaEstimation=1.1 GB (1073741824 bytes), so running garbage collection
13.12.2024 02:00:00.157 *INFO* [TarMK revision gc [/author/author65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #22: setting up a listener to cancel compaction if available memory on pool 'PS Survivor Space' drops below 5.5 MB (5505024 bytes) / 15%.
13.12.2024 02:00:00.157 *WARN* [TarMK revision gc [/author/author65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #22: canceling compaction because available memory level 76.8 kB (76760 bytes) is too low, expecting at least 5.5 MB (5505024 bytes)
13.12.2024 02:00:00.157 *INFO* [TarMK revision gc [/author/author65/crx-quickstart/repository/segmentstore]] org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #22: compaction skipped. Not enough memory

[konstantyn_diachenko]

Hi @Tom_Fought ,

First of all, you need to increase physical memory on your disk. This warning is descriptive:
org.apache.jackrabbit.oak.segment.file.FileStore TarMK GC #22: canceling compaction because available memory level 76.8 kB (76760 bytes) is too low, expecting at least 5.5 MB (5505024 bytes) 

In addition, I would suggest to stop instances and do offline compaction https://experienceleague.adobe.com/en/docs/experience-manager-65/content/implementing/deploying/depl... , then Online revision Cleanup.

Best regards,

Kostiantyn Diachenko.

[Tom_Fought]

Thanks for the suggestions. We have plenty of free disk space. 

I'm thinking this has something to do with how much memory is being allocated with on the java command line in the start script.

My dev/qa machines have 16gb. They were running with -Xmx10gb. I have reduced it to 8gb to see if that has any impact. I tested it manually through the daily maintenance schedule and it seems to have made a difference.

Production machines have 32gb. Reducing -Xmx to 22gb from 26gb.

Time will tell.

[konstantyn_diachenko]

I remember that we added additional JVM parameters to start command for AEM 6.5 on premise on Java 11.

Here is a documentation: https://experienceleague.adobe.com/en/docs/experience-manager-65/content/implementing/deploying/depl...

Java 11 Considerations
If you are running Oracle Java 11 (or generally versions of Java newer than 8), additional switches must be added to your command line when starting AEM.

The following - -add-opens switches need to be added to prevent related reflection access WARNING messages in the stdout.log

--add-opens=java.desktop/com.sun.imageio.plugins.jpeg=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.jrt=ALL-UNNAMED --add-opens=java.naming/javax.naming.spi=ALL-UNNAMED --add-opens=java.xml/com.sun.org.apache.xerces.internal.dom=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED -Dnashorn.args=--no-deprecation-warning

Additionally, you need to use the -XX:+UseParallelGC switch to mitigate any potential performance issues.
Below is a sample of how the additional JVM parameters should look like when starting AEM on Java 11:

-XX:+UseParallelGC --add-opens=java.desktop/com.sun.imageio.plugins.jpeg=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.jrt=ALL-UNNAMED --add-opens=java.naming/javax.naming.spi=ALL-UNNAMED --add-opens=java.xml/com.sun.org.apache.xerces.internal.dom=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED -Dnashorn.args=--no-deprecation-warning

Finally, if you are running an instance upgraded from AEM 6.3, make sure the following property is set to true under sling.properties:felix.bootdelegation.implicit