Expand my Community achievements bar.

Learn about Edge Delivery Services in upcoming GEM session
SOLVED

We are facing "running out of space issue" on Publishers when performing load testing. Applied SP2 for AEM 6 that addresses this, but issue persists.

Avatar

Level 10

(This is an issue from an AEM group and provided answer)

This issue is supposedly fixed with SP1 and SP2 is a superset. Have anyone faced this issue when running load tests?

https://rmengji.wordpress.com/2014/09/18/aem-6-0-tarmk-occupying-huge-amount-of-diskspace/

Thanks a lot

1 Accepted Solution

Avatar

Correct answer by
Level 10

The growth issues have been solved recently, but the fixes are still in final testing. For now, you should apply Oak 1.0.11 hotfix (https://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html). Then as article you linked mentions, you should run offline tar compaction.

1. Disable online compaction by going to CRXDE and setting granite.operations.condition.runmode from "crx3" to "crx3-disabled" on this node "/libs/granite/operations/config/maintenance/granite:daily/granite:RevisionGC"

2. See the Adobe docs for official steps on running offline compaction (see "Offline Compaction via the Oak-run Tool"):

https://docs.adobe.com/docs/en/aem/6-0/deploy/upgrade/microkernels-in-aem-6-0.html#Maintaining%20the...

Note that the article you linked makes has an error, it says to use rm-all in the checkpoint cleanup. Instead of rm-all you should use the rm-unreferenced instead. This way your system doesn't have to reindex all the async search indexes after run this cleanup.

If the problem persists even after running offline compaction then you should file a support request with adobe [1] as it could be something with the application or your system specifically doing a lot of writes. 
[1] https://helpx.adobe.com/marketing-cloud/contact-support.html

After the final fixes are released in the coming weeks then you would do these steps: 
1. Install SP2 and Oak 1.0.11 provided here:

https://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html 
2. Once they are publicly available then you should will also need to install hotfix NPR-5523 and NPR-59163

3. Revert any changes done to prevent online compaction like reverting back "/libs/granite/operations/config/maintenance/granite:daily/granite:RevisionGC" > property named granite.operations.condition.runmode to "crx3" from "crx3-disabled". The current value should be "crx3".

4. To enable the online compaction, you simply need to add the property "pauseCompaction" with value "false" in the below config.

You can create/use the cfg file to simply add it in the related config file for example: 
crx-quickstart/install/org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.cfg 
pauseCompaction=false

Please Note: The current content may still be containing old checkpoint entries, that you need to clean once using the command: 
java -Xmx2048m oak-run.jar checkpoints {segmentstore-path} rm-unreferenced 
to remove the existing checkpoints (as in the procedure described for the offline compaction). Then after that, the daily online compaction should work as expected.

View solution in original post

3 Replies

Avatar

Correct answer by
Level 10

The growth issues have been solved recently, but the fixes are still in final testing. For now, you should apply Oak 1.0.11 hotfix (https://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html). Then as article you linked mentions, you should run offline tar compaction.

1. Disable online compaction by going to CRXDE and setting granite.operations.condition.runmode from "crx3" to "crx3-disabled" on this node "/libs/granite/operations/config/maintenance/granite:daily/granite:RevisionGC"

2. See the Adobe docs for official steps on running offline compaction (see "Offline Compaction via the Oak-run Tool"):

https://docs.adobe.com/docs/en/aem/6-0/deploy/upgrade/microkernels-in-aem-6-0.html#Maintaining%20the...

Note that the article you linked makes has an error, it says to use rm-all in the checkpoint cleanup. Instead of rm-all you should use the rm-unreferenced instead. This way your system doesn't have to reindex all the async search indexes after run this cleanup.

If the problem persists even after running offline compaction then you should file a support request with adobe [1] as it could be something with the application or your system specifically doing a lot of writes. 
[1] https://helpx.adobe.com/marketing-cloud/contact-support.html

After the final fixes are released in the coming weeks then you would do these steps: 
1. Install SP2 and Oak 1.0.11 provided here:

https://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html 
2. Once they are publicly available then you should will also need to install hotfix NPR-5523 and NPR-59163

3. Revert any changes done to prevent online compaction like reverting back "/libs/granite/operations/config/maintenance/granite:daily/granite:RevisionGC" > property named granite.operations.condition.runmode to "crx3" from "crx3-disabled". The current value should be "crx3".

4. To enable the online compaction, you simply need to add the property "pauseCompaction" with value "false" in the below config.

You can create/use the cfg file to simply add it in the related config file for example: 
crx-quickstart/install/org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.cfg 
pauseCompaction=false

Please Note: The current content may still be containing old checkpoint entries, that you need to clean once using the command: 
java -Xmx2048m oak-run.jar checkpoints {segmentstore-path} rm-unreferenced 
to remove the existing checkpoints (as in the procedure described for the offline compaction). Then after that, the daily online compaction should work as expected.

Avatar

Employee Advisor

Hi Scott, 

Can we rely on online compaction job in AEM 6.1 or it is recommended to use offline compaction as best practice ? We are upgrading from 5.6.1 to 6.1 and would like to know what is recommended by Adobe for dev ops in production environment. 

Thanks,

Kunal

Avatar

Employee Advisor

Hi Scott,

just for completeness: please use the latest version of Oak provided by Adobe. At the moment it is 1.0.19 (for AEM 6.0).

Jörg