We are facing "running out of space issue" on Publishers when performing load testing. Applied SP2 for AEM 6 that addresses this, but issue persists. | Community
Skip to main content
smacdonald2008
Level 10
October 16, 2015
Solved

We are facing "running out of space issue" on Publishers when performing load testing. Applied SP2 for AEM 6 that addresses this, but issue persists.

  • October 16, 2015
  • 3 replies
  • 896 views

(This is an issue from an AEM group and provided answer)

This issue is supposedly fixed with SP1 and SP2 is a superset. Have anyone faced this issue when running load tests?

https://rmengji.wordpress.com/2014/09/18/aem-6-0-tarmk-occupying-huge-amount-of-diskspace/

Thanks a lot

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by smacdonald2008

The growth issues have been solved recently, but the fixes are still in final testing. For now, you should apply Oak 1.0.11 hotfix (https://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html). Then as article you linked mentions, you should run offline tar compaction.

1. Disable online compaction by going to CRXDE and setting granite.operations.condition.runmode from "crx3" to "crx3-disabled" on this node "/libs/granite/operations/config/maintenance/granite:daily/granite:RevisionGC"

2. See the Adobe docs for official steps on running offline compaction (see "Offline Compaction via the Oak-run Tool"):

https://docs.adobe.com/docs/en/aem/6-0/deploy/upgrade/microkernels-in-aem-6-0.html#Maintaining%20the%20Repository

Note that the article you linked makes has an error, it says to use rm-all in the checkpoint cleanup. Instead of rm-all you should use the rm-unreferenced instead. This way your system doesn't have to reindex all the async search indexes after run this cleanup.

If the problem persists even after running offline compaction then you should file a support request with adobe [1] as it could be something with the application or your system specifically doing a lot of writes. 
[1] https://helpx.adobe.com/marketing-cloud/contact-support.html

After the final fixes are released in the coming weeks then you would do these steps: 
1. Install SP2 and Oak 1.0.11 provided here:

https://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html 
2. Once they are publicly available then you should will also need to install hotfix NPR-5523 and NPR-59163

3. Revert any changes done to prevent online compaction like reverting back "/libs/granite/operations/config/maintenance/granite:daily/granite:RevisionGC" > property named granite.operations.condition.runmode to "crx3" from "crx3-disabled". The current value should be "crx3".

4. To enable the online compaction, you simply need to add the property "pauseCompaction" with value "false" in the below config.

You can create/use the cfg file to simply add it in the related config file for example: 
crx-quickstart/install/org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.cfg 
pauseCompaction=false

Please Note: The current content may still be containing old checkpoint entries, that you need to clean once using the command: 
java -Xmx2048m oak-run.jar checkpoints {segmentstore-path} rm-unreferenced 
to remove the existing checkpoints (as in the procedure described for the offline compaction). Then after that, the daily online compaction should work as expected.

3 replies

smacdonald2008
smacdonald2008AuthorAccepted solution
Level 10
October 16, 2015

The growth issues have been solved recently, but the fixes are still in final testing. For now, you should apply Oak 1.0.11 hotfix (https://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html). Then as article you linked mentions, you should run offline tar compaction.

1. Disable online compaction by going to CRXDE and setting granite.operations.condition.runmode from "crx3" to "crx3-disabled" on this node "/libs/granite/operations/config/maintenance/granite:daily/granite:RevisionGC"

2. See the Adobe docs for official steps on running offline compaction (see "Offline Compaction via the Oak-run Tool"):

https://docs.adobe.com/docs/en/aem/6-0/deploy/upgrade/microkernels-in-aem-6-0.html#Maintaining%20the%20Repository

Note that the article you linked makes has an error, it says to use rm-all in the checkpoint cleanup. Instead of rm-all you should use the rm-unreferenced instead. This way your system doesn't have to reindex all the async search indexes after run this cleanup.

If the problem persists even after running offline compaction then you should file a support request with adobe [1] as it could be something with the application or your system specifically doing a lot of writes. 
[1] https://helpx.adobe.com/marketing-cloud/contact-support.html

After the final fixes are released in the coming weeks then you would do these steps: 
1. Install SP2 and Oak 1.0.11 provided here:

https://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html 
2. Once they are publicly available then you should will also need to install hotfix NPR-5523 and NPR-59163

3. Revert any changes done to prevent online compaction like reverting back "/libs/granite/operations/config/maintenance/granite:daily/granite:RevisionGC" > property named granite.operations.condition.runmode to "crx3" from "crx3-disabled". The current value should be "crx3".

4. To enable the online compaction, you simply need to add the property "pauseCompaction" with value "false" in the below config.

You can create/use the cfg file to simply add it in the related config file for example: 
crx-quickstart/install/org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.cfg 
pauseCompaction=false

Please Note: The current content may still be containing old checkpoint entries, that you need to clean once using the command: 
java -Xmx2048m oak-run.jar checkpoints {segmentstore-path} rm-unreferenced 
to remove the existing checkpoints (as in the procedure described for the offline compaction). Then after that, the daily online compaction should work as expected.

Kunal_Gaba_
October 16, 2015

Hi Scott, 

Can we rely on online compaction job in AEM 6.1 or it is recommended to use offline compaction as best practice ? We are upgrading from 5.6.1 to 6.1 and would like to know what is recommended by Adobe for dev ops in production environment. 

Thanks,

Kunal

joerghoh
Adobe Employee
Adobe Employee
October 16, 2015

Hi Scott,

just for completeness: please use the latest version of Oak provided by Adobe. At the moment it is 1.0.19 (for AEM 6.0).

Jörg