Experience Manager 6.0 - Oak TarMK uses too much disk space

[Community_Admin]

If you are using Oak (CRX3) TarMK with your Adobe Experience Manager 6.0 instance, you may have noticed your disk space filling up disproportionately to the amount of content and assets in the repository.

This is a known issue, and has now been fixed with Service Pack 1 for AEM 6.0. (Service Pack 1 includes and supersedes Hotfix 4642, - which includes and supersedes Hotfix 4135 - which also resolves this issue.)
You can download the service pack from Package Share.

This and any other hotfixes for Adobe Experience Manager 6.0 will be listed here: http://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html

[DarrenBiz]

Community Administrator wrote...

If you are using Oak (CRX3) TarMK with your Adobe Experience Manager 6.0 instance, you may have noticed your disk space filling up disproportionately to the amount of content and assets in the repository.

This is a known issue, and has now been fixed with Hotfix 4135.

I just wanted to highlight that the issue hasn't been fixed with Hotfix 4135 and that it is an ongoing (open) issue with no current resolution other than to manually compact the TarMK repository in an offline state.

[Former Community Member]

Hotfix 4135 makes the things a bit less worse. We have set a test environment, and at first, the repository size almost doubled each day. Then we reinstalled and applied the fix, but it still increases a bit every day (no one is working on that instance of AEM6). After trying startRevisionGC() it went from 2.63G to 2.75G, increasing size insted of decreasing.

Any idea when will it be finally fixed?

[Sham_HC]

Micro-BIOS wrote...

Hotfix 4135 makes the things a bit less worse. We have set a test environment, and at first, the repository size almost doubled each day. Then we reinstalled and applied the fix, but it still increases a bit every day (no one is working on that instance of AEM6). After trying startRevisionGC() it went from 2.63G to 2.75G, increasing size insted of decreasing.

Any idea when will it be finally fixed?

We are working on it & internal references are GRANITE-6593, GRANITE-6493. Update as soon as available.  Make use official support (Daycare) if impacting your project timelines.

[DarrenBiz]

Sham HC wrote...

Micro-BIOS wrote...

Hotfix 4135 makes the things a bit less worse. We have set a test environment, and at first, the repository size almost doubled each day. Then we reinstalled and applied the fix, but it still increases a bit every day (no one is working on that instance of AEM6). After trying startRevisionGC() it went from 2.63G to 2.75G, increasing size insted of decreasing.

Any idea when will it be finally fixed?

We are working on it & internal references are GRANITE-6593, GRANITE-6493. Update as soon as available.  Make use official support (Daycare) if impacting your project timelines.

Hi Sham - can you confirm that GRANITE-6593 and 6493 have been resolved in AEM 6 SP1? I dont see any mention of them in the Release Notes: http://docs.adobe.com/docs/en/aem/6-0/release-notes-sp1.html

[Sham_HC]

DarrenBiz wrote...

Sham HC wrote...

Micro-BIOS wrote...

Hi Sham - can you confirm that GRANITE-6593 and 6493 have been resolved in AEM 6 SP1? I dont see any mention of them in the Release Notes: http://docs.adobe.com/docs/en/aem/6-0/release-notes-sp1.html

Yes it is fixed. Part of subtask of some of bugs mentioned at "Repository improvements"

[whcrow]

I installed this hotfix on a newly installed AEM 6.0 instance and now I get an error (HTTP ERROR: 503) on every request.

16.07.2014 18:18:37.921 *ERROR* [qtp636026460-36] org.apache.sling.engine.impl.SlingHttpContext handleSecurity: AuthenticationSupport service missing. Cannot authenticate request. 16.07.2014 18:18:37.921 *ERROR* [qtp636026460-36] org.apache.sling.engine.impl.SlingHttpContext handleSecurity: Possible reason is missing Repository service. Check AuthenticationSupport dependencies.

I did not install any other packages in order to keep the test clean.

P.S. Windows 8.1 Pro, Firefox 30

[Sam205505050]

Can you check all of your bundle is active or not?

[whcrow]

Bundle information: 392 bundles in total, 383 bundles active, 8 active fragments, 1 bundles resolved

The Oak content repository was resolved.

I manually started this bundle. The issue was resolved after that.

[Sam205505050]

mark it as resolve!

[ClintLundmark]

Installed AEM 6.0 a few weeks ago in a non-prod environment and it grew rapidly up to about 25 GB.  It is an out-of-the-box install with only Geometrixx sites.  After installing this hotfix the rapid growth has stopped, but it does not seem to be reducing the number of .tar files and the space used.  I can't find the equivalent of a CRX 2 Tar Optimization to start manually or one that is running scheduled.  Does OAK TarMK have something similar?  How do I get the .tar files reduced to a normal size after installing this hotfix?

[konradw58070066]

In theory that can be triggered with the revision GC (either from /system/console/jmx/org.apache.jackrabbit.oak%3Aid%3D6%2Cname%3D%22repository+manager%22%2Ctype%3D%22RepositoryManagement%22 or via /libs/granite/operations/content/maintenance/window.html/mnt/overlay/granite/operations/config/maintenance/_granite_daily). Please also compare with http://jackrabbit.apache.org/oak/docs/nodestore/segmentmk.html. Unfortunately even in the last version of the hotfix there seems to be a bug, as each call basically doubles the nodestore in size. Maybe related to https://issues.apache.org/jira/browse/OAK-2001.

[Ondrej_Semotan]

We have problem with DataStoreGarbage Collection and all "utilities" from  http://localhost:4502/system/console/jmx/com.adobe.granite%3Atype%3DRepository.   DataStoreGarbage Collection worked on new installation of CQ5, if i installed hotfixies (4135 and 4562), i have problem: DSGC, tarOptimization and other functions say:  javax.jcr.UnsupportedRepositoryOperationException. Is it OK or where is something wrong?  I cant find anything about this failure.

[ClintLundmark]

Ondrej Semotan wrote...

We have problem with DataStoreGarbage Collection and all "utilities" from  http://localhost:4502/system/console/jmx/com.adobe.granite%3Atype%3DRepository.   DataStoreGarbage Collection worked on new installation of CQ5, if i installed hotfixies (4135 and 4562), i have problem: DSGC, tarOptimization and other functions say:  javax.jcr.UnsupportedRepositoryOperationException. Is it OK or where is something wrong?  I cant find anything about this failure.

Assuming you are using AEM 6.0...  I believe you are running CRX 2 tools.  If AEM is upgraded from a previous version you have a choice to upgrade to Oak (CRX 3) or to stay on CRX 2.   The older CRX 2 tools exist for the situation of AEM 6 and CRX 2.  A new install uses Oak. If using Oak the CRX 2 tools will not work.  It is confusing and it would be nice if CRX 2  OR Oak tools showed up not both.

[DarrenBiz]

Is it possible to know what the fix in 4135 actually addresses?

We have installed this on two of our servers already and also compacted the TarMK manually but am still seeing the TarMK grow until it consumes all available space. We currently have an open support ticket to address this issue as this Hotfix is not resolving the space issue.

@ClintLundmark - there is a runtime OAK Jar file that support can provide to you to manually compact your repository. We are using it weekly to keep the size down, but you have to stop the aem instance first

[konradw58070066]

It depends on the version of 4135 you are using. Version 1.0.2 basically upgrades Oak to 1.0.2. All the fixed issues are mentioned in https://issues.apache.org/jira/browse/OAK/fixforversion/12327115. Apart from that all the Oak 1.0.1 issues are fixed as well: https://issues.apache.org/jira/browse/OAK/fixforversion/12326843.

[ClintLundmark]

Some detailed information about this patch from Adobe and what it is supposed to do would be helpful.  It does not seem to be working quite right or maybe it is not understood.  I have installed the 1.0.2 version of the patch.

Today I started with 22.2 GB and 85 files in repository/segmentstore.  I ran Web Console -> JMX -> org.apache.jackrabbit.oak Repository Management -> startRevisionGC().  It ran for about a minute.  Afterwards 22.2 GB and 86 files.

04.08.2014 12:09:52.140 *INFO* [TarMK compaction thread [/opt/aem/author/crx-quickstart/repository/segmentstore], active since Mon Aug 04 12:09:52 PDT 2014, previous max duration 77342ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK compaction started

04.08.2014 12:10:19.360 *INFO* [TarMK compaction thread [/opt/aem/author/crx-quickstart/repository/segmentstore], active since Mon Aug 04 12:09:52 PDT 2014, previous max duration 77342ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK compaction completed in 27220ms

04.08.2014 12:10:21.922 *INFO* [TarMK flush thread [/opt/aem/author/crx-quickstart/repository/segmentstore], active since Mon Aug 04 12:10:21 PDT 2014, previous max duration 2490ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK revision cleanup started

04.08.2014 12:10:25.987 *INFO* [TarMK flush thread [/opt/aem/author/crx-quickstart/repository/segmentstore], active since Mon Aug 04 12:10:21 PDT 2014, previous max duration 2490ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK revision cleanup reclaiming data00082a.tar

04.08.2014 12:10:27.228 *INFO* [TarMK flush thread [/opt/aem/author/crx-quickstart/repository/segmentstore], active since Mon Aug 04 12:10:21 PDT 2014, previous max duration 2490ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK revision cleanup completed in 5305ms

[DarrenBiz]

I just did a local test on my AEM 6.0 Author instance (hotfix-4499-1.0.2) with the same procedure as @ClintLundmark (Web Console -> JMX -> org.apache.jackrabbit.oak Repository Management -> startRevisionGC())

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvdj             20642428   7388528  12205324  38% /apps

05.08.2014 09:41:46.805 *INFO* [TarMK compaction thread [/apps/aem/author/crx-quickstart/repository/segmentstore], active since Tue Aug 05 09:41:46 EST 2014, previous max duration 223857ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK compaction started
05.08.2014 09:45:43.204 *INFO* [TarMK compaction thread [/apps/aem/author/crx-quickstart/repository/segmentstore], active since Tue Aug 05 09:41:46 EST 2014, previous max duration 223857ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK compaction completed in 236398ms
05.08.2014 09:45:43.589 *INFO* [TarMK flush thread [/apps/aem/author/crx-quickstart/repository/segmentstore], active since Tue Aug 05 09:45:43 EST 2014, previous max duration 3859ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK revision cleanup started
05.08.2014 09:45:45.250 *INFO* [TarMK flush thread [/apps/aem/author/crx-quickstart/repository/segmentstore], active since Tue Aug 05 09:45:43 EST 2014, previous max duration 3859ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK revision cleanup completed in 1661ms

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvdj             20642428   8354660  11239192  43% /apps

That's a 5% (~1GB) increase in the TarMK size by actually running the AEM built-in GC and doing nothing else.

Also saw exactly the same behaviour on Publish server (32% increase to 36% - 4% increase)

Is this expected behaviour for the hotfix?

edit - removed code styling...where is the <pre> tagging gone in this forum?

[Sham_HC]

DarrenBiz wrote...

I just did a local test on my AEM 6.0 Author instance (hotfix-4499-1.0.2) with the same procedure as @ClintLundmark (Web Console -> JMX -> org.apache.jackrabbit.oak Repository Management -> startRevisionGC())

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvdj             20642428   7388528  12205324  38% /apps

05.08.2014 09:41:46.805 *INFO* [TarMK compaction thread [/apps/aem/author/crx-quickstart/repository/segmentstore], active since Tue Aug 05 09:41:46 EST 2014, previous max duration 223857ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK compaction started
05.08.2014 09:45:43.204 *INFO* [TarMK compaction thread [/apps/aem/author/crx-quickstart/repository/segmentstore], active since Tue Aug 05 09:41:46 EST 2014, previous max duration 223857ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK compaction completed in 236398ms
05.08.2014 09:45:43.589 *INFO* [TarMK flush thread [/apps/aem/author/crx-quickstart/repository/segmentstore], active since Tue Aug 05 09:45:43 EST 2014, previous max duration 3859ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK revision cleanup started
05.08.2014 09:45:45.250 *INFO* [TarMK flush thread [/apps/aem/author/crx-quickstart/repository/segmentstore], active since Tue Aug 05 09:45:43 EST 2014, previous max duration 3859ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK revision cleanup completed in 1661ms

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvdj             20642428   8354660  11239192  43% /apps

That's a 5% (~1GB) increase in the TarMK size by actually running the AEM built-in GC and doing nothing else.

Also saw exactly the same behaviour on Publish server (32% increase to 36% - 4% increase)

Is this expected behaviour for the hotfix?

edit - removed code styling...where is the <pre> tagging gone in this forum?

We are working on GRANITE-6493 TarMK gc is not cleaning up tar files with very soon an official release of new Oak-core bundle address this issue. Please stay tuned, if something can't wait based on project timelines please file a daycare.

[gregoryp3536631]

Hi,

Using AEM 6.0 with the SP1 and the segmentStore is still growing up with no reason.

I'm sure that I use the SP1 because I can see "Adobe Experience Manager, Version 6.0.0.SP1" on my welcome page.

What I don't understand is that running the following commands (found here http://docs.adobe.com/docs/en/aem/6-0/deploy/upgrade/microkernels-in-aem-6-0.html)

java -jar oak-run-1.0.7.jar checkpoints D:\DMAM\DMAMSegmentStore\segmentstore

java -jar oak-run-1.0.7.jar checkpoints D:\DMAM\DMAMSegmentStore\segmentstore rm-all

java -jar oak-run-1.0.7.jar compact D:\DMAM\DMAMSegmentStore\segmentstore

Is actually reducing the segmentStore to is normal size.

Is there any maintenance operation that can fail ? How can I check ?

Can the configuration that put the segmentStore out of the crx-quickstart installation folder be the root cause ?

Thank in advance,

Grégory

[Sham_HC]

Gregory Paillard wrote...

Hi,

Using AEM 6.0 with the SP1 and the segmentStore is still growing up with no reason.

I'm sure that I use the SP1 because I can see "Adobe Experience Manager, Version 6.0.0.SP1" on my welcome page.

What I don't understand is that running the following commands (found here http://docs.adobe.com/docs/en/aem/6-0/deploy/upgrade/microkernels-in-aem-6-0.html)

java -jar oak-run-1.0.7.jar checkpoints D:\DMAM\DMAMSegmentStore\segmentstore

java -jar oak-run-1.0.7.jar checkpoints D:\DMAM\DMAMSegmentStore\segmentstore rm-all

java -jar oak-run-1.0.7.jar compact D:\DMAM\DMAMSegmentStore\segmentstore

Is actually reducing the segmentStore to is normal size.

Is there any maintenance operation that can fail ? How can I check ?

Can the configuration that put the segmentStore out of the crx-quickstart installation folder be the root cause ?

Thank in advance,

Grégory

Offline tool is temporary workaround. Verify with hotfix Hot fix 5354 from http://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html

Still see the issue file a official support request