Datastore issue in AEM test Publisher | Community
Skip to main content
seemak19887037
seemak19887037
Level 2
July 28, 2021
Solved

Datastore issue in AEM test Publisher

  • July 28, 2021
  • 1 reply
  • 2420 views

Hi Team,

We are getting datastore issue in our test publisher. Intially it was -111675 . we have done garbage collection, offline compaction, removed the debug logger but still it increasimg day by day and now it becomes- 198344

We have followed this document:- https://helpx.adobe.com/in/experience-manager/kb/analyze-unusual-repository-growth.html

Added the repogrowth.log in the logger and aslo run the diskusage report but it was running from 1 hr and we didnot get any result out of that. We are giving the sample of repogrowth.log:-

 

15.07.2021 07:05:38.530 *TRACE* [sling-default-2-com.cognifide.actions.core.persist.Resender.5335] org.apache.jackrabbit.oak.jcr.operations.writes [session-78366382] save 15.07.2021 07:05:39.374 *TRACE* [sling-default-1-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:05:44.373 *TRACE* [sling-default-1-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:05:44.440 *TRACE* [discovery.connectors.common.runner.5002ee23-1d23-43cf-a1f5-5fa9079589dc.connectorPinger] org.apache.jackrabbit.oak.jcr.operations.writes [session-78366389] refresh 15.07.2021 07:05:44.441 *TRACE* [discovery.connectors.common.runner.5002ee23-1d23-43cf-a1f5-5fa9079589dc.connectorPinger] org.apache.jackrabbit.oak.jcr.operations.writes [session-78366389] refresh 15.07.2021 07:05:49.373 *TRACE* [sling-default-4-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:05:54.374 *TRACE* [sling-default-1-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:05:59.374 *TRACE* [sling-default-3-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:06:00.058 *TRACE* [sling-default-2-Registered Service.170361] org.apache.jackrabbit.oak.jcr.operations.writes [session-78366411] save 15.07.2021 07:06:00.058 *TRACE* [sling-default-2-Registered Service.170361] org.apache.jackrabbit.oak.jcr.operations.writes [session-78366415] save 15.07.2021 07:06:01.451 *TRACE* [sling-default-3-health-biz.netcentric.cq.tools.actool.healthcheck.LastRunSuccessHealthCheck] org.apache.jackrabbit.oak.jcr.operations.writes [session-78366423] Adding node [/var/statistics/achistory] 15.07.2021 07:06:04.374 *TRACE* [sling-default-5-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:06:08.529 *TRACE* [sling-default-1-com.cognifide.actions.core.persist.Resender.5335] org.apache.jackrabbit.oak.jcr.operations.writes [session-78366482] save 15.07.2021 07:06:09.374 *TRACE* [sling-default-3-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:06:14.374 *TRACE* [sling-default-4-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:06:14.444 *TRACE* [discovery.connectors.common.runner.5002ee23-1d23-43cf-a1f5-5fa9079589dc.connectorPinger] org.apache.jackrabbit.oak.jcr.operations.writes [session-78366496] refresh 15.07.2021 07:06:14.445 *TRACE* [discovery.connectors.common.runner.5002ee23-1d23-43cf-a1f5-5fa9079589dc.connectorPinger] org.apache.jackrabbit.oak.jcr.operations.writes [session-78366496] refresh 15.07.2021 07:06:19.374 *TRACE* [sling-default-1-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:06:24.374 *TRACE* [sling-default-1-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:06:29.373 *TRACE* [sling-default-2-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:06:34.374 *TRACE* [sling-default-5-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh 15.07.2021 07:06:38.529 *TRACE* [sling-default-2-com.cognifide.actions.core.persist.Resender.5335] org.apache.jackrabbit.oak.jcr.operations.writes [session-78366548] save 15.07.2021 07:06:39.373 *TRACE* [sling-default-1-com.day.cq.rewriter.linkchecker.impl.LinkInfoStorageImpl.173880] org.apache.jackrabbit.oak.jcr.operations.writes [session-69917828] refresh

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by MarkusBullaAdobe

Hi @seemak19887037!

So from your posting I understand that you are experiencing unusual and unexpected repository growth.

Is that correct?

(Although I'm not sure about the figures shared. Is that ~112 GB vs. ~198 GB?)

 

There are several starting points to understand where this growth comes from:

  • To determine the size and distribution on the file system, you can leverage operating system tools such as "du -s" on the instance folder(s). This may help to tell instance, logs, repository and datastore usage apart. This will help you understand where the unexpected growth is happening and if you should take a deeper look inside repository (see following recommendation).
  • To get a better view on the distribution within the repository, you may want to check AEMs Disk Usage report as outlined in this blog post. You can drill down and check disk usage distributions on each level of the content hierarchy. Please be aware that the tool performs a full tree iteration of the selected path which may take some time and causes quite some load on the instance depending on the size and structure of your repository. This may help to identify the areas causing repository growth. Is it under /etc/packages? Is it /content/dam? Or any other specific location?
  • Please be aware that there are regular maintenance jobs that should be running on the AEM instance. While on the long term all of these jobs are meant to increase instance performance and reduce repository size, sometimes they may cause repository growth on a short term basis - especially if combined with offline compaction (see also next remark).
    Some of the relevant jobs are:
  • One important remark from this documentation:
    "So, when you run online revision cleanup after offline revision cleanup the following happens:
    1. After the first online revision cleanup run the repository will double in size. This happens because there are now two generations that are kept on disk.
    2. During the subsequent runs, the repository will temporarily grow while the new generation is created and then stabilize back to the size it had after the first run, as the online revision cleanup process reclaims the previous generation.

    [...]

    Due to this fact, it is recommended to size the disk at least two or three times larger than the initially estimated repository size." (Emphasis in quotation added by me.)

  • Please also check if there are regular jobs that work with packages / package manager. Each package installation creates a snapshot of the relevant area of the repository before the actual package installation takes place. Cleaning up unused packages (which will also cleanup the according snapshots) is often a good area to save on disk space.
  • You may want to check this blog post on managing repository size growth in AEM.

 

Hope that helps!

1 reply

MarkusBullaAdobe
Adobe Employee
MarkusBullaAdobeAdobe EmployeeAccepted solution
Adobe Employee
July 28, 2021

Hi @seemak19887037!

So from your posting I understand that you are experiencing unusual and unexpected repository growth.

Is that correct?

(Although I'm not sure about the figures shared. Is that ~112 GB vs. ~198 GB?)

 

There are several starting points to understand where this growth comes from:

  • To determine the size and distribution on the file system, you can leverage operating system tools such as "du -s" on the instance folder(s). This may help to tell instance, logs, repository and datastore usage apart. This will help you understand where the unexpected growth is happening and if you should take a deeper look inside repository (see following recommendation).
  • To get a better view on the distribution within the repository, you may want to check AEMs Disk Usage report as outlined in this blog post. You can drill down and check disk usage distributions on each level of the content hierarchy. Please be aware that the tool performs a full tree iteration of the selected path which may take some time and causes quite some load on the instance depending on the size and structure of your repository. This may help to identify the areas causing repository growth. Is it under /etc/packages? Is it /content/dam? Or any other specific location?
  • Please be aware that there are regular maintenance jobs that should be running on the AEM instance. While on the long term all of these jobs are meant to increase instance performance and reduce repository size, sometimes they may cause repository growth on a short term basis - especially if combined with offline compaction (see also next remark).
    Some of the relevant jobs are:
  • One important remark from this documentation:
    "So, when you run online revision cleanup after offline revision cleanup the following happens:
    1. After the first online revision cleanup run the repository will double in size. This happens because there are now two generations that are kept on disk.
    2. During the subsequent runs, the repository will temporarily grow while the new generation is created and then stabilize back to the size it had after the first run, as the online revision cleanup process reclaims the previous generation.

    [...]

    Due to this fact, it is recommended to size the disk at least two or three times larger than the initially estimated repository size." (Emphasis in quotation added by me.)

  • Please also check if there are regular jobs that work with packages / package manager. Each package installation creates a snapshot of the relevant area of the repository before the actual package installation takes place. Cleaning up unused packages (which will also cleanup the according snapshots) is often a good area to save on disk space.
  • You may want to check this blog post on managing repository size growth in AEM.

 

Hope that helps!

    seemak19887037
    seemak19887037Author
    Level 2
    July 30, 2021

    Hi @markusbullaadobe

    I have run the disusage report again and this time it run successfully. Sharing the result with you.

     

      Terms & ConditionsAccessibility statement

      Loading footer...