Re-indexing kicks off when AEM is restated

[Nitiks25]

Hello,

We have observed that whenever AEM is restarted, re-indexing is kicked off again. We keep seeing the asyncIndexUpdate messages in the log about indexing content and it goes on for 2-3 hours.

Because of which the instance takes a long time to get back up. Is this normal? Is it expected for re-indexing to be done every restart? Or is it a platform issue? We're using AEM 6.0. If this is expected, any way to turn reindexing off during startup? Please note, the reindex property is set to false on our oak indexes.

Thanks. Any help here will be much appreciated.

[smacdonald2008]

Talked to our support team about this -- the reply:

This is not normal

Most likely index are corrupted

When next time they stop it, they should:

• Run offline compaction
• With that use checkpoint rm-all which will trigger full re-indexing
• Remove /crx-quickstart/repository/index
• Remove all data*.tar.bak in /crx-quickstart/repository/segmentstore

[Jörg_Hoh]

What Oak version are you using? And you are strongly recommended to upgrade to a more recent version ...

regards,

Jörg

[swapnilg7731718]

1.0.27 oak version is used. Could you suggest any other option to temporary fix this rather than upgrade ?  Appreciate any sort of suggestion. Thanks

[Jörg_Hoh]

Hi,

Oak 1.0.27 is quite old and has some bugs, which cause strange issues; the one you mention can be one of them (don't remember all of them). I don't think that there is any other temporary solution than updating to a more recent Oak version (e.g. 1.0.39) and the final solution should be the upgrade to a newer AEM version.

To have an even more temporary workaround, we need to do a deeper analysis to understand what is happening. This behaviour is not expected. Can you post the relevant log information when the system kicks of this reindexing process? And have you already raised a ticket at Adobe support for this problem?

kind regards,

Jörg

[swapnilg7731718]

Share the logs and issue

While starting the publisher it takes  approx. 2 hours to finish the async index update. Once index updates then only server starts. Due to this server startup is taking more than 2.5 hours. Need to know why index is updated every time when server reboot. Is that something wrong? If no, then how can we skip the index update calls from publisher start?

Approx repository size is 250 GB.

Why system start forcing an index rebuild, until and unless even if I am not removing the index every time I bring AEM down

Error snapshot after in error log (using grep) :

1. Org.apache.sling.jcrrabbit.oak.plugins.index.indexUpdate /oak:index/<myIndex> => indexed 10000 nodes in 11.30 min …
2. Org.apache.sling.jcrrabbit.oak.plugins.index.indexUpdate /oak:index/<myIndex> => indexed 20000 nodes in 09.43 min …
3. Org.apache.sling.jcrrabbit.oak.plugins.index.indexUpdate /oak:index/<myIndex> => indexed 30000 nodes in  08.30 min …
4. Org.apache.sling.jcrrabbit.oak.plugins.index.indexUpdate /oak:index/<myIndex> => indexed 40000 nodes in 14.22 min …
5. Org.apache.sling.jcrrabbit.oak.plugins.index.indexUpdate /oak:index/<myIndex> => indexed 50000 nodes in 12.30 min …

and so on..

1. Org.apache.jackrabbit.oak.plugin.index.AsyncIndexUpdate AsyncIndex (async) update run completed in 6.654 min. Indexed 134512 nodes.
2. Org.apache.jackrabbit.oak.plugin.index.AsyncIndexUpdate AsyncIndex (async) update run completed in 8.123 min. Indexed 234562 nodes.
3. Org.apache.jackrabbit.oak.plugin.index.AsyncIndexUpdate AsyncIndex (async) update run completed in 5.763 min. Indexed 289877 nodes.
4. Org.apache.jackrabbit.oak.plugin.index.AsyncIndexUpdate AsyncIndex (async) update run completed in 9.814 min. Indexed 408744 nodes.
5. Org.apache.jackrabbit.oak.plugin.index.AsyncIndexUpdate AsyncIndex (async) update run completed in 10.765 min. Indexed 674646 nodes.

[Jörg_Hoh · Accepted solution]

Are there any mentions regarding "index" before the reindexing starts? I would assume there are. But besides this, I would strongly recommend to update your Oak version.

Issue Navigator - ASF JIRA  lists the fixed issues up to 1.0.40 (excluding 1.0.27 and older). And in my last project (where we used Oak 1.0/AEM 6.0) we always followed the Oak versions closely because of such issues. And at least up to 1.0.33 it was absolutly necessary for us.

btw: These long times for indexing 10k nodes (up to 14 minutes) look suspicious; are you using MongoDB or RDBMK? Or do you have a lot of assets and using an S3 blobstore? Because the indexing time looks very high. And I wonder also how these 2 blocks ("indexed N nodes ..." and "update run completed") fit; in one block it's logged, that indexing 10k nodes takes 10 minutes, while in the other indexing more than 500k nodes takes 10 minutes.

Jörg