Expand my Community achievements bar.

Dive into Adobe Summit 2024! Explore curated list of AEM sessions & labs, register, connect with experts, ask questions, engage, and share insights. Don't miss the excitement.

Upgrading to AEM 6.3 from 6.1 Indexing issues

Avatar

Level 2

I just upgraded an instance from AEM 6.1 to AEM 6.3. I ran through the Preupgrade checks, the upgrade process and everything went fine. finally when i started the server back up it started reindexing and then it went into an infinite loop. Please find below the error log.

21.12.2017 09:36:52.066 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing will be performed for following indexes: [/oak:index/slingeventJob, /oak:index/versionStoreIndex, /oak:index/cqTagLucene, /oak:index/workflowDataLucene, /oak:index/cqProjectLucene, /oak:index/authorizables, /oak:index/ntBaseLucene, /oak:index/commerceLucene, /oak:index/cqPageLucene]

21.12.2017 09:36:57.052 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate Marking [/oak:index/authorizables] as corrupt. The index is failing since Wed Dec 20 21:47:27 CST 2017 ,4 indexing cycles, failed 8153 times, skipped 0 time

21.12.2017 09:36:57.052 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate Marking [/oak:index/ntBaseLucene] as corrupt. The index is failing since Wed Dec 20 22:17:32 CST 2017 ,4 indexing cycles, failed 361 times, skipped 0 time

21.12.2017 09:36:57.052 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/workflowDataLucene]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.054 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/slingeventJob]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.055 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/versionStoreIndex]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.057 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/commerceLucene]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.058 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/cqProjectLucene]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.059 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/damAssetLucene]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.061 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/cqTagLucene]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.062 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/lucene]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.063 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/cqPageLucene]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.064 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/authorizables]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.065 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Removing corrupt flag from index [/oak:index/authorizables] which has been marked as corrupt since [2017-12-20T21:47:27.082-06:00]

21.12.2017 09:36:57.065 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/ntBaseLucene]. Changes in index definition would now only be effective post reindexing

21.12.2017 09:36:57.066 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Removing corrupt flag from index [/oak:index/ntBaseLucene] which has been marked as corrupt since [2017-12-20T22:17:32.086-06:00]

After this happened for 6 hours finally the server starting throwing "Too many open files" exception.

21.12.2017 09:38:22.066 *WARN* [qtp88097215-112-acceptor-0@5a27f8c3-ServerConnector@1eafc3ca{HTTP/1.1}{0.0.0.0:4502}] org.eclipse.jetty.server.ServerConnector

java.io.IOException: Too many open files

        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)

        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)

        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

        at java.lang.Thread.run(Thread.java:745)

Does anybody have any advice or is there anyway i can rollback and start over.i have taken a back up snapshot.

Thank you

3 Replies

Avatar

Level 3

Hi bhavanip67129846,

We are facing the same issue during 6.1 to 6.3 upgrade. Our problem statement is as same as your post. Did you get any RCA on the same? Your inputs would help in solving our issue.

Thanks!

Avatar

Employee

21.12.2017 09:38:22.066 *WARN* [qtp88097215-112-acceptor-0@5a27f8c3-ServerConnector@1eafc3ca{HTTP/1.1}{0.0.0.0:4502}] org.eclipse.jetty.server.ServerConnector

java.io.IOException: Too many open files

Check the ulimit set ? Increase it for now. Also, check the JMX console for IndexStats to get more idea on why indexing is in aloop

Avatar

Level 3

Thanks Kunwar!

The steps given in the following article helped in resolving the issue: Error: Too many open files