I just upgraded an instance from AEM 6.1 to AEM 6.3. I ran through the Preupgrade checks, the upgrade process and everything went fine. finally when i started the server back up it started reindexing and then it went into an infinite loop. Please find below the error log.
21.12.2017 09:36:52.066 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing will be performed for following indexes: [/oak:index/slingeventJob, /oak:index/versionStoreIndex, /oak:index/cqTagLucene, /oak:index/workflowDataLucene, /oak:index/cqProjectLucene, /oak:index/authorizables, /oak:index/ntBaseLucene, /oak:index/commerceLucene, /oak:index/cqPageLucene]
21.12.2017 09:36:57.052 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate Marking [/oak:index/authorizables] as corrupt. The index is failing since Wed Dec 20 21:47:27 CST 2017 ,4 indexing cycles, failed 8153 times, skipped 0 time
21.12.2017 09:36:57.052 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate Marking [/oak:index/ntBaseLucene] as corrupt. The index is failing since Wed Dec 20 22:17:32 CST 2017 ,4 indexing cycles, failed 361 times, skipped 0 time
21.12.2017 09:36:57.052 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/workflowDataLucene]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.054 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/slingeventJob]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.055 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/versionStoreIndex]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.057 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/commerceLucene]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.058 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/cqProjectLucene]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.059 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/damAssetLucene]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.061 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/cqTagLucene]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.062 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/lucene]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.063 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/cqPageLucene]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.064 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/authorizables]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.065 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Removing corrupt flag from index [/oak:index/authorizables] which has been marked as corrupt since [2017-12-20T21:47:27.082-06:00]
21.12.2017 09:36:57.065 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/ntBaseLucene]. Changes in index definition would now only be effective post reindexing
21.12.2017 09:36:57.066 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Removing corrupt flag from index [/oak:index/ntBaseLucene] which has been marked as corrupt since [2017-12-20T22:17:32.086-06:00]
After this happened for 6 hours finally the server starting throwing "Too many open files" exception.
21.12.2017 09:38:22.066 *WARN* [qtp88097215-112-acceptor-0@5a27f8c3-ServerConnector@1eafc3ca{HTTP/1.1}{0.0.0.0:4502}] org.eclipse.jetty.server.ServerConnector
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Does anybody have any advice or is there anyway i can rollback and start over.i have taken a back up snapshot.
Thank you
Views
Replies
Total Likes
Hi bhavanip67129846,
We are facing the same issue during 6.1 to 6.3 upgrade. Our problem statement is as same as your post. Did you get any RCA on the same? Your inputs would help in solving our issue.
Thanks!
Views
Replies
Total Likes
21.12.2017 09:38:22.066 *WARN* [qtp88097215-112-acceptor-0@5a27f8c3-ServerConnector@1eafc3ca{HTTP/1.1}{0.0.0.0:4502}] org.eclipse.jetty.server.ServerConnector
java.io.IOException: Too many open files
Check the ulimit set ? Increase it for now. Also, check the JMX console for IndexStats to get more idea on why indexing is in aloop
Thanks Kunwar!
The steps given in the following article helped in resolving the issue: Error: Too many open files
Views
Replies
Total Likes
Views
Likes
Replies
Views
Likes
Replies