Expand my Community achievements bar.

SOLVED

AEM 6.1 to AEM 6.3 Upgrade Index issue

Avatar

Level 2

Hi,

I am working on an Inplace upgrade from AEM 6.1 to AEM 6.3. I ran all the pre upgrade checks and they went fine. Also i checked the Repository consistency check. It was fine. Also ran the offline compaction. Then ran the upgrade. The unpacking and migration both went fine. But when i started the server back up using the 6.3 jar. The server is not starting back up. i left the server over night and i only see over and over only one exception.  i understand that it is reindexing, but it looks like reindexing is failing. Also does anybody know how to fix the BlobId issue. kind of stuck here, on what to do next

30.12.2017 12:50:16.246 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Removing corrupt flag from index [/oak:index/cqPageLucene] which has been marked as corrupt since [2017-12-30T03:39:42.087-06:00]

30.12.2017 12:50:16.246 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing will be performed for following indexes: [/oak:index/slingeventJob, /oak:index/versionStoreIndex, /oak:index/cqTagLucene, /oak:index/workflowDataLucene, /oak:index/cqProjectLucene, /oak:index/ntBaseLucene, /oak:index/authorizables, /oak:index/commerceLucene, /oak:index/cqPageLucene]

30.12.2017 12:50:21.225 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate Marking [/oak:index/authorizables] as corrupt. The index is failing since Sat Dec 30 03:39:46 CST 2017 ,3 indexing cycles, failed 6607 times, skipped 0 time

30.12.2017 12:50:21.226 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate Marking [/oak:index/cqPageLucene] as corrupt. The index is failing since Sat Dec 30 03:39:42 CST 2017 ,3 indexing cycles, failed 1 times, skipped 0 time

30.12.2017 12:50:21.226 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/workflowDataLucene]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.228 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/slingeventJob]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.229 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/versionStoreIndex]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.231 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/commerceLucene]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.232 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/cqProjectLucene]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.234 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/damAssetLucene]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.235 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/ntBaseLucene]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.236 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/cqTagLucene]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.237 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/lucene]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.239 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/authorizables]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.240 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Removing corrupt flag from index [/oak:index/authorizables] which has been marked as corrupt since [2017-12-30T03:39:46.261-06:00]

30.12.2017 12:50:21.240 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditorContext Stored the cloned index definition for [/oak:index/cqPageLucene]. Changes in index definition would now only be effective post reindexing

30.12.2017 12:50:21.241 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Removing corrupt flag from index [/oak:index/cqPageLucene] which has been marked as corrupt since [2017-12-30T03:39:42.087-06:00]

30.12.2017 12:50:21.241 *INFO* [async-index-update-async] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing will be performed for following indexes: [/oak:index/slingeventJob, /oak:index/versionStoreIndex, /oak:index/cqTagLucene, /oak:index/workflowDataLucene, /oak:index/cqProjectLucene, /oak:index/ntBaseLucene, /oak:index/authorizables, /oak:index/commerceLucene, /oak:index/cqPageLucene]

30.12.2017 12:50:22.455 *ERROR* [FelixStartLevel] org.apache.jackrabbit.oak.plugins.index.lucene.BadIndexTracker Could not access the Lucene index at [/oak:index/cqTagLucene] . since 9.755 h ,0 indexing cycles, accessed 2363 times

java.lang.IllegalStateException: Attempt to read external blob with blobId [8d15416c34093b37fbd9435128c5ff5bf2a45325#333321] without specifying BlobStore

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getBlob(SegmentBlob.java:247)

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getLength(SegmentBlob.java:256)

        at org.apache.jackrabbit.oak.segment.SegmentBlob.length(SegmentBlob.java:112)

        at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.<init>(OakDirectory.java:409)

        at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.<init>(OakDirectory.java:589)

        at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory.fileLength(OakDirectory.java:176)

        at org.apache.jackrabbit.oak.plugins.index.lucene.directory.IndexSanityChecker.isThereASizeMismatch(IndexSanityChecker.java:81)

        at org.apache.jackrabbit.oak.plugins.index.lucene.directory.IndexSanityChecker.check(IndexSanityChecker.java:53)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier.checkIntegrity(IndexCopier.java:250)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier.wrapForRead(IndexCopier.java:121)

        at org.apache.jackrabbit.oak.plugins.index.lucene.reader.DefaultIndexReaderFactory.createReader(DefaultIndexReaderFactory.java:102)

        at org.apache.jackrabbit.oak.plugins.index.lucene.reader.DefaultIndexReaderFactory.createReaders(DefaultIndexReaderFactory.java:61)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:68)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:227)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:179)

        at org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:239)

        at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:1018)

        at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:976)

        at org.apache.jackrabbit.oak.query.ast.SelectorImpl.prepare(SelectorImpl.java:280)

        at org.apache.jackrabbit.oak.query.QueryImpl.prepare(QueryImpl.java:641)

        at org.apache.jackrabbit.oak.query.QueryEngineImpl.prepareAndSelect(QueryEngineImpl.java:294)

1 Accepted Solution

Avatar

Correct answer by
Community Advisor

Full re-index might be easier, but could be too slow with lots of data.

Regards,

Peter

View solution in original post

4 Replies

Avatar

Level 2

Guys, Any advice or direction that i can follow. i am really out of ideas.  Even the system/console is not coming up. Should i just roll back the upgrade?

Avatar

Community Advisor

Hey,

Revert back upgrade to 6.1, re-index 6.1 to remove corrupted index from 6.1. Re-try upgrade.

Regards,

Peter

Avatar

Level 2

SHould i just delete all the indexes under the /repository/index or do i need to find the corrupt index?

Avatar

Correct answer by
Community Advisor

Full re-index might be easier, but could be too slow with lots of data.

Regards,

Peter