AEM 6.1 not starting due to Lucene errors

Hi,

have you already raised a daycare ticket?

Were there any changes made to the system before your restarted it? Is there enough disk space?

Regards,

Opkar

Prior to upgrading our AEM, the data consistency check did complain about errors.  Hence, we went back, restored the old backup, ran the data consistency check, fixed the errors, re-ran the data consistency check, confirmed that there were no more errors and executed the upgrade again.

Now that the upgrade is done, AEM is still not starting up.  There are no Lucene index errors and no missing blobs.  Out of the 408 bundles, only 208 bundles were installed.  System Bundle is stuck at Starting.  There are Installed bundles that refuse to start and the log keeps saying 

14.10.2015 23:48:28.885 *WARN* [pool-6-thread-1] org.apache.sling.commons.scheduler.impl.QuartzScheduler No discovery info available. Executing job org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate@3099f67 with name Registered Service.365 and config SINGLE anyway.

Thank You

Not sure why your instance is not starting. The only exception I see is that it can not find "sling:jobEvent" node type definition in the repository. Did you see any errors while you upgraded in the upgrade.log file ? You can try registering the node type manually by importing the contents of the cnd file here - http://localhost:4502/crx/explorer/nodetypes/index.jsp. But I am not sure whether this is the root cause of not starting up. 

Hi,

We are using java 1.7.

Attached is the error.log

Thank You!

p/s: I gotta hand it to you.  You are even more responsive than daycare support

I unchecked "Active by default" and I saw that the System Bundle (org.apache.felix.framework)'s Status is "Starting".  The other bundles are all Resolved.  I tried starting a couple of them but nothing happens.  I checked the error.log file and nothing was logged.

I went to <server>:<port> and no longer saw "Startup in progress".  Instead, it is now showing 

Problem accessing /. Reason:

Not Found

It seems that some of your OSGI bundles are not getting Active and thats why you see the startup in progress message.  Do you know which bundle is having the problems ?  You can disable the startup filter to just to go past this message and check any exceptions when you access any UIs.  Uncheck "Active by default" in the following configuration to disable the filter - http://localhost:4502/system/console/configMgr/org.apache.sling.startupfilter.impl.StartupFilterImpl

Which version of Java are you using ?  Could you please share your error log file ?

I am not sure what is happening, but I see many queries in Mongo.  With access to system/console, I boosted the log level for error.log to TRACE, but I do not see any new information.  I triggered startPropertyIndexAsyncReindex and DataStoreGC via JMX.  I am not sure if any of these will help...AEM is still saying "Startup in progress"

kunal23 wrote...

I think that will recreate the mongo collection index and not the OAK index. I am just guessing this but you can give it a try. Just rename the index folder under /crx-quickstart/repository folder and restart AEM. 

 

I did a db.nodes.reIndex() and AEM did not start.  I restored the backup I created when I completed the upgrade the first time and renamed index to index_old.  I tried to start AEM and I was able to get to system/console.  I was still not able to get to crx/de because AEM threw a "Missing Authentication Service".  The error.log file still had the missing blob error and the repository was shut down.  I followed your previous recommendation and copied a blob file from some other folder and gave it the ID AEM was complaining about.  I tried to start AEM 6 again and now I am back at 

org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker Could not access the Lucene index at /oak:index/lucene
org.apache.lucene.index.CorruptIndexException: Invalid CFS entry offset: 1173827933404397826 (resource: _sf.cfs)

I am still able to access system/console, but the bundles now show only 173 instead of the full 469.  I see a lot of mongo DB activities going on 

2015-10-05T22:20:20.258-0700 I QUERY    [conn14] query aem6.nodes query: { $query: { _id: { $gt: "8:/jcr:system/jcr:versionStorage/03/b4/00/03b4003a-7324-4af7-95be-999b5331180d/1.0/", $lt: "8:/jcr:system/jcr:versionStorage/03/b4/00/03b4003a-7324-4af7-95be-999b5331180d/1.00" } }, $orderby: { _id: 1 }, $hint: { _id: 1 }, $maxTimeMS: 60000 } planSummary: IXSCAN { _id: 1 } ntoreturn:0 ntoskip:0 nscanned:1 nscannedObjects:1 keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:119439 locks:{ Global: { acquireCount: { r: 4 } }, MMAPV1Journal: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { R: 2 } } } 244ms

Could it be that it is rebuilding the indexes?  Accessing crx/de at this point shows "Startup in progress".  I am waiting to see if AEM can start up.

I think that will recreate the mongo collection index and not the OAK index. I am just guessing this but you can give it a try. Just rename the index folder under /crx-quickstart/repository folder and restart AEM.