AEM 6.1 Startup Error: Missing Authentication Service due to missing blob

[Voon_Siong]

Hi,

We recently upgraded our CQ 5.6.1 to AEM 6.1 with Mongo DB.  Our upgraded AEM 6.1 is throwing the "Missing Authentication Service" error.  The error.log file is saying that a blob could not be found and a call was issued to shut down the repository.  How do we delete the blob Id in Mongo DB (assuming that's where it is getting the Id and complaining that the blob could not be found in the file system at crx-quickstart/repository/repository/datastore)?  

Thanks in advance.

[smacdonald2008]

I sent this to our support team to see if this is a known issue. 

[Kunal_Gaba_]

Did you start your AEM instance with the following run modes - 

• -r crx3,crx3mongo for MongoMK

Can you share the contents of following config files in crx-quickstart/install folder? Also ensure that there are no spaces in the paths of your aem installation folder. 

-org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.cfg

- org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg

There is a known issue with Windows environment -

• Update to 6.1 from 5.x and 6.0+CRX2 fails on Windows due to backslashes written to FileDataStore.cfg. Workaround is to modify the 'org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg' file with datastore path before we run Migration command (CQ-42291)

[Voon_Siong]

Hi,

The following shows the "java process" after we run our start script

java -server -Xmx4G -Xms4G -XX:MaxPermSize=2G -XX:PermSize=2G -Djava.awt.headless=true -Dsling.run.modes=author,crx3,crx3mongo -jar crx-quickstart/app/cq-quickstart-6.1.0-standalone-quickstart.jar start -c crx-quickstart -i launchpad -p 4502 -Dsling.properties=conf/sling.properties

The run modes are correct

The config files' contents are as follow

-----------org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg----------

path=./crx-quickstart/repository/repository/datastore
minRecordLength=4096

---------org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreService.cfg-----------

customBlobStore=true
mongouri=mongodb://localhost:47017/
db=aem6

We do not have a SegmentNodeStoreService.cfg file...The above cfg files were created by the upgrade process

Thank you

[Kunal_Gaba_]

Can you check whether the AEM process has read/write privileges on datastore folder and can you share the exception stack trace/log file ?  

[Voon_Siong]

Hi,

The AEM was started with root and the datastore folder has rwx rights for root.  I updated the rights to the datastore folder and its contents to 777, just to test if it's permission issues.  After restarting AEM, I still see the same error.

The full warnings and exceptions logged in error.log are in the attached text file

[Kunal_Gaba_ · Accepted solution]

From the logs it seems the error is thrown when a particular blob file is not found. It could be that the file was deleted erroneously or it does not exist. You can try the following steps for recovery but I would recommend you to contact Adobe daycare support team for analyzing the root cause and providing you the right fix. 

1. Go to crx-quickstart/repository/repository/datastore folder on the file system. 
2. Under the datastore folder, if the following path does not exist - /2f/c5/21 then create this path manually.
3. Inside the above path, add a dummy binary file(empty file) and just name it as "2fc521266a17b55fecf8dc95d4ffb455ada35e05". (It would be to get the original file from any past backups though)
4. Restart AEM and see if the exception disappears.  

Also, I found that this exception is thrown when there are errors in reading the datastore files. There is already a JIRA logged for this - https://issues.apache.org/jira/browse/OAK-3442

[Voon_Siong]

Hi,

I created the dummy binary file and the log states that the CRX Repository is started.  There were some additional errors related to oak:lucene and we are still unable to access the instance.  I'll wait a bit and see if AEM ever comes up.  

I have a ticket open with Support.  If they ever provide any "official" way to resolve blob related errors, I'll update here.

Thanks for your fabulous help!