Datastore got deleted

[saibul]

AEM 6.5.5 

RHEL7

datastore folder got deleted under crx-quickstart\repository. Is there a way to rebuild or restore. ?

Note: This is publish server author is fine.

[Vijayalakshmi_S]

Hi @saibul,

Could you please provide details on

• The environment (DEV/QA/Stage/PROD)
• Is it On-premise or AMS 
• If repository backup is configured/available (Done either online or offline)
• Is it the only publish instance available or additional publish instance is part of this environment

[ChitraMadan · Accepted solution]

Hi @saibul,

Assuming its a non-prod environment, since there are scenarios where datastore can be shared between multiple AEM instances, be it author-publisher or publisher-publisher, I would say, if you don't have any backup to restore from, try copying thee datastore folder from author and use it in publisher, that might work. Did you try to re-install the content?

Regards,

Chitra

[saibul]

Thanks, I will check

[saibul]

We copied the datastore from its author and removed the index folder and restarted the instance. It didn't help. The issue still persists.

[ChitraMadan]

Hi @saibul, what is the error you are getting? Is your instance not coming up? What error do you see in logs?

[saibul]

CRXDE, system/console, configMgr is up and running. sites.html, assets.html and other OOTB UIs are not opening. following logs are captured in error.log

[saibul]

22.06.2021 03:08:41.181 *WARN* [async-index-update-async] org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore Error occurred while loading bytes from steam while fetching for id a7e696d931275180bcc06280108cef2ce29f1f61ce028c37cc0018cc41a8429d#51446 java.util.concurrent.ExecutionException: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record a7e696d931275180bcc06280108cef2ce29f1f61ce028c37cc0018cc41a8429d does not exist at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1017) at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:974) at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:285) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:336) [org.apache.jackrabbit.oak-blob-plugins:1.10.2] at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:46) [org.apache.jackrabbit.oak-blob-plugins:1.10.2] at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:247) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:83) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.plugins.index.lucene.directory.OakStreamingIndexFile.setupInputStream(OakStreamingIndexFile.java:138) [org.apache.jackrabbit.oak-lucene:1.10.2] at org.apache.jackrabbit.oak.plugins.index.lucene.directory.OakStreamingIndexFile.readBytes(OakStreamingIndexFile.java:234) [org.apache.jackrabbit.oak-lucene:1.10.2] at org.apache.jackrabbit.oak.plugins.index.lucene.directory.OakIndexInput.readBytes(OakIndexInput.java:65) [org.apache.jackrabbit.oak-lucene:1.10.2] at org.apache.jackrabbit.oak.plugins.index.lucene.directory.OakIndexInput.readByte(OakIndexInput.java:72) [org.apache.jackrabbit.oak-lucene:1.10.2] at org.apache.lucene.store.DataInput.readVInt(DataInput.java:108) at org.apache.lucene.store.CompoundFileDirectory.readEntries(CompoundFileDirectory.java:137) at org.apache.lucene.store.CompoundFileDirectory.<init>(CompoundFileDirectory.java:104) at org.apache.lucene.index.SegmentReader.readFieldInfos(SegmentReader.java:202) [org.apache.jackrabbit.oak-lucene:1.10.2] at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:818) [org.apache.jackrabbit.oak-lucene:1.10.2] at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:770) [org.apache.jackrabbit.oak-lucene:1.10.2] at org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.getWriter(DefaultIndexWriter.java:154) [org.apache.jackrabbit.oak-lucene:1.10.2] at org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter.deleteDocuments(DefaultIndexWriter.java:93) [org.apache.jackrabbit.oak-lucene:1.10.2] at org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexEditor.childNodeDeleted(LuceneIndexEditor.java:230) [org.apache.jackrabbit.oak-lucene:1.10.2] at org.apache.jackrabbit.oak.spi.commit.CompositeEditor.childNodeDeleted(CompositeEditor.java:134) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.spi.commit.VisibleEditor.childNodeDeleted(VisibleEditor.java:110) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:165) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:582) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:598) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:598) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:408) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:495) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:398) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:408) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:495) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:398) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:422) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:408) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:449) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:495) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:440) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:398) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord$3.childNodeChanged(MapRecord.java:408) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:449) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:495) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:440) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:398) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:651) [org.apache.jackrabbit.oak-segment-tar:1.10.2] at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:51) [org.apache.jackrabbit.oak-store-spi:1.10.2] at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:728) [org.apache.jackrabbit.oak-core:1.10.2] at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.runWhenPermitted(AsyncIndexUpdate.java:573) [org.apache.jackrabbit.oak-core:1.10.2] at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:432) [org.apache.jackrabbit.oak-core:1.10.2] at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:347) [org.apache.sling.commons.scheduler:2.7.2] at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.apache.sling.commons.scheduler:2.7.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record a7e696d931275180bcc06280108cef2ce29f1f61ce028c37cc0018cc41a8429d does not exist at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:608) [org.apache.jackrabbit.oak-blob-plugins:1.10.2] at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:340) [org.apache.jackrabbit.oak-blob-plugins:1.10.2] at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:336) [org.apache.jackrabbit.oak-blob-plugins:1.10.2] at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1013) ... 63 common frames omitted Caused by: org.apache.jackrabbit.core.data.DataStoreException: Record a7e696d931275180bcc06280108cef2ce29f1f61ce028c37cc0018cc41a8429d does not exist at org.apache.jackrabbit.core.data.AbstractDataStore.getRecord(AbstractDataStore.java:59) [org.apache.jackrabbit.jackrabbit-data:2.18.0] at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:617) [org.apache.jackrabbit.oak-blob-plugins:1.10.2] at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:602) [org.apache.jackrabbit.oak-blob-plugins:1.10.2] ... 66 common frames omitted

[shelly-goel]

@saibul  There's no way to just restore the datastore. You can restore the complete instance from a backup.

[pb_mohan83]

I am getting the very same error in my macbook in local whenever there is reboot. Wondering if we have any solution for this?