Error in author standby setup

[msina]

Hi,

AEM 6.4.

Recently we have planned to add standby author instance in our environment.Currently we are running with one author instance .

We have followed the below document.

How to Run AEM with TarMK Cold Standby

We use NFS shared storage for FileDataStore

Standby node configuration:

[root@standbyauth01 install]# pwd

/opt/aem_author/AEM_6.4_Quickstart_Setup/crx-quickstart/install

[root@wisctvnewscoldauth01 install]# cd crx3/

[root@wisctvnewscoldauth01 crx3]# cat org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg

path=/nfsdata/stage_datastore

[root@standbyauth01 install.standby]# pwd

/opt/aem_author/AEM_6.4_Quickstart_Setup/crx-quickstart/install/install.standby

[root@wisctvnewscoldauth01 install.standby]# cat org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.cfg

org.apache.sling.installer.configuration.persist=false

customBlobStore=true

standby=true

[root@standbyauth01 install.standby]# cat org.apache.jackrabbit.oak.segment.standby.store.StandbyStoreService.config

mode="standby"

primary.host="10.123.6.15"

port="8023"

interval=I"5"

standby.autoclean=B"true"

Primary node configuration:

[root@primaryauth01l]# pwd

/opt/aem_author/AEM_6.4_Quickstart_Setup/crx-quickstart/install

[root@wisctvnewsauth01 install]# cd crx3/

[root@wisctvnewsauth01 crx3]# cat org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg

path=/nfsdata/stage_datastore

[root@primaryauth01 install.primary]# pwd

/opt/aem_author/AEM_6.4_Quickstart_Setup/crx-quickstart/install/install.primary

[root@wisctvnewsauth01 install.primary]# cat org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.cfg

org.apache.sling.installer.configuration.persist=false

customBlobStore=true

standby=false

[root@primaryauth01 install.primary]# cat org.apache.jackrabbit.oak.segment.standby.store.StandbyStoreService.config

org.apache.sling.installer.configuration.persist=false

mode="primary"

port=8023

Error in Standby node

18.09.2018 20:07:11.141 *DEBUG* [standby-run-251] org.apache.jackrabbit.oak.segment.SegmentBufferWriter Writing data segment: id=cd24f27b-afbc-46ab-a4c0-d86e70397849,size=262112,segmentIdCount=289,recordIdCount=29359,recordCount=3005

18.09.2018 20:07:11.142 *DEBUG* [standby-run-251] org.apache.jackrabbit.oak.segment.file.tar.TarWriter Writing segment cd24f27b-afbc-46ab-a4c0-d86e70397849 to /opt/aem_author/AEM_6.4_Quickstart_Setup/crx-quickstart/repository/segmentstore/data00008a.tar

18.09.2018 20:07:11.175 *DEBUG* [standby-run-251] org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Channel closed

18.09.2018 20:07:11.175 *ERROR* [standby-run-251] org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync Failed synchronizing state.

java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId [e16a8bbcbea5eb457a2047d6cae1bbfa1e01562820f3c92980d4b69090baf7d5#36943]

        at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:49) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:248) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.openStream(AbstractBlob.java:44) [org.apache.jackrabbit.oak-store-spi:1.8.2]

        at com.google.common.io.ByteSource.contentEquals(ByteSource.java:300) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]

        at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67) [org.apache.jackrabbit.oak-store-spi:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:191) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at com.google.common.base.Objects.equal(Objects.java:55) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]

        at org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:59) [org.apache.jackrabbit.oak-store-spi:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:632) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:526) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compareBranch(MapRecord.java:574) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:475) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:423) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:465) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:98) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:77) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:165) [org.apache.jackrabbit.oak-segment-tar:1.8.2]

        at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:347) [org.apache.sling.commons.scheduler:2.7.2]

        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.apache.sling.commons.scheduler:2.7.2]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record e16a8bbcbea5eb457a2047d6cae1bbfa1e01562820f3c92980d4b69090baf7d5 does not exist

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:589) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:342) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]

        at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]

        ... 85 common frames omitted

Caused by: org.apache.jackrabbit.core.data.DataStoreException: Record e16a8bbcbea5eb457a2047d6cae1bbfa1e01562820f3c92980d4b69090baf7d5 does not exist

        at org.apache.jackrabbit.core.data.AbstractDataStore.getRecord(AbstractDataStore.java:58) [org.apache.jackrabbit.jackrabbit-data:2.16.0]

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:598) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:583) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]

        ... 87 common frames omitted

Can anyone please suggest?

[smacdonald2008]

Is this an upgraded AEM instance of a new instance?

[msina]

smacdonald2008 no its just a new instance .We cloned the primary VM and created this new VM and copied the aem installation folder from primary according to the document.

[berliant]

You cannot have a mix of .cfg and .config files. Starting version 6.3 they all must be .config.

As such reaname FileDatastore file extension to .config and use syntax like

path=“/nfs/datastore”

[msina]

berliant​ we can try ,but do you think is it causing the error?

Because we are already using this ".cfg" file and syntax configuration in our primary server from long back and its working fine for FileDataStore.

Configuration provided above.

Thanks

[berliant]

You have mix of configuration files .cfg and .config:

org.apache.jackrabbit.oak.segment.standby.store.StandbyStoreService.config

org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg

org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.cfg

That is the issue. In AEM 6.4 they all have to be .config - Configuring node stores and data stores in AEM 6

It should be something like:

org.apache.jackrabbit.oak.segment.standby.store.StandbyStoreService.config

org.apache.sling.installer.configuration.persist=B"false"

mode="standby"

primary.host="10.123.6.15"

port="8023"

interval=I"5"

standby.autoclean=B"true"

org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config

org.apache.sling.installer.configuration.persist=B"false"

customBlobStore=B"true"

standby=B"true"

org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.config

org.apache.sling.installer.configuration.persist=B"false"

path="/nfsdata/stage_datastore"

The same should be corrected on your primary instance.

[msina]

Thanks will try.

Please note that when we first time configured node stores and data stores with ".config" following the above doc it did not work then we configured as .cfg.

Let us try now will let you know.

[msina]

We made all files ".config" and configure as mentioned in the doc but we started getting datastore exception error in primary aem which was not the case before.

[msina]

We fixed the issue and setup primary standby properly now.no more error .

Followed the below document and use the files suggested by Day care

How to setup cold standby instance in AEM