Error in author standby setup
Hi,
AEM 6.4.
Recently we have planned to add standby author instance in our environment.Currently we are running with one author instance .
We have followed the below document.
How to Run AEM with TarMK Cold Standby
We use NFS shared storage for FileDataStore
Standby node configuration:
[root@standbyauth01 install]# pwd
/opt/aem_author/AEM_6.4_Quickstart_Setup/crx-quickstart/install
[root@wisctvnewscoldauth01 install]# cd crx3/
[root@wisctvnewscoldauth01 crx3]# cat org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg
path=/nfsdata/stage_datastore
[root@standbyauth01 install.standby]# pwd
/opt/aem_author/AEM_6.4_Quickstart_Setup/crx-quickstart/install/install.standby
[root@wisctvnewscoldauth01 install.standby]# cat org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.cfg
org.apache.sling.installer.configuration.persist=false
customBlobStore=true
standby=true
[root@standbyauth01 install.standby]# cat org.apache.jackrabbit.oak.segment.standby.store.StandbyStoreService.config
mode="standby"
primary.host="10.123.6.15"
port="8023"
interval=I"5"
standby.autoclean=B"true"
Primary node configuration:
[root@primaryauth01l]# pwd
/opt/aem_author/AEM_6.4_Quickstart_Setup/crx-quickstart/install
[root@wisctvnewsauth01 install]# cd crx3/
[root@wisctvnewsauth01 crx3]# cat org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg
path=/nfsdata/stage_datastore
[root@primaryauth01 install.primary]# pwd
/opt/aem_author/AEM_6.4_Quickstart_Setup/crx-quickstart/install/install.primary
[root@wisctvnewsauth01 install.primary]# cat org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.cfg
org.apache.sling.installer.configuration.persist=false
customBlobStore=true
standby=false
[root@primaryauth01 install.primary]# cat org.apache.jackrabbit.oak.segment.standby.store.StandbyStoreService.config
org.apache.sling.installer.configuration.persist=false
mode="primary"
port=8023
Error in Standby node
18.09.2018 20:07:11.141 *DEBUG* [standby-run-251] org.apache.jackrabbit.oak.segment.SegmentBufferWriter Writing data segment: id=cd24f27b-afbc-46ab-a4c0-d86e70397849,size=262112,segmentIdCount=289,recordIdCount=29359,recordCount=3005
18.09.2018 20:07:11.142 *DEBUG* [standby-run-251] org.apache.jackrabbit.oak.segment.file.tar.TarWriter Writing segment cd24f27b-afbc-46ab-a4c0-d86e70397849 to /opt/aem_author/AEM_6.4_Quickstart_Setup/crx-quickstart/repository/segmentstore/data00008a.tar
18.09.2018 20:07:11.175 *DEBUG* [standby-run-251] org.apache.jackrabbit.oak.segment.standby.client.StandbyClient Channel closed
18.09.2018 20:07:11.175 *ERROR* [standby-run-251] org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync Failed synchronizing state.
java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId [e16a8bbcbea5eb457a2047d6cae1bbfa1e01562820f3c92980d4b69090baf7d5#36943]
at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:49) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:248) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.openStream(AbstractBlob.java:44) [org.apache.jackrabbit.oak-store-spi:1.8.2]
at com.google.common.io.ByteSource.contentEquals(ByteSource.java:300) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67) [org.apache.jackrabbit.oak-store-spi:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentBlob.equals(SegmentBlob.java:191) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at com.google.common.base.Objects.equal(Objects.java:55) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
at org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:59) [org.apache.jackrabbit.oak-store-spi:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentPropertyState.equals(SegmentPropertyState.java:249) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareProperties(SegmentNodeState.java:632) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:526) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compareBranch(MapRecord.java:574) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:475) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:423) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeChanged(CancelableDiff.java:85) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord$4.childNodeChanged(MapRecord.java:450) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:496) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:441) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.MapRecord.compare(MapRecord.java:465) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:619) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:98) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:77) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:165) [org.apache.jackrabbit.oak-segment-tar:1.8.2]
at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:347) [org.apache.sling.commons.scheduler:2.7.2]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.apache.sling.commons.scheduler:2.7.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record e16a8bbcbea5eb457a2047d6cae1bbfa1e01562820f3c92980d4b69090baf7d5 does not exist
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:589) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:342) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]
at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]
... 85 common frames omitted
Caused by: org.apache.jackrabbit.core.data.DataStoreException: Record e16a8bbcbea5eb457a2047d6cae1bbfa1e01562820f3c92980d4b69090baf7d5 does not exist
at org.apache.jackrabbit.core.data.AbstractDataStore.getRecord(AbstractDataStore.java:58) [org.apache.jackrabbit.jackrabbit-data:2.16.0]
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:598) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:583) [org.apache.jackrabbit.oak-blob-plugins:1.8.2]
... 87 common frames omitted
Can anyone please suggest?
