AEM Author OnlineBackup failure with Server startup error:Attempt to read external blob with blobId [82f7921e11be8b37162750055df323760eaaa4f2#29624] without specifying BlobStore

[thomas_2018]

I used AEM Author OnlineBackup feature for a backup action. The Author server size is round 280Gbyte and running well more than 2 months.

Action command:

curl -u admin:password -X POST http://localhost:4502/system/console/jmx/com.adobe.granite:type=Repository/op/startBackup/java.lang....

I have changed parameter delay=0 by JMX console before the action.

The backup process was done with completed status(in JMX console) and take about 8 hours:

ClusterId
TarOptimizationRunningSince	0
TarOptimizationDelay	1
ClusterProperties	key value
ClusterNodes	OS hostname id repositoryHome
ClusterMasterId
ClusterNodeId
ClusterNodeRevision	0
DataStoreGarbageCollectionDelay	10
BackupDelay	0
BackupInProgress	false
BackupProgress	0
CurrentBackupTarget
BackupWasSuccessful	true
BackupResult	Backup completed
LicenseKey	770ff3c0-89e3-7942-896a-95922a793388
AvailableDiskSpace	0
MaximumNumberOfOpenFiles	0
SessionTracker	false
WorkspaceNames	default

After that, i copied the backup folder to another server uses same path/location/temp folder, then chmod -R 777 to the folder.

And then start it with no any other changes.

Action command:

/apps/author/crx-quickstart/bin/start

Following is the startup info(stdout.log):

07.03.2018 13:10:25.134 *INFO * [main] Setting sling.home=crx-quickstart (command line)

07.03.2018 13:10:25.165 *INFO * [main] No Apache Sling running at /127.0.0.1:37305

07.03.2018 13:10:25.166 *INFO * [main] Starting Apache Sling in /apps/author/crx-quickstart

07.03.2018 13:10:25.169 *INFO * [Apache Sling Control Listener@/127.0.0.1:46119] Apache Sling Control Listener started

07.03.2018 13:10:25.171 *INFO * [main] Sling  Extension Lib Home : /apps/author/crx-quickstart/launchpad/ext

07.03.2018 13:10:25.195 *INFO * [main] Checking launcher JAR in folder /apps/author/crx-quickstart/launchpad

07.03.2018 13:10:25.243 *INFO * [main] Existing launcher is up to date, using it: 5.4.0.2_6_10-B004 (org.apache.sling.launchpad.base.jar)

07.03.2018 13:10:25.245 *INFO * [main] Loading launcher class org.apache.sling.launchpad.base.app.MainDelegate from org.apache.sling.launchpad.base.jar

07.03.2018 13:10:25.245 *INFO * [main] External Libs Home (ext) is null or does not exists.

07.03.2018 13:10:25.293 *INFO * [main] Setting sling.launchpad=/apps/author/crx-quickstart/launchpad

07.03.2018 13:10:25.294 *INFO * [main] Setting sling.home=crx-quickstart

07.03.2018 13:10:25.294 *INFO * [main] Setting sling.properties=conf/sling.properties

07.03.2018 13:10:25.294 *INFO * [main] Setting org.osgi.service.http.port=4502

07.03.2018 13:10:25.294 *INFO * [main] Starting launcher ...

07.03.2018 13:10:25.312 *INFO * [main] HTTP server port: 4502

07.03.2018 13:10:40.320 *INFO* [FelixStartLevel] org.apache.sling.commons.logservice Service [org.apache.sling.commons.logservice.internal.LogServiceFactory,13, [org.osgi.service.log.LogService]] ServiceEvent REGISTERED

07.03.2018 13:10:40.324 *INFO* [OsgiInstallerImpl] org.apache.sling.installer.core.impl.OsgiInstallerImpl Apache Sling OSGi Installer Service started.

07.03.2018 13:10:40.327 *INFO* [FelixStartLevel] org.apache.sling.commons.logservice Service [org.apache.sling.commons.logservice.internal.LogReaderServiceFactory,14, [org.osgi.service.log.LogReaderService]] ServiceEvent REGISTERED

07.03.2018 13:10:40.327 *INFO* [FelixStartLevel] org.apache.sling.commons.logservice BundleEvent STARTED

07.03.2018 13:10:40.333 *INFO* [FelixStartLevel] slf4j.api BundleEvent STARTING

07.03.2018 13:10:40.333 *INFO* [FelixStartLevel] slf4j.api BundleEvent STARTED

07.03.2018 13:10:40.333 *INFO* [FelixStartLevel] org.apache.sling.commons.log BundleEvent STARTING

07.03.2018 13:10:40.359 *INFO* [FelixStartLevel] org.apache.sling.commons.log Service [org.apache.sling.commons.log.LogManager,15, [org.osgi.service.cm.ManagedService]] ServiceEvent REGISTERED

07.03.2018 13:10:40.359 *INFO* [FelixStartLevel] org.apache.sling.commons.log Service [org.apache.sling.commons.log.LogManager.factory.writer,16, [org.osgi.service.cm.ManagedServiceFactory]] ServiceEvent REGISTERED

07.03.2018 13:10:40.360 *INFO* [FelixStartLevel] org.apache.sling.commons.log Service [org.apache.sling.commons.log.LogManager.factory.config,17, [org.osgi.service.cm.ManagedServiceFactory]] ServiceEvent REGISTERED

07.03.2018 13:10:40.361 *INFO* [FelixStartLevel] org.apache.sling.commons.log Service [Sling Log Support,18, [javax.servlet.Servlet]] ServiceEvent REGISTERED

07.03.2018 13:10:40.362 *INFO* [FelixStartLevel] org.apache.sling.commons.log Service [Sling Log Configuration Printer,19, [org.apache.sling.commons.log.logback.internal.SlingConfigurationPrinter]] ServiceEvent REGISTERED

07.03.2018 13:10:40.363 *INFO* [FelixStartLevel] org.apache.sling.commons.log Service [Sling Log Reset Event Handler,20, [org.osgi.service.event.EventHandler]] ServiceEvent REGISTERED

07.03.2018 13:10:40.375 *INFO* [FelixStartLevel] org.apache.sling.commons.log.logback.internal.Activator LogbackManager initialized at bundle startup

07.03.2018 13:10:40.380 *INFO* [FelixStartLevel] org.apache.sling.commons.log BundleEvent STARTED

07.03.2018 13:10:40.381 *INFO* [FelixStartLevel] org.apache.sling.settings BundleEvent STARTING

07.03.2018 13:10:40.394 *INFO* [FelixStartLevel] org.apache.sling.settings.impl.SlingSettingsServiceImpl Read Sling ID 9bd74279-7e17-4ab0-bfc3-691fde09339c from file /apps/author/crx-quickstart/launchpad/felix/bundle11/data/sling.id.file

07.03.2018 13:10:40.406 *INFO* [FelixStartLevel] org.apache.sling.settings.impl.SlingSettingsServiceImpl Active run modes: [author, crx3tar, crx3, qa, nosamplecontent, china]

07.03.2018 13:10:40.407 *INFO* [FelixStartLevel] org.apache.sling.launchpad.installer Service [Apache Sling Launchpad Startup Listener,22, [org.apache.sling.installer.api.event.InstallationListener]] ServiceEvent REGISTERED

07.03.2018 13:10:40.411 *INFO* [FelixStartLevel] org.apache.sling.launchpad.installer.impl.LaunchpadConfigInstaller Activating launchpad config installer, configuration path=resources/config, install path=resources/install

07.03.2018 13:10:45.551 *ERROR* [CM Configuration Updater (ManagedServiceFactory Update: factoryPid=[org.apache.sling.commons.log.LogManager.factory.config])] org.apache.felix.configadmin Service [org.apache.felix.cm.ConfigurationAdmin,26, [org.osgi.service.cm.ConfigurationAdmin]] [org.osgi.service.cm.ManagedServiceFactory, id=17, bundle=9/slinginstall:org.apache.sling.commons.log-3.0.0.jar]: Updating property org.apache.sling.commons.log.names of configuration org.apache.sling.commons.log.LogManager.factory.config.bf341473-b4ca-4785-9afd-80987e6243a5 caused a problem: Category log.history already defined by configuration org.apache.sling.commons.log.LogManager.factory.config.bf341473-b4ca-4785-9afd-80987e6243a5 (org.osgi.service.cm.ConfigurationException: org.apache.sling.commons.log.names : Category log.history already defined by configuration org.apache.sling.commons.log.LogManager.factory.config.bf341473-b4ca-4785-9afd-80987e6243a5)

org.osgi.service.cm.ConfigurationException: org.apache.sling.commons.log.names : Category log.history already defined by configuration org.apache.sling.commons.log.LogManager.factory.config.bf341473-b4ca-4785-9afd-80987e6243a5

        at org.apache.sling.commons.log.logback.internal.config.LoggerManagedServiceFactory.updated(LoggerManagedServiceFactory.java:36)

        at org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.updated(ManagedServiceFactoryTracker.java:159)

        at org.apache.felix.cm.impl.helper.ManagedServiceFactoryTracker.provideConfiguration(ManagedServiceFactoryTracker.java:93)

        at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.provide(ConfigurationManager.java:1602)

        at org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1545)

        at org.apache.felix.cm.impl.UpdateThread.run0(UpdateThread.java:143)

        at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:110)

        at java.lang.Thread.run(Thread.java:722)

Caused by: org.apache.sling.commons.log.logback.internal.config.ConfigurationException:

        at org.apache.sling.commons.log.logback.internal.LogConfigManager.updateLoggerConfiguration(LogConfigManager.java:533)

        at org.apache.sling.commons.log.logback.internal.config.LoggerManagedServiceFactory.updated(LoggerManagedServiceFactory.java:34)

        ... 7 common frames omitted

07.03.2018 13:10:45.628 *INFO* [CM Configuration Updater (ManagedService Update: pid=[org.apache.aries.transaction])] org.apache.aries.transaction.manager Service [40, [org.apache.aries.transaction.AriesTransactionManager, javax.transaction.TransactionManager, javax.transaction.TransactionSynchronizationRegistry, javax.transaction.UserTransaction, org.apache.geronimo.transaction.manager.RecoverableTransactionManager]] ServiceEvent REGISTERED

07.03.2018 13:10:45.668 *INFO* [FelixStartLevel] com.adobe.cq.cq-deserialization-firewall Service [com.adobe.cq.deserfw.impl.DeserializationFirewallImpl,41, [com.adobe.cq.deserfw.impl.intapi.DeserializationFirewallStatus]] ServiceEvent REGISTERED

07.03.2018 13:10:45.674 *INFO* [FelixStartLevel] com.adobe.cq.cq-deserialization-firewall Service [com.adobe.cq.deserfw.impl.healthcheck.DeserializationAgentVMClassCheck,42, [org.apache.sling.hc.api.HealthCheck]] ServiceEvent REGISTERED

07.03.2018 13:10:45.675 *INFO* [FelixStartLevel] com.adobe.cq.cq-deserialization-firewall Service [com.adobe.cq.deserfw.impl.healthcheck.DeserializationAgentLoadedCheck,43, [org.apache.sling.hc.api.HealthCheck]] ServiceEvent REGISTERED

07.03.2018 13:10:45.676 *INFO* [FelixStartLevel] com.adobe.cq.cq-deserialization-firewall Service [com.adobe.cq.deserfw.impl.healthcheck.DeserializationAgentFunctionalCheck,44, [org.apache.sling.hc.api.HealthCheck]] ServiceEvent REGISTERED

Attempting to load ESAPI.properties via file I/O.

Attempting to load ESAPI.properties as resource file via file I/O.

Not found in 'org.owasp.esapi.resources' directory or file not readable: /apps/author/ESAPI.properties

Not found in SystemResource Directory/resourceDirectory: .esapi/ESAPI.properties

Not found in 'user.home' (/root) directory: /root/esapi/ESAPI.properties

Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoundException

Attempting to load ESAPI.properties via the classpath.

SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from '/ (root)' using class loader for DefaultSecurityConfiguration class!

Attempting to load validation.properties via file I/O.

Attempting to load validation.properties as resource file via file I/O.

Not found in 'org.owasp.esapi.resources' directory or file not readable: /apps/author/validation.properties

Not found in SystemResource Directory/resourceDirectory: .esapi/validation.properties

Not found in 'user.home' (/root) directory: /root/esapi/validation.properties

Loading validation.properties via file I/O failed.

Attempting to load validation.properties via the classpath.

SUCCESSFULLY LOADED validation.properties via the CLASSPATH from '/ (root)' using class loader for DefaultSecurityConfiguration class!

RAWProcessor succesfully installed

Exception in thread "sling-event-processing-pool-1" java.lang.IllegalStateException: Attempt to read external blob with blobId [82f7921e11be8b37162750055df323760eaaa4f2#29624] without specifying BlobStore

        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readBlob(FileStore.java:1337)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getBlob(SegmentBlob.java:261)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getLength(SegmentBlob.java:269)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.length(SegmentBlob.java:109)

        at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.<init>(OakDirectory.java:330)

        at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.<init>(OakDirectory.java:507)

        at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory.fileLength(OakDirectory.java:144)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.copyFilesToLocal(IndexCopier.java:457)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.prefetchIndexFiles(IndexCopier.java:413)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.<init>(IndexCopier.java:324)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier.wrapForRead(IndexCopier.java:140)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:53)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:179)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:154)

        at org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:243)

        at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:1016)

        at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:949)

        at org.apache.jackrabbit.oak.query.ast.SelectorImpl.prepare(SelectorImpl.java:288)

        at org.apache.jackrabbit.oak.query.QueryImpl.prepare(QueryImpl.java:631)

        at org.apache.jackrabbit.oak.query.QueryEngineImpl.prepareAndSelect(QueryEngineImpl.java:298)

        at org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:273)

        at org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:233)

        at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:314)

        at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:308)

        at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:304)

        at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.getTree(IdentifierManager.java:133)

        at org.apache.jackrabbit.oak.security.user.AuthorizableBaseProvider.getByContentID(AuthorizableBaseProvider.java:56)

        at org.apache.jackrabbit.oak.security.user.AuthorizableBaseProvider.getByID(AuthorizableBaseProvider.java:51)

        at org.apache.jackrabbit.oak.security.user.UserProvider.getAuthorizable(UserProvider.java:211)

        at org.apache.jackrabbit.oak.security.user.UserManagerImpl.getAuthorizable(UserManagerImpl.java:110)

        at org.apache.jackrabbit.oak.jcr.delegate.UserManagerDelegator$1.performNullable(UserManagerDelegator.java:64)

        at org.apache.jackrabbit.oak.jcr.delegate.UserManagerDelegator$1.performNullable(UserManagerDelegator.java:61)

        at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performNullable(SessionDelegate.java:243)

        at org.apache.jackrabbit.oak.jcr.delegate.UserManagerDelegator.getAuthorizable(UserManagerDelegator.java:61)

        at org.apache.sling.jcr.resource.internal.JcrSystemUserValidator.isValid(JcrSystemUserValidator.java:114)

        at org.apache.sling.serviceusermapping.impl.ServiceUserMapperImpl.isValidUser(ServiceUserMapperImpl.java:379)

        at org.apache.sling.serviceusermapping.impl.ServiceUserMapperImpl.getServiceUserID(ServiceUserMapperImpl.java:196)

        at org.apache.sling.resourceresolver.impl.ResourceResolverFactoryImpl.getServiceResourceResolver(ResourceResolverFactoryImpl.java:82)

        at com.day.cq.wcm.core.impl.devicedetection.DeviceIdentificationModeImpl$2.run(DeviceIdentificationModeImpl.java:245)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:722)

07.03.2018 13:11:59.703 *INFO * [main] Startup completed

After startup completed,i try to open Welcome page(http://backupserver:4502/welcome) in all browsers(IE/Firefox/Chrome)after wait 2 minutes, got following error infomation. meanwhile, CPU works in high usage rate!

HTTP ERROR: 500

Problem accessing /libs/granite/core/content/login.html. Reason:

    java.lang.StackOverflowError

---

Powered by Jetty://

And so much error output(circularly) in file error.log:

07.03.2018 14:20:36.154 *INFO* [qtp859942854-181] org.apache.sling.auth.core.impl.SlingAuthenticator handleLoginFailure: Unable to authenticate anonymous user: java.lang.IllegalStateException: Attempt to read external blob with blobId [82f7921e11be8b37162750055df323760eaaa4f2#29624] without specifying BlobStore

        at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readBlob(FileStore.java:1337)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getBlob(SegmentBlob.java:261)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getLength(SegmentBlob.java:269)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.length(SegmentBlob.java:109)

        at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.<init>(OakDirectory.java:330)

        at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.<init>(OakDirectory.java:507)

        at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory.fileLength(OakDirectory.java:144)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.copyFilesToLocal(IndexCopier.java:457)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.prefetchIndexFiles(IndexCopier.java:413)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.<init>(IndexCopier.java:324)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier.wrapForRead(IndexCopier.java:140)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:53)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:179)

        at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:154)

        at org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:243)

        at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:1016)

        at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:949)

        at org.apache.jackrabbit.oak.query.ast.SelectorImpl.prepare(SelectorImpl.java:288)

        at org.apache.jackrabbit.oak.query.QueryImpl.prepare(QueryImpl.java:631)

        at org.apache.jackrabbit.oak.query.QueryEngineImpl.prepareAndSelect(QueryEngineImpl.java:298)

        at org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:273)

        at org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:233)

        at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:314)

        at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:308)

        at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.resolveUUID(IdentifierManager.java:304)

        at org.apache.jackrabbit.oak.plugins.identifier.IdentifierManager.getTree(IdentifierManager.java:133)

        at org.apache.jackrabbit.oak.security.user.AuthorizableBaseProvider.getByContentID(AuthorizableBaseProvider.java:56)

        at org.apache.jackrabbit.oak.security.user.AuthorizableBaseProvider.getByID(AuthorizableBaseProvider.java:51)

        at org.apache.jackrabbit.oak.security.user.UserProvider.getAuthorizable(UserProvider.java:211)

        at org.apache.jackrabbit.oak.security.user.UserManagerImpl.getAuthorizable(UserManagerImpl.java:110)

        at org.apache.jackrabbit.oak.security.user.UserAuthentication.authenticate(UserAuthentication.java:100)

        at org.apache.jackrabbit.oak.security.authentication.user.LoginModuleImpl.login(LoginModuleImpl.java:131)

        at org.apache.felix.jaas.boot.ProxyLoginModule.login(ProxyLoginModule.java:52)

        at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:601)

        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:784)

        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)

        at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721)

        at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:718)

        at javax.security.auth.login.LoginContext.login(LoginContext.java:590)

        at org.apache.jackrabbit.oak.core.ContentRepositoryImpl.login(ContentRepositoryImpl.java:165)

        at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(RepositoryImpl.java:280)

        at com.adobe.granite.repository.impl.CRX3RepositoryImpl.login(CRX3RepositoryImpl.java:94)

        at org.apache.jackrabbit.oak.jcr.repository.RepositoryImpl.login(RepositoryImpl.java:219)

        at org.apache.sling.jcr.base.AbstractSlingRepository2.login(AbstractSlingRepository2.java:288)

        at org.apache.sling.jcr.resource.internal.helper.jcr.JcrProviderStateFactory.createProviderState(JcrProviderStateFactory.java:121)

        at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.authenticate(JcrResourceProvider.java:267)

        at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.authenticate(JcrResourceProvider.java:78)

        at org.apache.sling.resourceresolver.impl.providers.stateful.ProviderManager.authenticate(ProviderManager.java:161)

        at org.apache.sling.resourceresolver.impl.providers.stateful.ProviderManager.getOrCreateProvider(ProviderManager.java:87)

        at org.apache.sling.resourceresolver.impl.providers.stateful.ProviderManager.authenticateAll(ProviderManager.java:129)

        at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.createControl(ResourceResolverImpl.java:154)

        at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.<init>(ResourceResolverImpl.java:116)

        at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.<init>(ResourceResolverImpl.java:110)

        at org.apache.sling.resourceresolver.impl.CommonResourceResolverFactoryImpl.getResourceResolverInternal(CommonResourceResolverFactoryImpl.java:258)

        at org.apache.sling.resourceresolver.impl.CommonResourceResolverFactoryImpl.getResourceResolver(CommonResourceResolverFactoryImpl.java:163)

        at org.apache.sling.resourceresolver.impl.ResourceResolverFactoryImpl.getResourceResolver(ResourceResolverFactoryImpl.java:101)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.getAnonymousResolver(SlingAuthenticator.java:861)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.handleLoginFailure(SlingAuthenticator.java:964)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.getAnonymousResolver(SlingAuthenticator.java:883)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.handleLoginFailure(SlingAuthenticator.java:964)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.getAnonymousResolver(SlingAuthenticator.java:883)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.handleLoginFailure(SlingAuthenticator.java:964)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.getAnonymousResolver(SlingAuthenticator.java:883)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.handleLoginFailure(SlingAuthenticator.java:964)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.getAnonymousResolver(SlingAuthenticator.java:883)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.handleLoginFailure(SlingAuthenticator.java:964)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.getAnonymousResolver(SlingAuthenticator.java:883)

        at org.apache.sling.auth.core.impl.SlingAuthenticator.handleLoginFailure(SlingAuthenticator.java:964)

... ...

... ...

Any good guys can help to see if it's a AEM Onlinebackup functional bug or is my operational issue?

Many thanks!!

[thomas_2018]

AEM Version:

Adobe Experience Manager, Version 6.2.0

[thomas_2018]

Jörg Hoh​​

Dear @Jörg Hoh

Could you please help to check the root cause or provide some solution?

this issue may cause our Production server in a non-backup situation.

Thanks a lot!

[Jörg_Hoh]

What Oak version do you use? Any Service Packs or Cumulative Fixpacks installed?

Jörg

[Kunwar]

What's the architecture of this instance ? I assume we are using a FileDatastore but is it inside the /crx-quickstart or outside ?

I remember pre AEM 6.3, When using an external FileDataStore, then online backup (with the segment store) is only consistent if the FileDataStore is backed up after segment store.

[thomas_2018]

Oak version is 1.4.5

-----------------------------------------------------

jcr.repository.name Apache Jackrabbit Oak

jcr.repository.vendor The Apache Software Foundation

jcr.repository.vendor.url http://www.apache.org/

jcr.repository.version 1.4.5

jcr.specification.name Content Repository for Java Technology API

jcr.specification.version 2.0

------------------------------------------------------------------

Name Granite Oak

Version 0.3.0

HomeDir /apps/author/crx-quickstart/repository

----------------------------------------------------------------

Operating System is RedHat

----------------------------------------

LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch

Distributor ID: RedHatEnterpriseServer

Description:    Red Hat Enterprise Linux Server release 6.4 (Santiago)

Release:        6.4

Codename:       Santiago

-----------------------------------------

JDK Version 1.8

-----------------------------------------------------------------------------------

java version "1.8.0_102"

Java(TM) SE Runtime Environment (build 1.8.0_102-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

----------------------------------------------------------------------------------------------------------

The AEM version was from 5.6.0 upgrated to 6.2.0 last year.

-----------------------------------------------------------------------------------------------------

Following are packages installed:

cq-6.2.0-hotfix-11700-1.0.zip

cq-dam-bundles-content-1.1.362.zip

cq-6.2.0-hotfix-10956-1.2.zip

cq-ui-touch-optimized-content-2.2.998.zip

cq-ui-classic-content-1.2.124.zip

cq-foundation-content-1.2.352.zip

cq-platform-content-1.2.404.zip

cq-6.2.0-hotfix-10936-1.0.zip

com.adobe.granite.backup.content-0.0.52-CQ620-B0002.zip

cq-6.2.0-hotfix-10870-1.0.zip

cq-6.2.0-hotfix-11687-1.0.zip

cq-6.2.0-hotfix-11490-1.2.zip

screens-client-pkg-1.0.48.zip

cq-content-insight-content-1.2.74.zip

cq-commerce-content-1.4.336.zip

cq-social-scf-pkg-1.1.39.zip

... ...

... ...

so many....

thanks Jrg!

[thomas_2018]

Yes, we uses FileDatastore inside the /crx-quickstart:

/apps/author/crx-quickstart/repository/index

/apps/author/crx-quickstart/repository/segmentstore

/apps/author/crx-quickstart/repository/repository/datastore

---------------------------------------------------------------------

The AEM version is 6.2.0 which upgrated from 5.6.0 last year.

-------------------------------------------------------------

There are some settings in folder:

/apps/author/crx-quickstart/install

--------------------------------------------------------------------

org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg

path=./crx-quickstart/repository/repository/datastore

minRecordLength=4096

org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.cfg

customBlobStore=true

Thanks Kunwar

[Kunwar · Accepted solution]

I think this is the same thing as I mentioned earlier, Do a change if possible in the architecture by moving things around

- stop AEM

- move the datastore folder outside the crx-quickstart folder and adjust the filedatastore config in the install folder

- start aem

- initiate an online backup

now the datastore(major portion of AEM) is outside the online backup process now shouldnt take much time and once the backup is complete you’d have a backup of crx-quickstart which includes segmentstore and other folders inside it.

Once backup is complete, backup the datastore separately using rsync or any backup tool. Just make sure you backup the datastore only after online backup is complete

Setup this online backed-up repository and backed up datastore to new location(or server) and start this and am certain there won’t be an issue now.

[thomas_2018]

Thank you Kunwar​!

Will check your solution!

[terenceleung]

Could the backup of the datastore be done online?

[Jörg_Hoh]

the datastore can be backed up online (as the files on there are read-only). But make sure that you backup the segment store first and then the datastore.

Jörg