AEM 6.1 to 6.3 Inplace Upgrade Steps are failing

[indrasena_reddy]

Hi All,

We are upgrading the AEM Server from 6.1 to 6.3 version, completed all the Upgrade activities like maintenance , pre-upgrade , tar compaction & oak migration except the replacing the AEM 6.3 jar and bringing up instance is not successful.

Command

java -Xmx4096m -XX:MaxPermSize=2048m  -jar cq-quickstart-630-author-p4502.jar

Also we are using the osgi configuration for File Data storage & Node storage.

org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore

org.apache.jackrabbit.oak.segment.SegmentNodeStoreService

I have analyzed the issue and following are the observations. Please suggest me how to solve this issue.

1) Generated 5 GB of error.log with so many warn and error message.

2) Project related bundles, apps, etc are missing in crxde.

3) Start page and Welcome page (/aem/start.html, /libs/cq/core/content/welcome.html) are not opening throwing exception.

stderr.log

21.03.2018 06:26:24.231 *ERROR* [FelixStartLevel] ERROR: Bundle '360' EventDispatcher: Error during dispatch. (java.lang.NullPointerException)

java.lang.NullPointerException

at com.day.cq.dam.pim.impl.sourcing.selection.ProductAssetSelector$1.addingService(ProductAssetSelector.java:219)

at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:941)

at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:870)

at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)

at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229)

at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:901)

at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)

at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)

at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)

at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4558)

at org.apache.felix.framework.Felix.registerService(Felix.java:3550)

at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348)

error.logs(some error message)

-> 21.03.2018 06:28:35.912 *WARN* [FelixStartLevel] com.day.cq.cq-replication [com.day.cq.replication.impl.ReplicatorImpl(463)] Could not get service from ref [com.day.cq.replication.AgentManager, org.apache.sling.event.jobs.consumer.JobExecutor]

21.03.2018 06:28:35.912 *WARN* [FelixStartLevel] com.day.cq.wcm.cq-wcm-core [com.day.cq.wcm.core.impl.PageManagerFactoryImpl(1839)] Could not get service from ref [com.day.cq.replication.Replicator]

21.03.2018 06:28:35.912 *ERROR* [FelixDispatchQueue] com.day.cq.cq-replication FrameworkEvent ERROR (org.osgi.framework.ServiceException: Service factory returned null. (Component: org.apache.sling.event.impl.jobs.JobManagerImpl (637)))

org.osgi.framework.ServiceException: Service factory returned null. (Component: org.apache.sling.event.impl.jobs.JobManagerImpl (637))

at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:380)

at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:247)

at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:344)

at org.apache.felix.framework.Felix.getService(Felix.java:3700)

at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:470)

at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)

at org.apache.felix.scr.impl.inject.BindMethod.getServiceObject(BindMethod.java:648)

at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)

-> 21.03.2018 06:19:53.133 *ERROR* [FelixDispatchQueue] org.apache.felix.http.whiteboard FrameworkEvent ERROR (org.osgi.framework.ServiceException: Service factory returned null. (org.apache.felix.http.base.internal.service.HttpServiceFactory@662cc759))

org.osgi.framework.ServiceException: Service factory returned null. (org.apache.felix.http.base.internal.service.HttpServiceFactory@662cc759)

-> 21.03.2018 06:20:25.008 *WARN* [OsgiInstallerImpl] org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl Unable to revert package content adobe/granite:com.adobe.granite.activitystreams.content:0.0.14. Snapshot missing.

21.03.2018 06:20:25.024 *WARN* [OsgiInstallerImpl] org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl Unable to revert package content adobe/granite:com.adobe.granite.backup.content:0.0.32. Snapshot missing.

21.03.2018 06:20:25.039 *WARN* [OsgiInstallerImpl] org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl Unable to revert package content adobe/granite:com.adobe.granite.cloudsettings.content:1.0.28. Snapshot missing.

21.03.2018 06:20:25.055 *WARN* [OsgiInstallerImpl] org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl Unable to revert package content adobe/granite:com.adobe.granite.cluster.content:0.0.36-CQ610-B0001. Snapshot missing.

21.03.2018 06:20:25.071 *WARN* [OsgiInstallerImpl] org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl Unable to revert package content adobe/granite:com.adobe.granite.contexthub.content:0.0.276-CQ610-B0001. Snapshot missing.

-> 21.03.2018 06:21:37.328 *WARN* [OsgiInstallerImpl] org.apache.jackrabbit.oak.security.user.UserImporter New member of Group 'winter-sports-admins': No such authorizable (NodeID = f4f0918c-f09e-3217-89b9-040b43601144)

21.03.2018 06:21:37.341 *WARN* [OsgiInstallerImpl] org.apache.jackrabbit.oak.security.user.UserImporter New member of Group 'surfing-admins': No such authorizable (NodeID = 89b46f84-2f98-31de-a505-9ae063311f57)

21.03.2018 06:21:37.348 *WARN* [OsgiInstallerImpl] org.apache.jackrabbit.oak.security.user.UserImporter New member of Group 'surfing-members': No such authorizable (NodeID = 27c55f02-efd9-39da-a63d-bb455048cd27)

-> 21.03.2018 06:23:56.506 *INFO* [OsgiInstallerImpl] com.adobe.granite.installer.factory.packages.impl.PackageTransformer Unable to check content package {}

java.io.IOException: Stream is not a content package. Missing 'jcr_root'.

-> 21.03.2018 06:24:41.222 *WARN* [OsgiInstallerImpl] org.apache.jackrabbit.vault.fs.io.Importer Error while committing changes. Retrying import from checkpoint at /. Retries 3/10

21.03.2018 06:24:41.718 *ERROR* [OsgiInstallerImpl] org.apache.jackrabbit.vault.fs.impl.io.DocViewSAXImporter Error during processing of /libs/cq/core/content/nav/tools/communities: javax.jcr.nodetype.ConstraintViolationException: No matching node definition found for communities

21.03.2018 06:24:41.718 *ERROR* [OsgiInstallerImpl] org.apache.jackrabbit.vault.fs.io.Importer E /libs/cq/core/content/nav/tools/communities (javax.jcr.nodetype.ConstraintViolationException: No matching node definition found for communities)

21.03.2018 06:24:41.719 *ERROR* [OsgiInstallerImpl] org.apache.jackrabbit.vault.fs.impl.io.DocViewSAXImporter Error during processing of /libs/cq/core/content/nav/tools/communities/componentguide: javax.jcr.nodetype.ConstraintViolationException: No matching node definition found for communities

21.03.2018 06:24:41.719 *ERROR* [OsgiInstallerImpl] org.apache.jackrabbit.vault.fs.io.Importer E /libs/cq/core/content/nav/tools/communities/componentguide (javax.jcr.nodetype.ConstraintViolationException: No matching node definition found for communities)

21.03.2018 06:24:41.798 *ERROR* [OsgiInstallerImpl] org.apache.jackrabbit.vault.fs.io.AutoSave error during auto save - retrying after refresh...

-> Caused by: org.apache.jackrabbit.oak.api.CommitFailedException: OakConstraint0001: /libs/cq/core/content/nav/tools[[nt:folder]]: No matching definition found for child node communities with effective type [nt:unstructured]

-> 21.03.2018 06:26:10.261 *ERROR* [FelixDispatchQueue] com.adobe.granite.workflow.core FrameworkEvent ERROR (org.osgi.framework.ServiceException: Service factory returned null. (Component: org.apache.sling.event.impl.jobs.JobManagerImpl (637)))

org.osgi.framework.ServiceException: Service factory returned null. (Component: org.apache.sling.event.impl.jobs.JobManagerImpl (637))

[Hemant_arora]

This command worked well for me migrating from 6.1 to 6.3

java -Xmx4096m -jar aem-author-6.jar -v -x crx2oak -xargs -- --load-profile segment-no-ds --skip-checkpoints

[indrasena_reddy]

Hi Hemanth,

We have used the below command for migration from 6.1 to 6.3, it was successfully completed but during bringing the instance up seeing so many errors.

java -Xmx4096m -XX:MaxPermSize=2048m -jar cq-quickstart-630-author-p4502.jar -v -x crx2oak -xargs -- --load-profile segment-fds

profile: In our project there is File Data store configuration because of which we used the profile as "segment-fds" instead of "segment-no-ds".

[Hemant_arora]

Can you share the command that you used for migration ?

[indrasena_reddy]

Command used for migration is:   java -Xmx4096m -XX:MaxPermSize=2048m -jar cq-quickstart-630-author-p4502.jar -v -x crx2oak -xargs -- --load-profile segment-fds

[Hemant_arora]

You need to specify your datastore configuration.

Add --src-datastore=/path/to/datastore to the command

Where /path/to/datastore represents the path to your File Datastore.

Refer to Performing an In-Place Upgrade

[indrasena_reddy]

Hi,

I have added --src-datastore=<datastore_path> to my command and tried it yesterday but no luck.

[Hemant_arora]

Can you try making the configurations correct ? Have you referred to the doc below ? https://helpx.adobe.com/experience-manager/6-3/sites/deploying/using/data-store-config.html

[indrasena_reddy]

Thank you Hemanth for your reply.

I have modified the configuration according to AEM 6.3,  still getting the same error messages during the AEM 6.3 startup.

Configurations

org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.config

path="crx-quickstart/repository/datastore"

minRecordLength=I"4096"

org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config

repository.home="crx-quickstart/repository"

tarmk.size="256"

customBlobStore=B"true"

Stderr Log

21.03.2018 12:43:09.711 *ERROR* [FelixStartLevel] ERROR: Bundle '360' EventDispatcher: Error during dispatch. (java.lang.NullPointerException)

java.lang.NullPointerException

at com.day.cq.dam.pim.impl.sourcing.selection.ProductAssetSelector$1.addingService(ProductAssetSelector.java:219)

Error Log

21.03.2018 12:41:07.768 *ERROR* [OsgiInstallerImpl] org.apache.jackrabbit.vault.packaging.impl.ZipVaultPackage Error during install.

javax.jcr.nodetype.ConstraintViolationException: OakConstraint0001: /libs/cq/core/content/nav/tools[[nt:folder]]: No matching definition found for child node communities with effective type [nt:unstructured]

21.03.2018 12:41:07.768 *ERROR* [OsgiInstallerImpl] com.adobe.granite.installer.factory.packages.impl.PackageTransformer Error while processing install task of day/cq561/social/component-guide:community-components-guide-pkg:1.1.69.

org.apache.jackrabbit.vault.packaging.PackageException: javax.jcr.nodetype.ConstraintViolationException: OakConstraint0001: /libs/cq/core/content/nav/tools[[nt:folder]]: No matching definition found for child node communities with effective type [nt:unstructured]

21.03.2018 12:41:34.234 *ERROR* [OsgiInstallerImpl] org.apache.jackrabbit.vault.packaging.impl.ZipVaultPackage Error during install.

javax.jcr.nodetype.ConstraintViolationException: OakConstraint0001: /libs/cq/core/content[[nt:folder]]: No matching definition found for child node heartbeat with effective type [nt:unstructured]

21.03.2018 12:41:28.292 *ERROR* [OsgiInstallerImpl] org.apache.jackrabbit.vault.fs.io.Importer Error while committing /libs/cq/reporting/widgets/source/charts/ChartData.js: javax.jcr.nodetype.ConstraintViolationException: OakConstraint0001: /libs/cq/core/content[[nt:folder]]: No matching definition found for child node heartbeat with effective type [nt:unstructured]

21.03.2018 12:41:28.292 *ERROR* [OsgiInstallerImpl] org.apache.jackrabbit.vault.fs.io.Importer Error while committing /libs/cq/reporting/widgets/source/charts: javax.jcr.nodetype.ConstraintViolationException: OakConstraint0001: /libs/cq/core/content[[nt:folder]]: No matching definition found for child node heartbeat with effective type [nt:unstructured]

21.03.2018 12:41:28.292 *ERROR* [OsgiInstallerImpl] org.apache.jackrabbit.vault.fs.io.Importer Error while committing /libs/cq/reporting/widgets/source: javax.jcr.nodetype.ConstraintViolationException: OakConstraint0001: /libs/cq/core/content[[nt:folder]]: No matching definition found for child node heartbeat with effective type [nt:unstructured]

[Kunwar]

Hey Indra,

Can you confirm if the migration was complete and you got the COMPLETED message on the CLI ?  Can you share the upgrade.log at the time of migration was done ?

[indrasena_reddy]

Hi Kunwar,

I have analyzed the upgrade log and found the following error in upgrade.log. For download the log from below link.

Upgrade file link:https://drive.google.com/file/d/1Z3V3JZiEaWqT9xCvv5edcG2qtLx-rg8T/view

• 22.03.2018 16:54:00.182 *INFO* [FelixStartLevel] com.adobe.cq.upgradesexecutor.Activator UPGRADE STARTS - StartupMode is UPDATE

22.03.2018 16:55:02.579 *ERROR* [FelixStartLevel] com.adobe.cq.upgradesexecutor.Activator OSGI installer finished and the upgrade task condition was not satisfied. Executing installed upgrade tasks but most likely one of the mandatory upgrade tasks was not executed. Please check the log file.

• 22.03.2018 16:55:38.893 *INFO* [FelixStartLevel] com.day.cq.compat.codeupgrade.impl.cq62.Cq62ProjectsCodeUpgrade Update old projects in /content/projects

22.03.2018 16:55:39.014 *INFO* [FelixStartLevel] com.day.cq.compat.codeupgrade.impl.cq62.Cq62ProjectsCodeUpgrade Done updating old projects in           /content/projects

22.03.2018 16:55:39.014 *INFO* [FelixStartLevel] com.day.cq.compat.codeupgrade.impl.cq62.Cq62ProjectsCodeUpgrade Begin adjusting projects resource types.

22.03.2018 16:55:39.016 *WARN* [FelixStartLevel] com.day.cq.compat.codeupgrade.impl.cq62.Cq62ProjectsCodeUpgrade Exception in Cq62ProjectsCodeUpgrade.run()

     java.lang.IllegalArgumentException: Value for key jcr:mixinTypes can't be put into node: [Ljava.lang.String;@5ae86d27

at org.apache.sling.jcr.resource.internal.JcrModifiableValueMap.put(JcrModifiableValueMap.java:458)

at org.apache.sling.jcr.resource.internal.JcrModifiableValueMap.put(JcrModifiableValueMap.java:47)

at com.day.cq.compat.codeupgrade.impl.projects.ProjectsAdjuster.checkResource(ProjectsAdjuster.java:90)

at com.day.cq.compat.codeupgrade.impl.projects.ProjectsAdjuster.checkResource(ProjectsAdjuster.java:96)

at com.day.cq.compat.codeupgrade.impl.projects.ProjectsAdjuster.adjustProjectResourceTypes(ProjectsAdjuster.java:64)

at com.day.cq.compat.codeupgrade.impl.cq62.Cq62ProjectsCodeUpgrade.doUpgrade(Cq62ProjectsCodeUpgrade.java:124)

at com.day.cq.compat.codeupgrade.impl.AbstractCodeUpgradeTask.run(AbstractCodeUpgradeTask.java:64)

at com.adobe.cq.upgradesexecutor.Activator.runUpgradeTasks(Activator.java:350)

at com.adobe.cq.upgradesexecutor.Activator.finishUpgradeExecution(Activator.java:220)

at com.adobe.cq.upgradesexecutor.Activator.start(Activator.java:170)

at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:697)

at org.apache.felix.framework.Felix.activateBundle(Felix.java:2227)

at org.apache.felix.framework.Felix.startBundle(Felix.java:2145)

at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1372)

at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)

at java.lang.Thread.run(Thread.java:745)

     Caused by: javax.jcr.nodetype.NoSuchNodeTypeException: Node type cq:Project does not exist

at org.apache.jackrabbit.oak.util.TreeUtil.addMixin(TreeUtil.java:217)

at org.apache.jackrabbit.oak.jcr.delegate.NodeDelegate.addMixin(NodeDelegate.java:392)

at org.apache.jackrabbit.oak.jcr.session.NodeImpl$26.performVoid(NodeImpl.java:967)

at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performVoid(SessionDelegate.java:274)

at org.apache.jackrabbit.oak.jcr.session.NodeImpl.addMixin(NodeImpl.java:956)

at org.apache.sling.jcr.resource.internal.NodeUtil.handleMixinTypes(NodeUtil.java:63)

at org.apache.sling.jcr.resource.internal.JcrModifiableValueMap.put(JcrModifiableValueMap.java:449)

... 15 common frames omitted

[Kunwar]

For a test, You can try to package the /content/projects folder and later remove all the contents under and redo the upgrade to make sure it succeeds. If it works, You can open a daycare case, I can have a look with deeper analysis on what could have happened.

My hunch is nodetype cq:Project is somehow not getting registered in the nodetype index but would need deeper analysis on this.

[indrasena_reddy]

Hi kunwar,

I have deleted the "/content/projects" from server and reran the upgrade but still seeing some more exception in upgrade log file. Also we have raised the Day care ticket.

Have you ever faced the javax.jcr.AccessDeniedException: Root node is not accessible Exception while upgrading the AEM. Now  only Cq62ProjectsCodeUpgrade activity is successful out of below 20 upgrade activities. Please let me know if there is any solution for fixing the Root node is not accessible.

[Cq561SocialContentUpgrade, Cq60SocialContentUpgrade, Cq62WorkflowContentUpgrade, CQ63ProcessingProfileConfigUpdate, Cq62TargetContentUpgrade, CQ63MetadataSchemaConfigUpdate, CQ63InboxItemsUpgrade, Cq62TargetCampaignsContentUpgrade, Cq62MediaContentUpgrade, Cq62MobileContentUpgrade, Cq62PrimaryComponentsUpgrade, CQ62Html5SmartFileUpgrade, Cq62ProjectsCodeUpgrade, CQ63MonitoringDashboardsConfigUpdate, CQ63AssetsMetadataFormsUpdate, CQ63MobileAppsContentUpgrade, CQ63ToolsMenuEntriesContentUpgrade, Cq62ConfContentUpgrade, Cq62FormsContentUpgrade, CQ63AssetsSearchFacetsUpdate, CqMergeBackupUpgrade]

• 21.03.2018 12:14:16.706 *WARN* [FelixStartLevel] com.day.cq.compat.codeupgrade.impl.cq62.Cq62WorkflowContentUpgrade Exception in Cq62WorkflowContentUpgrade.run() javax.jcr.AccessDeniedException: Root node is not accessible.at org.apache.jackrabbit.oak.jcr.session.SessionImpl$4.perform(SessionImpl.java:294)

• 21.03.2018 12:14:48.056 *INFO* [FelixStartLevel] com.adobe.cq.upgradesexecutor.Activator UPGRADE FINISHED: 1 CodeUpgradeTasks executed , 20 CodeUpgradeTasks failed (out of 25), total time about 32 seconds

[Kunwar]

I think something is messed up, Can you create a daycare case with us and I can do a deep dive and understand what’s happening here.

make sure you upload the upgrade.log and what all steps you took for this upgrade for review

[mohammadm377380]

Hi indrasena reddyc20610255​

I have also faced the same problem as yours. I went to a lot of migration iterations just to settle with the steps below.

Please check the instructions below.

Let me know if it helps you.

------ STEPS -------

Performing an In-Place Upgrade

https://helpx.adobe.com/experience-manager/6-3/sites/deploying/using/in-place-upgrade.html

1. Preparation of the AEM Quickstart jar file
   1. Stop the instance if it is running.
   2. Download the new AEM jar file and use it to replace the old one outside the crx-quickstart folder.
   3. Unpack the new quickstart jar by running:
      

2. Content Repository Migration
   
   JDK 1.7
   
   
   JDK 1.8
   

Performing The Upgrade

1. Check configurations
   • Remove crx-quickstart\launchpad\config\org\apache\jackrabbit\oak\plugins\segment folder as it is already deprecated
   • Update crx-quickstart\install folder and remove other configurations
   • Rename org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.cfg to org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config
     
     Paste this code:
     

   • Rename org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.cfg to org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.config
     
     Paste this code:
     

2. Run aem-quickstart.jar first
   • Wait until the migration is completed
     
     
3. Determining the correct upgrade start command
   
   - Just wait till the command finished starting the instance
   - You should be able to see /system/console, crx/de/index.jsp, crx/packmgr/index.jsp