OSGI Bundles/JAR's not getting synced in AEM 6.2 cold standby instance.

[Gunalan_V]

Hi Friends,

I'm using AEM 6.2 cold standby instance and whenever I upload any bundles/jar in primary author those are not getting synced/reflected in cold standby instance.

I'm not seeing any specific errors in syncing and also as per the logs I see the sync is not getting initiated. However there are few errors thrown when opening cold standby OSGI console.

Can some one please let me know how to resolve this if you have already faced this issue?

Note: All other contents are getting synced from primary to cold standby only issue is with OSGI bundles/JAR.

Error Logs (error.log) in cold standby instance: (While opening OSGI console)

    08.08.2016 17:30:33.261 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://283.0:0/OSGI-INF/metatype/com.day.cq.workflow.compatibility.AbsoluteTimeoutHandlerProxyImpl.xml
08.08.2016 17:30:33.262 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://283.0:0/OSGI-INF/metatype/com.day.cq.workflow.compatibility.CQDynamicParticipantExecutor.xml
08.08.2016 17:30:33.262 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://283.0:0/OSGI-INF/metatype/com.day.cq.workflow.compatibility.CQWorkflowExtProcessProxy.xml
08.08.2016 17:30:33.262 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://283.0:0/OSGI-INF/metatype/com.day.cq.workflow.compatibility.CQWorkflowProcessRunner.xml
08.08.2016 17:30:33.262 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://283.0:0/OSGI-INF/metatype/com.day.cq.workflow.compatibility.ScriptContextProviderImpl.xml
08.08.2016 17:30:33.262 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://283.0:0/OSGI-INF/metatype/com.day.cq.workflow.impl.CQPayloadMap.xml
08.08.2016 17:30:33.299 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://406.0:0/OSGI-INF/metatype/com.adobe.aemds.guide.cache.CacheManager.xml
08.08.2016 17:30:33.299 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://406.0:0/OSGI-INF/metatype/com.adobe.aemds.guide.cache.impl.DefaultCacheStoreProvider.xml
08.08.2016 17:30:33.303 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://420.0:0/OSGI-INF/metatype/com.adobe.cq.media.publishing.impl.servlets.OriginalImageServlet.xml
08.08.2016 17:30:33.320 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://430.0:0/OSGI-INF/metatype/com.adobe.cq.social.site.endpoints.impl.SiteOperationService.xml
08.08.2016 17:30:33.320 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://430.0:0/OSGI-INF/metatype/com.adobe.cq.social.site.impl.AnalyticsComponentConfigurationServiceImpl.xml
08.08.2016 17:30:33.321 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://430.0:0/OSGI-INF/metatype/com.adobe.cq.social.site.impl.SiteConfiguratorImpl.xml
08.08.2016 17:30:33.321 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://430.0:0/OSGI-INF/metatype/com.adobe.cq.social.user.endpoints.impl.CommunityUserOperationService.xml
08.08.2016 17:30:33.321 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://430.0:0/OSGI-INF/metatype/com.adobe.cq.social.user.endpoints.impl.UserProfileOperationService.xml
08.08.2016 17:30:33.322 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://430.0:0/OSGI-INF/metatype/com.adobe.cq.social.user.impl.transport.HttpToPublisherImpl.xml
08.08.2016 17:30:33.322 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://434.0:0/OSGI-INF/metatype/com.adobe.cq.social.enablement.services.impl.EnablementContentReplicationFilterFactory.xml
08.08.2016 17:30:33.322 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://434.0:0/OSGI-INF/metatype/com.adobe.cq.social.enablement.services.impl.EnablementSiteServiceImpl.xml
08.08.2016 17:30:33.331 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://468.0:0/OSGI-INF/metatype/com.adobe.forms.test.service.impl.FormDataXMLProviderRegistryImpl.xml
08.08.2016 17:30:33.332 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://476.0:0/OSGI-INF/metatype/com.adobe.acs.commons.fam.DeferredActions.xml
08.08.2016 17:30:33.337 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://479.1:0/OSGI-INF/metatype/com.nnanet.dps.core.services.impl.DPSJsonBinderImpl.xml
08.08.2016 17:30:33.338 *ERROR* [qtp446715529-238] org.apache.felix.metatype Missing element AD in element OCD : bundle://479.1:0/OSGI-INF/metatype/com.nnanet.dps.core.services.impl.HeaderProfileUser.xml

[Opkar_Gill]

Hi Gunalan,

this seems to be a known issue with 6.2, can you raise a daycare ticket please? The errors should not affect the running of your instance, they are more annoying than fatal.

Did you set your tarmk log to debug when installing the gundles/jar's and see a sync happening?

Regards,

Opkar

[Gunalan_V]

Thanks for your response Opkar, 

I have validated using debug logging mode however I'm not seeing any sync related logs while installing bundles/jar's in primary. (But below logs are occurring every 3 seconds in primary and cold standby)

This bundle sync issue also a known issue in AEM 6.2?

tarmk-coldstandby.log (In Primary)

08.08.2016 20:45:14.262 *DEBUG* [nioEventLoopGroup-3-9] org.apache.jackrabbit.oak.plugins.segment.standby.store.CommunicationObserver got message 'h' from client a37c5502-5e34-48fe-8c6a-e2b5f8d0dac4

tarmk-coldstandby.log (In Cold Standby)

08.08.2016 20:45:14.248 *DEBUG* [nioEventLoopGroup-2372-1] org.apache.jackrabbit.oak.plugins.segment.standby.client.StandbyClientHandler sending head request
08.08.2016 20:45:14.249 *DEBUG* [nioEventLoopGroup-2372-1] org.apache.jackrabbit.oak.plugins.segment.standby.client.StandbyClientHandler did send head request
08.08.2016 20:45:14.250 *DEBUG* [nioEventLoopGroup-2372-1] org.apache.jackrabbit.oak.plugins.segment.standby.codec.RecordIdDecoder received type 0 with id 8ac6b35e-efd4-4912-afb5-831f9dd4c16f.ff0d
08.08.2016 20:45:14.250 *DEBUG* [nioEventLoopGroup-2372-1] org.apache.jackrabbit.oak.plugins.segment.standby.client.StandbyClientHandler updating current head to 8ac6b35e-efd4-4912-afb5-831f9dd4c16f.ff0d
08.08.2016 20:45:14.250 *DEBUG* [pool-2643-thread-1] org.apache.jackrabbit.oak.plugins.segment.standby.client.SegmentLoaderHandler new head id 8ac6b35e-efd4-4912-afb5-831f9dd4c16f.ff0d
08.08.2016 20:45:14.250 *DEBUG* [pool-2643-thread-1] org.apache.jackrabbit.oak.plugins.segment.standby.client.SegmentLoaderHandler updated head state successfully: true in 0ms.

Thanks,

Gunalan V

[Opkar_Gill]

Hi Gunalan,

From the docs it's only when installing a hot fix or service pack that we are asked to rebuild the standby, so for normal bundle installs it should work.

Bundle sync should not be an issue, an interesting local test would be to install a bundle on your local instance, wait a few minutes and then take down your primary and bring up the standby. The bundle should be there. The only thing unlikely to happen in the standby is that the bundle will probably just be in an installed state. Did you even see the bundle in the standby?

Regards,

Opkar

[Gunalan_V]

Opkar,

I tried installing one of the jar's in primary author how ever there is no sync happening to standby.

No sync logs has been noticed in tarmk-coldstandby.log also as per your suggesstion after few minutes I have changed runmode to primary and validated and still I couldn't see the newly installed jar.

Could you please let me know what might be the possible issue?

Thanks,

Gunalan V

[Opkar_Gill]

Hi Gunalan,

I just tested this as well and can confirm your findings, bundles installed in the Felix console do not appear to get sync'd.

I have consulted with colleagues and the reason for this is most probably the following: Cold Standby is used to sync the repository, so if you upload a package it will get saved to the repository and sync'd to the cold standby instance: Including bundles. If you make any changes to the OSGI configurations, these also get saved to the repository and so are sync'd.

When you directly upload a bundle jar to the Felix console, then this get's saved to the filesystem: "crx-quickstart/launchpad/Felix" and not the repository, and hence is not sync'd. So, if you ever have to instal any bundles then they should always be part of a package to ensure they get sync'd, otherwise in the case of failover, you would have to manually apply any changes made directly to bundles in Felix. If you see hot fixes and SP's, these are always packages, so would get sync'd, it seems this use case is not covered in the cold standby setup. If I hear anything else, I'll update this thread.

Regards,

Opkar

[Gunalan_V]

Sure Opkar and Thanks for your detailed response to my question. Much Appreciated :)

I understood better now. However I will try to install our application bundles through package manager and will let you know if I face any issues.

Regards,

Gunalan V