AEM publish servers aren't using the configuration in my OSGi xml files

[fionas76543059]

hi folks,

My AEM publish servers suddenly stopped reading the OSGi files for configuration.

I haven't changed the files in months, years for some of them. BTW I never configure via the console.

But when I look at the system/console/configMgr console, the correct values for the run modes (publish,stage) aren't  there.

It is like the server configuration has reverted back to the default values and naturally lots of our features don't work.

Has anyone seen anything like this before ? I'm going to open a help ticket anyway

thanks

[SantoshSai]

Hi @fionas76543059 

Strange! My silly but possible suspect - Deployment/Package Installation overriding configs?

Some of your configurations might be stored already under /apps/syste/config, for more details please watch this https://www.techinnovia.com/run-modes/ (from 7:00) video where I have demonstrated how wrong configurations are reading in OSGI.

Hope that helps!

Regards,
Santosh

[fionas76543059]

hi Santosh,

Thank you for your reply and interesting video.

I watched it and checked  my own problem server.

The run modes were o.k.

I noticed that there are lots of configs under /apps/system/config but they are all Adobe ones.

But all my configs were gone!  .... i.e the ones under /apps/*/config

The other odd thing I noticed is that in crx/packmgr, the "all" package was fine but all its subpackages had this message under them. "A different version seems to be installed..."

[fionas76543059]

I did find in the error.log  where the configurations were being deleting. This happened during the Cloud Manager deploy that resulted in the crx/packmgr messages "A different version seems to be installed..." and after which the servers had their problems

I don't know why the OsgiInstallerImpl deleted them.

25.05.2022 21:22:31.914 *INFO* [OsgiInstallerImpl] org.apache.sling.audit.osgi.installer Deleted configuration com.xxx.soap.services.impl.ConsumeSOAPServiceImpl from resource TaskResource(url=jcrinstall:/apps/xxx/config.stage/com.xxx.soap.services.impl.ConsumeSOAPServiceImpl, entity=config:com.xxx.soap.services.impl.ConsumeSOAPServiceImpl, state=UNINSTALL, attributes=[org.apache.sling.installer.api.tasks.ResourceTransformer=:3985:4219:4220:54:, service.pid=com.xxx.soap.services.impl.ConsumeSOAPS..

Also from the error log

25.05.2022 21:22:31.213 *INFO* [JcrInstaller.1] org.apache.sling.installer.provider.jcr.impl.JcrInstaller Deleting WatchedFolder:/apps/xxx/config.author, path does not exist anymore
25.05.2022 21:22:31.213 *INFO* [JcrInstaller.1] org.apache.sling.installer.provider.jcr.impl.JcrInstaller Deleting WatchedFolder:/apps/xxx/config.author.stage, path does not exist anymore
25.05.2022 21:22:31.213 *INFO* [JcrInstaller.1] org.apache.sling.installer.provider.jcr.impl.JcrInstaller Deleting WatchedFolder:/apps/xxx/config, path does not exist anymore
25.05.2022 21:22:31.213 *INFO* [JcrInstaller.1] org.apache.sling.installer.provider.jcr.impl.JcrInstaller Deleting WatchedFolder:/apps/xxx/config.stage, path does not exist

[SantoshSai]

Hi @fionas76543059 

Ok, when your OSGi configurations unexpectedly deleted in your environment you can analyze how those configurations were deleted.

1. Review the OSGi Installer Web Console

The OSGi Web Console has a plugin that allows you to view when OSGi configurations were applied and where they came from.

• Go to http://host:port/system/console/osgi-installer and log in as admin.

• On this user interface, it shows which bundles and configurations were applied and when. This includes files applied via the crx-quickstart/install folder and ones in JCR (Oak) repository install folders.

2. Review the error.log files for related log messages

Every time a configuration is changed, the error.log files get updated. Here are the important log messages to look out for:

• JCR Installer observes install folders removed from the JCR repository
• JCR Installer observes the configurations being deleted (In your case)
• OsgiInstallerImpl installs configurations
  
  

Some possible causes for a configuration getting deleted are:

• Package filter that causes a folder to get deleted.  For example if a package filter included all of the folder /apps/cq instead of specific folders under it.  e.g.

<?xml version="1.0" encoding="UTF-8"?>
<workspaceFilter version="1.0">
<filter root="/apps/cq"/>
<filter root="/content/geometrixx"/>
</workspaceFilter>

• Another cause might be a user accidentally deleting the configuration
• A user modifying the configuration via the /system/console/configMgr UI

To debug the root cause:​

• Search for package installations surrounding these log messages you might look for package install.  Search the log for org.apache.jackrabbit.vault.packaging.impl.ZipVaultPackage Extracting
• Search the access.log for a user posting to /system/console/configMgr or /crx/de during the same timeframe as when the configurations were deleted.

Hope that helps!

Regards,
Santosh

[fionas76543059]

Thanks Santosh,

A subsequent deploy to an updated "develop" worked o.k. so I 'll have to try reproduce it again. I am the only person who installs packages on AEM prod and I do it via Cloud Manager always.

With the problem, in crx/packmgr, the "all" package was fine but all its subpackages had this message under them. "A different version seems to be installed..."

There is a subpackage, aem.ui.apps.<timestamp> which has a lot of filters including /apps/xxx ( which gets 3 folders, clientlibs,components and templates )

There is also a aem.ui.config.<timestamp> package which also has a filter /apps/xxx which grabs the osgi config files.

Is this o.k. ? It has always been like this since I moved to Cloud Manager.

thanks

Fiona

[SantoshSai]

@fionas76543059 

That seems to be OK, since that has the maven archetype standard structure itself. 

[fionas76543059]

Thanks Santosh,. Have you idea why I got the above crx/packmgr error messages.?

My support ticket guy thought it was because I was flicking between branches that had already been deployed earlier on Staging (i.e. branches that already had been built into packages with checksums that were recognized.)

But I've tried flicking between branches a few times on Staging and it didn't reproduce the  problem.

[fionas76543059]

The filter for the aem.config...<timestamp> package has a mode=update and runs after the aem.apps one.

You don't suppose the order of these could ever get mixed up?

[SantoshSai]

@fionas76543059 
As per my understanding, those errors occurs because one out of this https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/aem-publish-servers-aren-t... except this I don't have anything in my mind as of now.

[fionas76543059]

Thanks Santosh.

I might proceed to deploying to Prod, but keep a backup file of the configs on Prod just in case, the same thing happens in that environment. At least I can recover then.

[fionas76543059]

The problem seems to occur when we are reusing old artifacts on Stage environment. There is a pipeline directive that you can use so that you stop the pipeline reusing old artifacts so I'm going to do this. Hopefully this will resolve my problems.