My AEM publish servers suddenly stopped reading the OSGi files for configuration.
I haven't changed the files in months, years for some of them. BTW I never configure via the console.
But when I look at the system/console/configMgr console, the correct values for the run modes (publish,stage) aren't there.
It is like the server configuration has reverted back to the default values and naturally lots of our features don't work.
Has anyone seen anything like this before ? I'm going to open a help ticket anyway
Strange! My silly but possible suspect - Deployment/Package Installation overriding configs? 😉
Some of your configurations might be stored already under /apps/syste/config, for more details please watch this https://www.techinnovia.com/run-modes/ (from 7:00) video where I have demonstrated how wrong configurations are reading in OSGI.
Hope that helps!
Thank you for your reply and interesting video.
I watched it and checked my own problem server.
The run modes were o.k.
I noticed that there are lots of configs under /apps/system/config but they are all Adobe ones.
But all my configs were gone! .... i.e the ones under /apps/*/config
The other odd thing I noticed is that in crx/packmgr, the "all" package was fine but all its subpackages had this message under them. "A different version seems to be installed..."
I did find in the error.log where the configurations were being deleting. This happened during the Cloud Manager deploy that resulted in the crx/packmgr messages "A different version seems to be installed..." and after which the servers had their problems
I don't know why the OsgiInstallerImpl deleted them.
25.05.2022 21:22:31.914 *INFO* [OsgiInstallerImpl] org.apache.sling.audit.osgi.installer Deleted configuration com.xxx.soap.services.impl.ConsumeSOAPServiceImpl from resource TaskResource(url=jcrinstall:/apps/xxx/config.stage/com.xxx.soap.services.impl.ConsumeSOAPServiceImpl, entity=config:com.xxx.soap.services.impl.ConsumeSOAPServiceImpl, state=UNINSTALL, attributes=[org.apache.sling.installer.api.tasks.ResourceTransformer=:3985:4219:4220:54:, service.pid=com.xxx.soap.services.impl.ConsumeSOAPS..
Also from the error log
25.05.2022 21:22:31.213 *INFO* [JcrInstaller.1] org.apache.sling.installer.provider.jcr.impl.JcrInstaller Deleting WatchedFolder:/apps/xxx/config.author, path does not exist anymore
25.05.2022 21:22:31.213 *INFO* [JcrInstaller.1] org.apache.sling.installer.provider.jcr.impl.JcrInstaller Deleting WatchedFolder:/apps/xxx/config.author.stage, path does not exist anymore
25.05.2022 21:22:31.213 *INFO* [JcrInstaller.1] org.apache.sling.installer.provider.jcr.impl.JcrInstaller Deleting WatchedFolder:/apps/xxx/config, path does not exist anymore
25.05.2022 21:22:31.213 *INFO* [JcrInstaller.1] org.apache.sling.installer.provider.jcr.impl.JcrInstaller Deleting WatchedFolder:/apps/xxx/config.stage, path does not exist
Ok, when your OSGi configurations unexpectedly deleted in your environment you can analyze how those configurations were deleted.
1. Review the OSGi Installer Web Console
The OSGi Web Console has a plugin that allows you to view when OSGi configurations were applied and where they came from.
Go to http://host:port/system/console/osgi-installer and log in as admin.
On this user interface, it shows which bundles and configurations were applied and when. This includes files applied via the crx-quickstart/install folder and ones in JCR (Oak) repository install folders.
2. Review the error.log files for related log messages
Every time a configuration is changed, the error.log files get updated. Here are the important log messages to look out for:
Some possible causes for a configuration getting deleted are:
<?xml version="1.0" encoding="UTF-8"?> <workspaceFilter version="1.0"> <filter root="/apps/cq"/> <filter root="/content/geometrixx"/> </workspaceFilter>
To debug the root cause:
Hope that helps!
A subsequent deploy to an updated "develop" worked o.k. so I 'll have to try reproduce it again. I am the only person who installs packages on AEM prod and I do it via Cloud Manager always.
With the problem, in crx/packmgr, the "all" package was fine but all its subpackages had this message under them. "A different version seems to be installed..."
There is a subpackage, aem.ui.apps.<timestamp> which has a lot of filters including /apps/xxx ( which gets 3 folders, clientlibs,components and templates )
There is also a aem.ui.config.<timestamp> package which also has a filter /apps/xxx which grabs the osgi config files.
Is this o.k. ? It has always been like this since I moved to Cloud Manager.
Thanks Santosh,. Have you idea why I got the above crx/packmgr error messages.?
My support ticket guy thought it was because I was flicking between branches that had already been deployed earlier on Staging (i.e. branches that already had been built into packages with checksums that were recognized.)
But I've tried flicking between branches a few times on Staging and it didn't reproduce the problem.
The filter for the aem.config...<timestamp> package has a mode=update and runs after the aem.apps one.
You don't suppose the order of these could ever get mixed up?
As per my understanding, those errors occurs because one out of this https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/aem-publish-servers-aren-t... except this I don't have anything in my mind as of now.
I might proceed to deploying to Prod, but keep a backup file of the configs on Prod just in case, the same thing happens in that environment. At least I can recover then.
The problem seems to occur when we are reusing old artifacts on Stage environment. There is a pipeline directive that you can use so that you stop the pipeline reusing old artifacts so I'm going to do this. Hopefully this will resolve my problems.