App Dynamics does not work with AEM : Error - Error 503: SRVE0295E: Error reported: 503

[rajatkumar]

Hi All,

I am trying to get App Dynamics monitoring tool working with AEM but whenever I am adding the -javaagent:/opt/adaajnat/javaagent.jar.
AEM is giving me  Error - Error 503: SRVE0295E: Error reported: 503 on all lc pages.

Application Stack details :
-AEM On-Premise 6.5
-AEMForms-6.5.0-0040

-WebSphere Base 9.0.5.11

-MSSQL 2019 -CPU 8x2394 MHz (Intel Xeon Gold 6240R CPU @ 2.40GHz - 24 GB RAM
-Linux RedHat_79
-App Dynamics version v22.1.0-3252

Steps to recreate issue:- 
in JVM arguments, please add the following argument : -javaagent:/opt/adaajnat/javaagent.jar Given App dynamics agent is installed in /opt/adaajnat directory. Restart the JVM to take effect. After a restart, AEM all lc pages are unreachable although JEE side works fine. Error - Error 503: SRVE0295E: Error reported: 503



We reached out to the App Dynamics community and seems like to make the App Dynamics Agent work, AEM required an update of its allowed list in the deserialization firewall.
so, after tweaking the below parameters, got it working.

3 Step configurations change :

WAS level settings
Go to Application servers > cluster > Process definition > Java Virtual Machine and update below parameters to Generic JVM arguments
-Javaagent:/opt/adaajnat javaagent.jar
-Dappdynamics.agent applicationName= <application-name>
-Dappdynamics.bciengine should implement.new.interfaces=false

AEM system configuration level settings:

Access the below URL with admin credentials - http://localhost:8080/lc/system/console/configMgr/com.adobe.cq.deserfw.lmplDeserialization Firewallimpl
Configuration name com.adobe.cq deserfw.impDeserialization Firewallimpl.name
Please add com.singularity. under com.adobe.cq deserfw.impl.Deserialization Firewallimpl.firewall.desertalization.whitelist.name
Save and return

Sling properties settings:
Access the server and access AEM Installation files through command line.
Goto the path: /opt/adobe/adobe_experience_manager_Forms/CTXrepository/launchpad
Edit sling properties file For property - org.osgiframework.bootdelegation, add the following class path-com singularity.*.
Restart JVMS

I would like to understand what com.adobe.cą.deserfw.Impl Deserialization Firewallimpl.name config affects and can it have any negative impact on our Application since we added com singularity.
Similarly clarify the significance of org.osgl.framework.bootdelegation property under sling properties and adding com.singularity.* in class path.


Thanks,
Rajat

[Pulkit_Jain_ · Accepted solution]

@rajatkumar 

Thanks for sharing the details!

AEM uses a deserialization firewall, and only explicitly white-listed/Black-listed classes are allowed/disallowed to be de-serialized. As per the AppDynamics blogs[0][1], com.singularity.* classes have to be whitelisted when a particular AppDynamics config (MIDC) is set.

Also, a similar help doc[2] suggests that WebSphere uses Equinox as its OSGi container. Sometimes, you may need to add the Java agent packages to the OSGi boot delegation system property: com.singularity.*

We don't have any reference implementation documents for configuring App Dynamics with AEM Forms, but these configs might be required for the integration. Still, if you want more clarity, I suggest you log a support ticket, and we will try to get more insights from the product team.

[0] https://community.appdynamics.com/t5/Java-Java-Agent-Installation-JVM/com-singulartity-exception-in-...

[1] https://docs.appdynamics.com/appd/22.x/latest/en/application-monitoring/install-app-server-agents/ja...

[2] https://docs.appdynamics.com/appd/22.x/latest/en/application-monitoring/install-app-server-agents/ja...

[rajatkumar]

Hi @Pulkit_Jain_ , 
Thanks for shedding some light on the configuration required for AEM forms integration with App Dynamics.
While researching the App Dynamics community, the steps seemed valid, and it worked in one of our lower environments.
But It would be really helpful if we could get some more clarity and trade offs if any from the product team itself before applying this config given the risks in higher environments. 

We have raised a ticket with Adobe Support for the same. Would be a great deal if Adobe could provide us with a Hot-fix for the same so that a manual tweak in the higher environment would not be needed.

[Pulkit_Jain_]

@rajatkumar sure, please DM the ticket#

[rajatkumar]

@Pulkit_Jain_ sent a mailer communication!!
Thank you for looking into this