AEM monitoring best practices

Hi @jkpanera!

From my perspective, there are two different areas to look at:

• What happened to your application and why did it break?
  I recommend to perform a root cause analysis to identify the reason for the outage.
  Once you have identified the root cause you can either fix it or - if that's not easily possible for some reason - add monitoring probes for relevant parts of the application stack.
• How can you pro-actively monitor your AEM instance to get an early notification if things start to go wrong?

Performing the root cause analysis

When it comes to production outages, it's always a bit hard to do a proper root cause analysis as you don't want to further affect your services uptime and performance (e. g. by increasing log levels, attaching debuggers or other analysis tools). So the ideal scenario is that you can somehow reproduce the behavior/issue on a lower environment or a production clone. Is this a one-time issue or is it reoccurring?
First touch points for an analysis should be:

• Your general, historical operating system level monitoring: CPU and memory usage, storage, network traffic, number of incoming requests, concurrent users, number of error status code responses, etc.
  Are there any abnormalities? Anything that looks suspicious? Any peaks that haven't been there before?
• Your Java Virtual Machine (JVM): heap space, garbage collection, memory usage, etc.
  How is your application running inside the JVM? Could there me a memory leak? Is garbage collection working as expected? Does the JVM have enough memory? You may want to create and analyze garbage collection logs, heap dumps, thread dumps.
• Your logs from shortly before the outage: Are there any outstanding messages/errors? Are there any jobs running that could influence the instances behavior? How is the amount of written logs compared to other days? The number of errors and exceptions?

With Dynatrace in place, you should have deep insight into the JVM and the application and probably have a good starting point for your root cause analysis.
Looking at the error messages from your logs, I'm not sure if these are related in terms of a cause or if they are a consequence of the outage.

Looking at the monitoring part of your question, you should first identify the root cause and add according monitoring probes that will notify you on all aspects that initially may have lead to the outage. There is no one-size-fits-all monitoring concept as most outages are - in my experience - not caused by AEM product code but by projects custom application development inside the AEM framework/stack.

However, I can provide a generic list of monitoring points that I usually recommend:

• Standard system level monitoring, such as hardware, filesystem, cpu load, memory usage, network traffic, etc.
• JVM monitoring, such as heap space, threads, garbage collection, etc.
• AEM specific monitoring, such as status of OSGI bundles, components, services; status of replication queues, workflows, job queues; log files with regards to response status codes, number of (concurrent) requests, specific errors/exceptions, amount of errors, exceptions, size of log files; repository size; security checks according to official Security Checklists.
• Application specific monitoring - checking relevant probes of you projects custom application.

Apart from certain JMX checks, please also take a look at the existing and custom Sling Health Checks for in-depth application monitoring. See also Building Health Checks for AEM and this article for further details.

Hope that helps!