Performance Degradation after AEM 6.5 LTS Upgrade

As this is happening after Java 21 upgrade check for the JVM flags before and after.

also analyze the heap dump with load run and modify the GC params accordingly this may fix or atleast better the performance.

​@akhilraj We do not currently see Adobe documentation indicating a known general regression in AEM 6.5 LTS + Java 21 that would by itself explain the increase from sub-second responses to ~11 seconds under load. Adobe does, however, recommend explicit performance validation after upgrade and notes that Java 17/21 GC settings should be tuned for optimal performance rather than relying on defaults. Based on the current behavior, this appears more consistent with a load-sensitive tuning or integration issue involving JVM behavior, custom code, DB/Redis concurrency, or request/query execution under high load. [1] [2] [3]

[1] https://experienceleague.adobe.com/en/docs/experience-manager-65-lts/content/release-notes/service-pack/ga

[2]  https://experienceleague.adobe.com/en/docs/experience-manager-65-lts/content/implementing/deploying/practices/best-practices-for-performance-testing

[3] https://experienceleague.adobe.com/en/docs/experience-manager-65-lts/content/implementing/deploying/configuring/configuring-performance

Our platform team tried all options, but couldnt figure out what is blocking the api calls, and increasing the response time. We are not observing any error n publisher and dispatcher, so couldnt identify the rootcause as well.

​@akhilraj 

At present, since the APIs are functionally working and no explicit errors are being recorded in Publisher or Dispatcher logs, the issue appears to be a load-related performance bottleneck rather than a functional application error. In these situations, Adobe recommends collecting runtime diagnostics during the performance event—including thread dumps, GC logs, system metrics, request timing, and backend dependency metrics—because latency under high concurrency may be caused by thread contention, JVM behavior, connection-pool waits, or downstream service delays, none of which necessarily produce visible application errors. Once these diagnostics are captured during the affected load window, you can be correlated to identify the actual blocking point and root cause.

Sources:

• Best Practices for Performance Testing
• Performance Optimization
• Adobe Experience Manager: Troubleshooting Performance Degradation – Memory/CPU spike

Hi ​@akhilraj 

Java 21 uses a completely modern, NIO-based underlying Socket implementation. While it is more efficient, it exposes lock contention issues in older third-party connection pools under high load.

• The Redis/DB Clients: If you are using Jedis (which relies on Apache Commons Pool), Lettuce, or HikariCP for your database, and you did not update these dependencies during the Java 21 migration, the older pooling mechanisms can suffer from severe synchronized block contention.

• SSL/TLS Handshakes: If your connections to the Azure Database or Redis are over SSL/TLS, Java 21 introduces stricter security defaults. If connection pooling isn't configured correctly and connections are being rapidly recreated instead of reused, the CPU overhead of Java 21's TLS handshakes at 18,000 concurrent requests will cause an immediate bottleneck.

• Action: Verify that your Redis client and JDBC drivers are updated to their latest, Java 21-certified versions.

Hi ​@akhilraj ,

We hit something similar after a Java upgrade on an AEM 6.5 instance, not identical but close enough that some of these should be worth checking.
The jump from under 1 second to 11 seconds under the same load is a pretty classic sign of thread contention or connection pool exhaustion rather than raw compute performance. Java 21 handles virtual threads differently and some of the underlying Jetty and Felix HTTP settings that worked fine on older JVMs can become bottlenecks under high concurrency when you upgrade.
First thing I’d check is your Sling thread pool configuration in the OSGi console. Specifically the minimum and maximum thread counts for the default thread pool. These defaults are often too conservative for 18,000 concurrent users and don’t automatically scale with the JVM upgrade. Under heavy load you’ll see requests queuing behind available threads which shows up as that kind of dramatic latency spike.
Second thing is your Redis connection pool settings. If your APIs are fetching from database and caching in Redis on every homepage load, under 18k concurrent users you can saturate the Redis connection pool fast. Check your Jedis or Lettuce pool configuration maxTotal, maxIdle, and the timeout values. If connections are being borrowed and not returned fast enough you’ll see exactly this kind of latency pattern where everything works functionally but response times blow up under load.
Third is the Azure OIDC token validation piece. If every API request is going through OIDC token validation on the AEM side, that validation call can become a bottleneck under load if it’s not being cached. After an upgrade it’s worth checking whether your token cache is still wired correctly or if it got reset to defaults.
For Java 21 specifically, check if G1GC is still your garbage collector and look at GC pause times in your logs during the load test window. Sometimes a JVM upgrade shifts GC behavior in ways that show up only under sustained load.
Would be helpful to know if the degradation is consistent throughout the 30 minutes or if it starts fine and degrades over time that would point more toward memory pressure or connection pool exhaustion vs a configuration issue that’s constant from the start.