AEM Site 404 performance issue

[Ian_2021]

I had ran a load test for 404 response on a 8 core 16GB AEM publish instance. 

The request urls are something like https://www.myhost.com/content/abc/${__Random(1,1000000)}/,
and the response is set in /apps/sling/servlet/errorhandler/404.jsp:

<%
    response.setStatus(404);
    response.setContentType("text/html");
    response.getWriter().write("<h1>Not Found</h1> <p>The requested URL was not found on this server.</p>");
%>

 
AEM can handle about 700 requests/second, which seems quite low for this simple scenario.

Does anybody konw how to improve the 404 throughtout for AEM instance?

[Mohit_KBansal]

This is a strange scenario where you are trying to measure the performance of the server by putting a load of 404 requests. In such case, the dispatcher/CDN cache can not be leveraged and requests will always reach to the publisher.

Ideally, in the case of 404, the publisher should do only resource resolution, and return a very lightweight 404 response. Your code seems to be doing the same.

Next, you need to make sure to apply best practices [1] [2] [3] to improve the performance of publisher servers.

I also suggest finding common URL patterns, which are hitting publishers with 404 response code and adding such patterns in your dispatcher farm filter or CDN filter to avoid unwanted load on publishers.

[1] https://experienceleague.adobe.com/docs/experience-manager-65/deploying/configuring/performance-guid...

[2] https://experienceleague.adobe.com/docs/experience-manager-65/deploying/practices/best-practices-for...

[3] https://blogs.perficient.com/2020/02/19/website-performance-with-the-adobe-experience-cloud/ 

[Ian_2021]

I know page should be cached in dispatcher and cdn. 

I'm worrying some malicious site scan requests may cause publish  cpu exhaustion.

[Mohit_KBansal · Accepted solution]

Understood. I recommend the following in such a scenario:

1. Make sure to apply recommended best practices to the publisher [1] [2] [3]
2. Configure DDoS Protection [4] on CDN, this will block any attempt of attack on the server.
3. Continuously scan your publisher logs, to find common 404 patterns and add these patterns to the dispatcher's farm filter or CDN.

[1] https://experienceleague.adobe.com/docs/experience-manager-65/deploying/configuring/performance-guid...

[2] https://experienceleague.adobe.com/docs/experience-manager-65/deploying/practices/best-practices-for...

[3] https://blogs.perficient.com/2020/02/19/website-performance-with-the-adobe-experience-cloud/ 

[4] https://aws.amazon.com/shield/ddos-attack-protection/ 

[Ian_2021]

   Continuously scan your publisher logs, to find common 404 patterns and add these patterns to the dispatcher's farm filter or CDN.

   -- this won't work for us since the pattern change every time.

   Configure DDoS Protection [4] on CDN, this will block any attempt of attack on the server.

   -- that should work, but we also want to improve the throughtout of AEM publish for handling 404 request.  

And it's strange that AEM publish can only hanle about 700 requests/second, since most web framework should be able to handle over ten thousand request/second in this scenario

[Mohit_KBansal]

@Ian_2021 Scanning logs is my personal recommendation based on capacity. I find it helpful to improve the health of the server.

Where did you find about publisher throughput of 700 req/sec ? Is it documented in Adobe documentation? 

Please note that you can do vertical/horizontal scaling to increase throughput. If you are on AEMaaCS, scaling is OOB.

[Ian_2021]

Where did you find about publisher throughput of 700 req/sec ? Is it documented in Adobe documentation? 

-- On my own load testing for publish on 404 scenario. Not documented in Adobe documentation.

Please note that you can do vertical/horizontal scaling to increase throughput. If you are on AEMaaCS, scaling is OOB.

-- scaling is expensive, specially for horizontal scaling, since it involves AEM licenses.

[Mohit_KBansal]

I agree with @arunpatidar , 700 req/sec is a sign of attack. DDoS should be able to handle it.

If you have applied all best practices and looking for better throughput, you have to consider scaling options.

[arunpatidar]

Hi,

You can setup 404 page at dispatcher/apache level

Please check

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/aem-dispatcher-errordocume... 

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/dispatcher-redirection-and... 

[Ian_2021]

I  can setup 404 page at dispatcher/apache level, but AEM publish have to return 404 response code before apache can handle it.

[arunpatidar]

approx 700 request/second to 404 page would be signature of attack until you have issue with code/design.

You should rely on DDoS/attack protection.