AEMaaCS DDoS Attack Analysis

[Rohan_Garg]

Hi All,

I have the below warning and error logs from our PROD environment - 

A number of requests are being simulated to get the JCR content for the page - 

/content/brandsA/us/en/ALFA_DATA.html

/content/brandsA/us/en/ALFA_DATA/alfacgiapi.html

/content/brandsA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html

/content/brandsA/us/en/ALFA_DATA/alfacgiapi/py.alfa.html

/content/brandsA/us/en/ALFA_DATA/alfacgiapi/bash.alfa.html

/content/brandsA/us/en/ALFA_DATA/alfacgiapi/aspx.aspx.html

All these requests are either generating 404/Resource not found exception.

As per the security checklist for DDoS attack in AEM Systems (https://experienceleague.adobe.com/docs/experience-manager-65/administering/security/security-checkl...) - 

• By requesting a content page with an unlimited number of URLs, The URL can include a handle, some selectors, an extension, and a suffix - any of which can be modified.

  For example, .../en.html can also be requested as:

  • .../en.ExtensionDosAttack
  • .../en.SelectorDosAttack.html
  • .../en.html/SuffixDosAttack

  All valid variations (e.g. return a 200 response and are configured to be cached) will be cached by the dispatcher, eventually leading to a full file system and no service for further requests.

What is the best way to handle these requests ?

Blocking requests with string ALFA_DATA in dispatcher seems to be a temporary fix as the string can always change. 

@arunpatidar, @Jörg_Hoh, @markus_bulla_adobe, @Mohit_KBansal, @B_Sravan

12.12.2022 04:14:13.341 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818453338] GET /content/brandA/us/en/ALFA_DATA.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA.html not found
12.12.2022 04:14:13.341 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818453338] GET /content/brandA/us/en/ALFA_DATA.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA.html not found
12.12.2022 04:14:15.189 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818455186] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi.html not found
12.12.2022 04:14:15.189 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818455186] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi.html not found
12.12.2022 04:14:17.174 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818457170] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html not found
12.12.2022 04:14:17.174 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818457170] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html not found
12.12.2022 04:14:18.966 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818458961] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/py.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/py.alfa.html not found
12.12.2022 04:14:18.966 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818458961] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/py.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/py.alfa.html not found
12.12.2022 04:14:20.430 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818460427] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/bash.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/bash.alfa.html not found
12.12.2022 04:14:20.430 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818460427] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/bash.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/bash.alfa.html not found
12.12.2022 04:14:22.381 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818462377] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/aspx.aspx.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/aspx.aspx.html not found
12.12.2022 04:14:22.381 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818462377] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/aspx.aspx.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/aspx.aspx.html not found
12.12.2022 09:15:07.711 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *ERROR* [34.93.61.237 [1670836507686] POST /content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.servlets.post.impl.operations.ModifyOperation Unable to create resource named bdl in /content/brandA/us/en/content
12.12.2022 09:15:07.717 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *WARN* [34.93.61.237 [1670836507686] POST /content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.servlets.post.impl.SlingPostServlet Exception while handling POST on path [/content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html] with operation [org.apache.sling.servlets.post.impl.operations.ModifyOperation]
12.12.2022 09:15:07.717 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *WARN* [34.93.61.237 [1670836507686] POST /content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.servlets.post.impl.SlingPostServlet Exception while handling POST on path [/content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html] with operation [org.apache.sling.servlets.post.impl.operations.ModifyOperation]
12.12.2022 09:15:08.627 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *ERROR* [34.93.61.237 [1670836508622] POST /content/brandA/us/en/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.servlets.post.impl.operations.ModifyOperation Unable to create resource named bdl in /content/brandA/us/en/content
12.12.2022 09:36:31.743 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [sling-default-3-Registered Service.5118] com.day.cq.rewriter.linkchecker.impl.LinkCheckerTask Checked URL https://www.browndoglodge.com/content/brandA/us/en/ALFA_DATA/alfacgiapi: 404 (invalid)
12.12.2022 09:36:31.788 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [sling-default-3-Registered Service.5118] com.day.cq.rewriter.linkchecker.impl.LinkCheckerTask Checked URL https://www.brandsA.com/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa/: 404 (invalid)
12.12.2022 10:36:32.315 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [sling-default-3-Registered Service.5118] com.day.cq.rewriter.linkchecker.impl.LinkCheckerTask Checked URL https://www.brandsA.com/content/brandA/us/en/ALFA_DATA/alfacgiapi: 404 (invalid)
12.12.2022 10:36:32.356 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [sling-default-3-Registered Service.5118] com.day.cq.rewriter.linkchecker.impl.LinkCheckerTask Checked URL https://www.brandsA.com/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa/: 404 (invalid)

[Jörg_Hoh]

It is technically possible to create many valid urls for a single page. I see 3 options:

1. you ignore this problem; and that's an approach many do, and there is hardly anyone to blame for it.
2. You start to implement a strict ruleset on dispatcher which tries to prevent such situations.
3. You implement your AEM code to reject such requests.

In general filling the dispatcher disk is a sound approach to impact any AEM system; but there are many ways to DDOS any application (AEM is not especially susceptible to such), so investing much effort in preventing one vector is probably a waste of resources. It should just be impossible (or at least hardly possible) that a single user is able to DOS an instance. But that should be done on a CDN layer.

[Rohan_Garg]

Hi @Jörg_Hoh, Thanks for your reply!

I will explore more on Point #2 in implementing a strict ruleset by denying dispatcher to serve these requests. But like you said there are many possible combinations.

Adobe's documentation has also suggested - 
Use a firewall to filter access to your instance.

• The use of an operating system level firewall is necessary in order to filter access to points of your instance that might lead to denial of service attacks if left unprotected.

I will try this out. Thanks again, I will update the fix as I progress on this.

[Jörg_Hoh]

Just looking again: You can probably get rid of a lot of such noise when you deny any POST request on the dispatcher (just if your application does not use POST, of course).

[Rohan_Garg]

We do use POST requests as well in our application. However, almost all the bad requests are GET based. I have blocked the requests based on selectors. The extensions are already limited to html.

#Block Bad Requests
/0108 { /type "deny" /selectors '(php|up|bak|alfa)' /extension 'html' }

 Most of the bad requests have the below URLs (Content Path + Below Endpoints) -

/wp-admin
/wp-content

/wp-admin/admin.php.html
/radio.php.html
/config.bak.php.html
/wp-content/db-cache.php.html
/wp-content/plugins/ubh/up.php.html
/en/shells.php.html
/wp-content/shell20211028.php.html/ALFA_DATA or alfacgiapi
.png.html
db.php.html

Let me know if you think there is a better way to block out these above requests.
In default_filters.any there is the below rule -

# This rule allows content to be access
/0010 { /type "allow" /extension '(css|eot|gif|ico|jpeg|jpg|js|gif|pdf|png|svg|swf|ttf|woff|woff2|html|mp4|mov|m4v)' /path "/content/*" }  # disable this rule to allow mapped content only

Do you think we should disable this rule to only allow mapped content to be accessed ?

[Jörg_Hoh]

Currently blocking them at the dispatcher level is the best you can do. On the other hand, these requests are normally just responded with a 404 and that should be quite fast. You will always have a low amount of such noise.

[Jagadeesh_Prakash]

@Rohan_Garg 

I think you can do the analysis from at CDN (= which IP addresses flooded them with traffic). Do you have a WAF (web application firewall) in place? The Web Application Firewall typically has functionality in place to block DDoS attacks ("This IP address is sending too many requests (for a given time interval), we will block it for n minutes"). The WAF (= its logs) should provide detailed information where the traffic originated

[Rohan_Garg]

Hi Jagadeesh,

I am not sure about the WAF, I will check with Adobe if the AEMaaCS architecture has any option to configure or add one.

I will check on this & let you know.

Thanks,

Rohan Garg

[Manu_Mathew_]

@Rohan_Garg I think, using the CDN option to for DDoS protection is an option : a CDN will help to ensure it doesn’t reach the origin server and render your site completely unavailable. If a server is hit with more traffic than it can handle, it simply sends the traffic to other servers. Your site won’t experience any downtime.

[Rohan_Garg · Accepted solution]

Hi @Manu_Mathew_, Thanks for your reply!

Do you suggest we use a custom CDN ? We already have Fastly as the default CDN for AEMaaCS ecosystem.

Currently, the problem is not that the traffic is too much or running into a risk of downtime but just these unwanted requests filtering to the publish instance.

I have added the below filter rule to block all bad requests seen so far -

#Block Bad Requests
/0108 { /type "deny" /selectors '(php|up|bak|alfa)' /extension 'html' }