AEMaaCS DDoS Attack Analysis | Community
Skip to main content
Rohan_Garg
Community Advisor
Rohan_GargCommunity Advisor
Community Advisor
December 12, 2022
Solved

AEMaaCS DDoS Attack Analysis

  • December 12, 2022
  • 3 replies
  • 8289 views

Hi All,

 

I have the below warning and error logs from our PROD environment - 

A number of requests are being simulated to get the JCR content for the page - 

/content/brandsA/us/en/ALFA_DATA.html

/content/brandsA/us/en/ALFA_DATA/alfacgiapi.html

/content/brandsA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html

/content/brandsA/us/en/ALFA_DATA/alfacgiapi/py.alfa.html

/content/brandsA/us/en/ALFA_DATA/alfacgiapi/bash.alfa.html

/content/brandsA/us/en/ALFA_DATA/alfacgiapi/aspx.aspx.html

 

All these requests are either generating 404/Resource not found exception.

As per the security checklist for DDoS attack in AEM Systems (https://experienceleague.adobe.com/docs/experience-manager-65/administering/security/security-checklist.html?lang=en#mitigate-denial-of-service-dos-attacks) - 

  • By requesting a content page with an unlimited number of URLs, The URL can include a handle, some selectors, an extension, and a suffix - any of which can be modified.

    For example, .../en.html can also be requested as:

    • .../en.ExtensionDosAttack
    • .../en.SelectorDosAttack.html
    • .../en.html/SuffixDosAttack

    All valid variations (e.g. return a 200 response and are configured to be cached) will be cached by the dispatcher, eventually leading to a full file system and no service for further requests.

What is the best way to handle these requests ?

Blocking requests with string ALFA_DATA in dispatcher seems to be a temporary fix as the string can always change. 

@arunpatidar, @joerghoh, @markusbullaadobe, @mohit_kbansal, @b_sravan

 

12.12.2022 04:14:13.341 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818453338] GET /content/brandA/us/en/ALFA_DATA.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA.html not found
12.12.2022 04:14:13.341 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818453338] GET /content/brandA/us/en/ALFA_DATA.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA.html not found
12.12.2022 04:14:15.189 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818455186] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi.html not found
12.12.2022 04:14:15.189 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818455186] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi.html not found
12.12.2022 04:14:17.174 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818457170] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html not found
12.12.2022 04:14:17.174 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818457170] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html not found
12.12.2022 04:14:18.966 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818458961] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/py.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/py.alfa.html not found
12.12.2022 04:14:18.966 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818458961] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/py.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/py.alfa.html not found
12.12.2022 04:14:20.430 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818460427] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/bash.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/bash.alfa.html not found
12.12.2022 04:14:20.430 [cm-pabc12-exyzabc-aem-publish-78f968c46d-bl6hf] *INFO* [194.165.17.8 [1670818460427] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/bash.alfa.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/bash.alfa.html not found
12.12.2022 04:14:22.381 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818462377] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/aspx.aspx.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/aspx.aspx.html not found
12.12.2022 04:14:22.381 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [194.165.17.8 [1670818462377] GET /content/brandA/us/en/ALFA_DATA/alfacgiapi/aspx.aspx.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/aspx.aspx.html not found
12.12.2022 09:15:07.711 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *ERROR* [34.93.61.237 [1670836507686] POST /content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.servlets.post.impl.operations.ModifyOperation Unable to create resource named bdl in /content/brandA/us/en/content
12.12.2022 09:15:07.717 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *WARN* [34.93.61.237 [1670836507686] POST /content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.servlets.post.impl.SlingPostServlet Exception while handling POST on path [/content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html] with operation [org.apache.sling.servlets.post.impl.operations.ModifyOperation]
12.12.2022 09:15:07.717 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *WARN* [34.93.61.237 [1670836507686] POST /content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.servlets.post.impl.SlingPostServlet Exception while handling POST on path [/content/brandA/us/en/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa.html] with operation [org.apache.sling.servlets.post.impl.operations.ModifyOperation]
12.12.2022 09:15:08.627 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *ERROR* [34.93.61.237 [1670836508622] POST /content/brandA/us/en/alfacgiapi/perl.alfa.html HTTP/1.1] org.apache.sling.servlets.post.impl.operations.ModifyOperation Unable to create resource named bdl in /content/brandA/us/en/content
12.12.2022 09:36:31.743 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [sling-default-3-Registered Service.5118] com.day.cq.rewriter.linkchecker.impl.LinkCheckerTask Checked URL https://www.browndoglodge.com/content/brandA/us/en/ALFA_DATA/alfacgiapi: 404 (invalid)
12.12.2022 09:36:31.788 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [sling-default-3-Registered Service.5118] com.day.cq.rewriter.linkchecker.impl.LinkCheckerTask Checked URL https://www.brandsA.com/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa/: 404 (invalid)
12.12.2022 10:36:32.315 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [sling-default-3-Registered Service.5118] com.day.cq.rewriter.linkchecker.impl.LinkCheckerTask Checked URL https://www.brandsA.com/content/brandA/us/en/ALFA_DATA/alfacgiapi: 404 (invalid)
12.12.2022 10:36:32.356 [cm-pabc12-exyzabc-aem-publish-78f968c46d-kv2fz] *INFO* [sling-default-3-Registered Service.5118] com.day.cq.rewriter.linkchecker.impl.LinkCheckerTask Checked URL https://www.brandsA.com/content/brandA/us/en/ALFA_DATA/alfacgiapi/perl.alfa/: 404 (invalid)

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by Rohan_Garg

    Hi @manu_mathew_, Thanks for your reply!

    Do you suggest we use a custom CDN ? We already have Fastly as the default CDN for AEMaaCS ecosystem.

    Currently, the problem is not that the traffic is too much or running into a risk of downtime but just these unwanted requests filtering to the publish instance.

    I have added the below filter rule to block all bad requests seen so far -

    #Block Bad Requests
    /0108 { /type "deny" /selectors '(php|up|bak|alfa)' /extension 'html' }

    3 replies

    joerghoh
    Adobe Employee
    joerghohAdobe Employee
    Adobe Employee
    December 12, 2022

    It is technically possible to create many valid urls for a single page. I see 3 options:

     

    1. you ignore this problem; and that's an approach many do, and there is hardly anyone to blame for it.
    2. You start to implement a strict ruleset on dispatcher which tries to prevent such situations.
    3. You implement your AEM code to reject such requests.

    In general filling the dispatcher disk is a sound approach to impact any AEM system; but there are many ways to DDOS any application (AEM is not especially susceptible to such), so investing much effort in preventing one vector is probably a waste of resources. It should just be impossible (or at least hardly possible) that a single user is able to DOS an instance. But that should be done on a CDN layer.

     

     

      Rohan_Garg
      Community Advisor
      Rohan_GargCommunity AdvisorAuthor
      Community Advisor
      December 12, 2022

      Hi @joerghoh, Thanks for your reply!

      I will explore more on Point #2 in implementing a strict ruleset by denying dispatcher to serve these requests. But like you said there are many possible combinations.

       

      Adobe's documentation has also suggested - 
      Use a firewall to filter access to your instance.

      • The use of an operating system level firewall is necessary in order to filter access to points of your instance that might lead to denial of service attacks if left unprotected.

      I will try this out. Thanks again, I will update the fix as I progress on this.

        joerghoh
        Adobe Employee
        joerghohAdobe Employee
        Adobe Employee
        December 13, 2022

        Just looking again: You can probably get rid of a lot of such noise when you deny any POST request on the dispatcher (just if your application does not use POST, of course).

          Jagadeesh_Prakash
          Community Advisor
          Jagadeesh_PrakashCommunity Advisor
          Community Advisor
          December 12, 2022

          @rohan_garg 

           

          I think you can do the analysis from at CDN (= which IP addresses flooded them with traffic). Do you have a WAF (web application firewall) in place? The Web Application Firewall typically has functionality in place to block DDoS attacks ("This IP address is sending too many requests (for a given time interval), we will block it for n minutes"). The WAF (= its logs) should provide detailed information where the traffic originated

            Rohan_Garg
            Community Advisor
            Rohan_GargCommunity AdvisorAuthor
            Community Advisor
            December 12, 2022

            Hi Jagadeesh,

             

            I am not sure about the WAF, I will check with Adobe if the AEMaaCS architecture has any option to configure or add one.

            I will check on this & let you know.

             

            Thanks,

            Rohan Garg

              Manu_Mathew_
              Community Advisor
              Manu_Mathew_Community Advisor
              Community Advisor
              December 13, 2022

              @rohan_garg I think, using the CDN option to for DDoS protection is an option : a CDN will help to ensure it doesn’t reach the origin server and render your site completely unavailable. If a server is hit with more traffic than it can handle, it simply sends the traffic to other servers. Your site won’t experience any downtime.

                Rohan_Garg
                Community Advisor
                Rohan_GargCommunity AdvisorAuthorAccepted solution
                Community Advisor
                December 14, 2022

                Hi @manu_mathew_, Thanks for your reply!

                Do you suggest we use a custom CDN ? We already have Fastly as the default CDN for AEMaaCS ecosystem.

                Currently, the problem is not that the traffic is too much or running into a risk of downtime but just these unwanted requests filtering to the publish instance.

                I have added the below filter rule to block all bad requests seen so far -

                #Block Bad Requests
                /0108 { /type "deny" /selectors '(php|up|bak|alfa)' /extension 'html' }
                  Terms & ConditionsAccessibility statement

                  Loading footer...