AEM Dispatcher Logs: How to add extra data to dispatcher logs? | Community
Skip to main content
A
Anonymous
August 9, 2017
Solved

AEM Dispatcher Logs: How to add extra data to dispatcher logs?

  • August 9, 2017
  • 8 replies
  • 8412 views

Background/Use-case: We are using AEM6.1; have configured AEM to inject log to Splunk tool.

For creating SPLUNK dashboards, we need  custom/additional data from dispatcher logs.The  Splunk expects data in a specific format, to filter data from AEM logs- KEY:VALUE pair format(path:/content/us/en_us/testPage.html)

ASK:

Can we modify Dispatcher config/service to add new KEY:VALUE pair?  appreciate any suggestions/ideas to support the above use case.

Thanks

Guru

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by joerghoh

    This is not possible.

    But the dispatcher uses the httpd facilities to log, so instead of a logfile, you can log into a program which parses the log messages and dump them in the expected format.

    In the httpd.conf you can use something like this:

    DispatcherLog "|/path/to/my/script"

    Documented in How to Set Up Log Rotation for Dispatcher  in a slightly different scenario, but the same mechanic.

    Jörg

    8 replies

    joerghoh
    Adobe Employee
    joerghohAdobe EmployeeAccepted solution
    Adobe Employee
    August 9, 2017

    This is not possible.

    But the dispatcher uses the httpd facilities to log, so instead of a logfile, you can log into a program which parses the log messages and dump them in the expected format.

    In the httpd.conf you can use something like this:

    DispatcherLog "|/path/to/my/script"

    Documented in How to Set Up Log Rotation for Dispatcher  in a slightly different scenario, but the same mechanic.

    Jörg

      A
      Anonymous
      August 9, 2017

      Thanks Jorg, for the quick response.

      Is it possible to log to log file & to  a program? so I can keep the current flow & in my program I'll just extract specific dataset.

      joerghoh
      Adobe Employee
      joerghohAdobe Employee
      Adobe Employee
      August 9, 2017

      DispatcherLog "| tee -a myLogfile | /path/to/my/script"

      Will still write to myLogfile and also forward it to your script. "tee" is a standard unix command, please see its manpage for details.

      Jörg

        A
        Anonymous
        August 9, 2017

        great, thanks Jorg.

        A
        Anonymous
        August 18, 2017

        All

        I was able to solve the issue by using customLog.

        Solution:

        LogFormat "{ \"time\":\"%t\", \"remoteIP\":\"%a\", \"host\":\"%V\", \"urlpath\":\"%U\", \"query\":\"%q\", \"method\":\"%m\", \"status\":\"%>s\", \"True-Client-IP\":\"%{True-Client-IP}i\", \"userAgent\":\"%{User-agent}i\", \"referer\":\"%{Referer}i\" }" splunk

        CustomLog /cq/log/exp-splunk.log splunk

          ajayk94129834
          ajayk94129834
          May 7, 2020

          Hi,

           

          I am trying to implement custom log to capture details in to splunk and this article completely aligned with my requirement.

          Could you please share the custom log script it will be helpful to me.

           

          Thank you.

          P
          paull10780674
          January 16, 2018

          May I know how to configure AEM to inject log to Splunk tool?

          joerghoh
          Adobe Employee
          joerghohAdobe Employee
          Adobe Employee
          January 16, 2018

          You can feed logfiles into Splunk.

          Getting data in | Splunk

          Jörg

          MaxBarrass
          MaxBarrass
          Level 3
          June 25, 2019

          I have configured my dispatcher container to return logs like this, which should suit any current log management tools

          https://hub.docker.com/r/aemdesign/dispatcher

          ### Container Logs

          All apache and dispatcher logs are piped to container stdout, following is an example of ```docker logs -f publish-dispatcher```

          ```
          {"@timestamp":"2019-06-25 13:09:31.317603", "level":"notice", "module":"suexec", "process":"6", "message":"AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)", "host":"dispatcher",
          {"@timestamp":"2019-06-25 13:09:31.317775", "level":"info", "module":"ssl", "process":"6", "message":"AH01887: Init: Initializing (virtual) servers for SSL", "host":"dispatcher",
          {"@timestamp":"2019-06-25 13:09:31.317797", "level":"info", "module":"ssl", "process":"6", "message":"AH01876: mod_ssl/2.4.6 compiled against Server: Apache/2.4.6, Library: OpenSSL/1.0.2k", "host":"dispatcher",
          {"@timestamp":"2019-06-25 13:09:31.344707", "level":"notice", "module":"auth_digest", "process":"6", "message":"AH01757: generating secret for digest authentication ...", "host":"dispatcher",
          {"@timestamp":"2019-06-25 13:09:31.345385", "level":"notice", "module":"lbmethod_heartbeat", "process":"6", "message":"AH02282: No slotmem from mod_heartmonitor", "host":"dispatcher",
          {"@timestamp":"2019-06-25 13:09:31.345490", "level":"warn", "module":"ssl", "process":"6", "message":"AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]", "host":"dispatcher",
          {"@timestamp":"2019-06-25 13:09:31.345497", "level":"info", "module":"ssl", "process":"6", "message":"AH01887: Init: Initializing (virtual) servers for SSL", "host":"dispatcher",
          {"@timestamp":"2019-06-25 13:09:31.345510", "level":"info", "module":"ssl", "process":"6", "message":"AH01876: mod_ssl/2.4.6 compiled against Server: Apache/2.4.6, Library: OpenSSL/1.0.2k", "host":"dispatcher",
          {"@timestamp":"2019-06-25 13:09:31.349115", "level":"notice", "module":"mpm_prefork", "process":"6", "message":"AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips Communique/4.3.2 configured -- resuming normal operations", "host":"dispatcher",
          {"@timestamp":"2019-06-25 13:09:31.349154", "level":"info", "module":"mpm_prefork", "process":"6", "message":"AH00164: Server built: Apr 24 2019 13:45:48", "host":"dispatcher",
          {"@timestamp":"2019-06-25 13:09:31.349174", "level":"notice", "module":"core", "process":"6", "message":"AH00094: Command line: 'httpd -f /dispatcher/httpd/httpd.conf -D FOREGROUND'", "host":"dispatcher",
          {"@date":"Tue Jun 25 13:09:31 2019", "level":"I", "type":"dispatcher", "format":"simple", "pid":"pid 6", "message":"IP list for backend rend01 will be resolved on every request."}
          {"@date":"Tue Jun 25 13:09:31 2019", "level":"I", "type":"dispatcher", "format":"simple", "pid":"pid 6", "message":"Dispatcher initialized (build 4.3.2/apache2.4-linux-x86_64)"}
          {"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60260", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/aemdesign-showcase/au/en/component/lists/page-list.html", "query_string":"", "referer":"-", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"200", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2375", "bytes_out":"8486", "keepalive":"+", "duration_micros":"4810243", "duration_sec": 4, "status":"200", "short_message": "GET /content/aemdesign-showcase/au/en/component/lists/page-list.html HTTP/1.1" },
          {"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 10", "method":"GET", "path":"/content/aemdesign-showcase/au/en/component/lists/page-list.html", "status":"200", "size":"158170", "duration":"4809ms", "renderer":"publish/rend01"}
          {"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60260", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"404", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2334", "bytes_out":"920", "keepalive":"+", "duration_micros":"22320", "duration_sec": 0, "status":"404", "short_message": "GET /content/dam/aemdesign-showcase/en/components/media/image/city1.jpg HTTP/1.1" },
          {"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 10", "method":"GET", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "status":"404", "size":"568", "duration":"22ms", "renderer":"publish/rend01"}
          {"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60262", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/libs/granite/csrf/token.json", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"200", "response_location":"-", "Content-Type":"application/json", "bytes_in":"2260", "bytes_out":"348", "keepalive":"+", "duration_micros":"17257", "duration_sec": 0, "status":"200", "short_message": "GET /libs/granite/csrf/token.json HTTP/1.1" },
          {"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 11", "method":"GET", "path":"/libs/granite/csrf/token.json", "status":"200", "size":"2", "duration":"17ms", "renderer":"publish/rend01"}
          {"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60262", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"404", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2334", "bytes_out":"920", "keepalive":"+", "duration_micros":"10906", "duration_sec": 0, "status":"404", "short_message": "GET /content/dam/aemdesign-showcase/en/components/media/image/city1.jpg HTTP/1.1" },
          {"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 11", "method":"GET", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "status":"404", "size":"568", "duration":"11ms", "renderer":"publish/rend01"}
          ```

          #### Log Format
          - All records with ```@timestamp``` are from apache generated using CustomLog and ErrorLog

          - All records with ```@date``` are from apache Dispatcher module generated using DispatcherLog

          - Apache can spit out non JSON formatted logs into STDOUT

          Terms & ConditionsAccessibility statement

          Loading footer...