Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
Bedrock Mission!

Learn more

View all

Sign in to view all badges

SOLVED

AEM Dispatcher Logs: How to add extra data to dispatcher logs?

Avatar

Level 3

Background/Use-case: We are using AEM6.1; have configured AEM to inject log to Splunk tool.

For creating SPLUNK dashboards, we need  custom/additional data from dispatcher logs.The  Splunk expects data in a specific format, to filter data from AEM logs- KEY:VALUE pair format(path:/content/us/en_us/testPage.html)

ASK:

Can we modify Dispatcher config/service to add new KEY:VALUE pair?  appreciate any suggestions/ideas to support the above use case.

Thanks

Guru

1 Accepted Solution

Avatar

Correct answer by
Employee Advisor

This is not possible.

But the dispatcher uses the httpd facilities to log, so instead of a logfile, you can log into a program which parses the log messages and dump them in the expected format.

In the httpd.conf you can use something like this:

DispatcherLog "|/path/to/my/script"

Documented in How to Set Up Log Rotation for Dispatcher  in a slightly different scenario, but the same mechanic.

Jörg

View solution in original post

9 Replies

Avatar

Correct answer by
Employee Advisor

This is not possible.

But the dispatcher uses the httpd facilities to log, so instead of a logfile, you can log into a program which parses the log messages and dump them in the expected format.

In the httpd.conf you can use something like this:

DispatcherLog "|/path/to/my/script"

Documented in How to Set Up Log Rotation for Dispatcher  in a slightly different scenario, but the same mechanic.

Jörg

Avatar

Level 3

Thanks Jorg, for the quick response.

Is it possible to log to log file & to  a program? so I can keep the current flow & in my program I'll just extract specific dataset.

Avatar

Employee Advisor

DispatcherLog "| tee -a myLogfile | /path/to/my/script"

Will still write to myLogfile and also forward it to your script. "tee" is a standard unix command, please see its manpage for details.

Jörg

Avatar

Level 3

All

I was able to solve the issue by using customLog.

Solution:

LogFormat "{ \"time\":\"%t\", \"remoteIP\":\"%a\", \"host\":\"%V\", \"urlpath\":\"%U\", \"query\":\"%q\", \"method\":\"%m\", \"status\":\"%>s\", \"True-Client-IP\":\"%{True-Client-IP}i\", \"userAgent\":\"%{User-agent}i\", \"referer\":\"%{Referer}i\" }" splunk

CustomLog /cq/log/exp-splunk.log splunk

Avatar

Level 1

Hi,

 

I am trying to implement custom log to capture details in to splunk and this article completely aligned with my requirement.

Could you please share the custom log script it will be helpful to me.

 

Thank you.

Avatar

Level 1

May I know how to configure AEM to inject log to Splunk tool?

Avatar

Level 3

I have configured my dispatcher container to return logs like this, which should suit any current log management tools

https://hub.docker.com/r/aemdesign/dispatcher

### Container Logs

All apache and dispatcher logs are piped to container stdout, following is an example of ```docker logs -f publish-dispatcher```

```
{"@timestamp":"2019-06-25 13:09:31.317603", "level":"notice", "module":"suexec", "process":"6", "message":"AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.317775", "level":"info", "module":"ssl", "process":"6", "message":"AH01887: Init: Initializing (virtual) servers for SSL", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.317797", "level":"info", "module":"ssl", "process":"6", "message":"AH01876: mod_ssl/2.4.6 compiled against Server: Apache/2.4.6, Library: OpenSSL/1.0.2k", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.344707", "level":"notice", "module":"auth_digest", "process":"6", "message":"AH01757: generating secret for digest authentication ...", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.345385", "level":"notice", "module":"lbmethod_heartbeat", "process":"6", "message":"AH02282: No slotmem from mod_heartmonitor", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.345490", "level":"warn", "module":"ssl", "process":"6", "message":"AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.345497", "level":"info", "module":"ssl", "process":"6", "message":"AH01887: Init: Initializing (virtual) servers for SSL", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.345510", "level":"info", "module":"ssl", "process":"6", "message":"AH01876: mod_ssl/2.4.6 compiled against Server: Apache/2.4.6, Library: OpenSSL/1.0.2k", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.349115", "level":"notice", "module":"mpm_prefork", "process":"6", "message":"AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips Communique/4.3.2 configured -- resuming normal operations", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.349154", "level":"info", "module":"mpm_prefork", "process":"6", "message":"AH00164: Server built: Apr 24 2019 13:45:48", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.349174", "level":"notice", "module":"core", "process":"6", "message":"AH00094: Command line: 'httpd -f /dispatcher/httpd/httpd.conf -D FOREGROUND'", "host":"dispatcher",
{"@date":"Tue Jun 25 13:09:31 2019", "level":"I", "type":"dispatcher", "format":"simple", "pid":"pid 6", "message":"IP list for backend rend01 will be resolved on every request."}
{"@date":"Tue Jun 25 13:09:31 2019", "level":"I", "type":"dispatcher", "format":"simple", "pid":"pid 6", "message":"Dispatcher initialized (build 4.3.2/apache2.4-linux-x86_64)"}
{"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60260", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/aemdesign-showcase/au/en/component/lists/page-list.html", "query_string":"", "referer":"-", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"200", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2375", "bytes_out":"8486", "keepalive":"+", "duration_micros":"4810243", "duration_sec": 4, "status":"200", "short_message": "GET /content/aemdesign-showcase/au/en/component/lists/page-list.html HTTP/1.1" },
{"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 10", "method":"GET", "path":"/content/aemdesign-showcase/au/en/component/lists/page-list.html", "status":"200", "size":"158170", "duration":"4809ms", "renderer":"publish/rend01"}
{"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60260", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"404", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2334", "bytes_out":"920", "keepalive":"+", "duration_micros":"22320", "duration_sec": 0, "status":"404", "short_message": "GET /content/dam/aemdesign-showcase/en/components/media/image/city1.jpg HTTP/1.1" },
{"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 10", "method":"GET", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "status":"404", "size":"568", "duration":"22ms", "renderer":"publish/rend01"}
{"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60262", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/libs/granite/csrf/token.json", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"200", "response_location":"-", "Content-Type":"application/json", "bytes_in":"2260", "bytes_out":"348", "keepalive":"+", "duration_micros":"17257", "duration_sec": 0, "status":"200", "short_message": "GET /libs/granite/csrf/token.json HTTP/1.1" },
{"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 11", "method":"GET", "path":"/libs/granite/csrf/token.json", "status":"200", "size":"2", "duration":"17ms", "renderer":"publish/rend01"}
{"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60262", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"404", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2334", "bytes_out":"920", "keepalive":"+", "duration_micros":"10906", "duration_sec": 0, "status":"404", "short_message": "GET /content/dam/aemdesign-showcase/en/components/media/image/city1.jpg HTTP/1.1" },
{"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 11", "method":"GET", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "status":"404", "size":"568", "duration":"11ms", "renderer":"publish/rend01"}
```

#### Log Format
- All records with ```@timestamp``` are from apache generated using CustomLog and ErrorLog

- All records with ```@date``` are from apache Dispatcher module generated using DispatcherLog

- Apache can spit out non JSON formatted logs into STDOUT