Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

AEM Dispatcher Logs: How to add extra data to dispatcher logs?

Avatar

Avatar
Validate 1
Level 2
gurub46428210
Level 2

Likes

9 likes

Total Posts

19 posts

Correct Reply

2 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
Affirm 1
View profile

Avatar
Validate 1
Level 2
gurub46428210
Level 2

Likes

9 likes

Total Posts

19 posts

Correct Reply

2 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
Affirm 1
View profile
gurub46428210
Level 2

09-08-2017

Background/Use-case: We are using AEM6.1; have configured AEM to inject log to Splunk tool.

For creating SPLUNK dashboards, we need  custom/additional data from dispatcher logs.The  Splunk expects data in a specific format, to filter data from AEM logs- KEY:VALUE pair format(path:/content/us/en_us/testPage.html)

ASK:

Can we modify Dispatcher config/service to add new KEY:VALUE pair?  appreciate any suggestions/ideas to support the above use case.

Thanks

Guru

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Avatar
Coach
Employee
Jörg_Hoh
Employee

Likes

1,133 likes

Total Posts

3,161 posts

Correct Reply

1,079 solutions
Top badges earned
Coach
Give back 600
Ignite 5
Ignite 3
Ignite 1
View profile

Avatar
Coach
Employee
Jörg_Hoh
Employee

Likes

1,133 likes

Total Posts

3,161 posts

Correct Reply

1,079 solutions
Top badges earned
Coach
Give back 600
Ignite 5
Ignite 3
Ignite 1
View profile
Jörg_Hoh
Employee

09-08-2017

This is not possible.

But the dispatcher uses the httpd facilities to log, so instead of a logfile, you can log into a program which parses the log messages and dump them in the expected format.

In the httpd.conf you can use something like this:

DispatcherLog "|/path/to/my/script"

Documented in How to Set Up Log Rotation for Dispatcher  in a slightly different scenario, but the same mechanic.

Jörg

Answers (7)

Answers (7)

Avatar

Avatar
Validate 1
Level 2
gurub46428210
Level 2

Likes

9 likes

Total Posts

19 posts

Correct Reply

2 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
Affirm 1
View profile

Avatar
Validate 1
Level 2
gurub46428210
Level 2

Likes

9 likes

Total Posts

19 posts

Correct Reply

2 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
Affirm 1
View profile
gurub46428210
Level 2

18-08-2017

All

I was able to solve the issue by using customLog.

Solution:

LogFormat "{ \"time\":\"%t\", \"remoteIP\":\"%a\", \"host\":\"%V\", \"urlpath\":\"%U\", \"query\":\"%q\", \"method\":\"%m\", \"status\":\"%>s\", \"True-Client-IP\":\"%{True-Client-IP}i\", \"userAgent\":\"%{User-agent}i\", \"referer\":\"%{Referer}i\" }" splunk

CustomLog /cq/log/exp-splunk.log splunk

Avatar

Avatar
Coach
Employee
Jörg_Hoh
Employee

Likes

1,133 likes

Total Posts

3,161 posts

Correct Reply

1,079 solutions
Top badges earned
Coach
Give back 600
Ignite 5
Ignite 3
Ignite 1
View profile

Avatar
Coach
Employee
Jörg_Hoh
Employee

Likes

1,133 likes

Total Posts

3,161 posts

Correct Reply

1,079 solutions
Top badges earned
Coach
Give back 600
Ignite 5
Ignite 3
Ignite 1
View profile
Jörg_Hoh
Employee

09-08-2017

DispatcherLog "| tee -a myLogfile | /path/to/my/script"

Will still write to myLogfile and also forward it to your script. "tee" is a standard unix command, please see its manpage for details.

Jörg

Avatar

Avatar
Boost 5
Level 3
MaxBarrass
Level 3

Likes

18 likes

Total Posts

42 posts

Correct Reply

4 solutions
Top badges earned
Boost 5
Boost 3
Boost 10
Boost 1
Affirm 3
View profile

Avatar
Boost 5
Level 3
MaxBarrass
Level 3

Likes

18 likes

Total Posts

42 posts

Correct Reply

4 solutions
Top badges earned
Boost 5
Boost 3
Boost 10
Boost 1
Affirm 3
View profile
MaxBarrass
Level 3

25-06-2019

I have configured my dispatcher container to return logs like this, which should suit any current log management tools

https://hub.docker.com/r/aemdesign/dispatcher

### Container Logs

All apache and dispatcher logs are piped to container stdout, following is an example of ```docker logs -f publish-dispatcher```

```
{"@timestamp":"2019-06-25 13:09:31.317603", "level":"notice", "module":"suexec", "process":"6", "message":"AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.317775", "level":"info", "module":"ssl", "process":"6", "message":"AH01887: Init: Initializing (virtual) servers for SSL", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.317797", "level":"info", "module":"ssl", "process":"6", "message":"AH01876: mod_ssl/2.4.6 compiled against Server: Apache/2.4.6, Library: OpenSSL/1.0.2k", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.344707", "level":"notice", "module":"auth_digest", "process":"6", "message":"AH01757: generating secret for digest authentication ...", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.345385", "level":"notice", "module":"lbmethod_heartbeat", "process":"6", "message":"AH02282: No slotmem from mod_heartmonitor", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.345490", "level":"warn", "module":"ssl", "process":"6", "message":"AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.345497", "level":"info", "module":"ssl", "process":"6", "message":"AH01887: Init: Initializing (virtual) servers for SSL", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.345510", "level":"info", "module":"ssl", "process":"6", "message":"AH01876: mod_ssl/2.4.6 compiled against Server: Apache/2.4.6, Library: OpenSSL/1.0.2k", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.349115", "level":"notice", "module":"mpm_prefork", "process":"6", "message":"AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips Communique/4.3.2 configured -- resuming normal operations", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.349154", "level":"info", "module":"mpm_prefork", "process":"6", "message":"AH00164: Server built: Apr 24 2019 13:45:48", "host":"dispatcher",
{"@timestamp":"2019-06-25 13:09:31.349174", "level":"notice", "module":"core", "process":"6", "message":"AH00094: Command line: 'httpd -f /dispatcher/httpd/httpd.conf -D FOREGROUND'", "host":"dispatcher",
{"@date":"Tue Jun 25 13:09:31 2019", "level":"I", "type":"dispatcher", "format":"simple", "pid":"pid 6", "message":"IP list for backend rend01 will be resolved on every request."}
{"@date":"Tue Jun 25 13:09:31 2019", "level":"I", "type":"dispatcher", "format":"simple", "pid":"pid 6", "message":"Dispatcher initialized (build 4.3.2/apache2.4-linux-x86_64)"}
{"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60260", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/aemdesign-showcase/au/en/component/lists/page-list.html", "query_string":"", "referer":"-", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"200", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2375", "bytes_out":"8486", "keepalive":"+", "duration_micros":"4810243", "duration_sec": 4, "status":"200", "short_message": "GET /content/aemdesign-showcase/au/en/component/lists/page-list.html HTTP/1.1" },
{"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 10", "method":"GET", "path":"/content/aemdesign-showcase/au/en/component/lists/page-list.html", "status":"200", "size":"158170", "duration":"4809ms", "renderer":"publish/rend01"}
{"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60260", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"404", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2334", "bytes_out":"920", "keepalive":"+", "duration_micros":"22320", "duration_sec": 0, "status":"404", "short_message": "GET /content/dam/aemdesign-showcase/en/components/media/image/city1.jpg HTTP/1.1" },
{"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 10", "method":"GET", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "status":"404", "size":"568", "duration":"22ms", "renderer":"publish/rend01"}
{"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60262", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/libs/granite/csrf/token.json", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"200", "response_location":"-", "Content-Type":"application/json", "bytes_in":"2260", "bytes_out":"348", "keepalive":"+", "duration_micros":"17257", "duration_sec": 0, "status":"200", "short_message": "GET /libs/granite/csrf/token.json HTTP/1.1" },
{"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 11", "method":"GET", "path":"/libs/granite/csrf/token.json", "status":"200", "size":"2", "duration":"17ms", "renderer":"publish/rend01"}
{"@timestamp":"uc", "client_ip":"172.17.0.1", "client_port":"60262", "server_ip":"172.17.0.7", "Connection": "-:- ", "X-Forwarded-For":"-", "X-Powered-By": "-" "user":"-", "REMOTE_USER":"-", "JSESSIONID":"-", "host":"dispatcher", "pid":"80", "tid":"tid", "request":"-", "protocol":"HTTP/1.1", "http_method":"GET", "vhost":"localhost", "service_port":"80", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "query_string":"", "referer":"http://localhost/content/aemdesign-showcase/au/en/component/lists/page-list.html", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "response_code":"404", "response_location":"-", "Content-Type":"text/html", "bytes_in":"2334", "bytes_out":"920", "keepalive":"+", "duration_micros":"10906", "duration_sec": 0, "status":"404", "short_message": "GET /content/dam/aemdesign-showcase/en/components/media/image/city1.jpg HTTP/1.1" },
{"@date":"Tue Jun 25 13:10:43 2019", "level":"I", "type":"dispatcher", "format":"render", "pid":"pid 11", "method":"GET", "path":"/content/dam/aemdesign-showcase/en/components/media/image/city1.jpg", "status":"404", "size":"568", "duration":"11ms", "renderer":"publish/rend01"}
```

#### Log Format
- All records with ```@timestamp``` are from apache generated using CustomLog and ErrorLog

- All records with ```@date``` are from apache Dispatcher module generated using DispatcherLog

- Apache can spit out non JSON formatted logs into STDOUT

Avatar

Avatar
Coach
Employee
Jörg_Hoh
Employee

Likes

1,133 likes

Total Posts

3,161 posts

Correct Reply

1,079 solutions
Top badges earned
Coach
Give back 600
Ignite 5
Ignite 3
Ignite 1
View profile

Avatar
Coach
Employee
Jörg_Hoh
Employee

Likes

1,133 likes

Total Posts

3,161 posts

Correct Reply

1,079 solutions
Top badges earned
Coach
Give back 600
Ignite 5
Ignite 3
Ignite 1
View profile
Jörg_Hoh
Employee

16-01-2018

You can feed logfiles into Splunk.

Getting data in | Splunk

Jörg

Avatar

Avatar
Level 1
paull10780674
Level 1

Likes

0 likes

Total Posts

1 post

Correct Reply

0 solutions
View profile

Avatar
Level 1
paull10780674
Level 1

Likes

0 likes

Total Posts

1 post

Correct Reply

0 solutions
View profile
paull10780674
Level 1

16-01-2018

May I know how to configure AEM to inject log to Splunk tool?

Avatar

Avatar
Validate 1
Level 2
gurub46428210
Level 2

Likes

9 likes

Total Posts

19 posts

Correct Reply

2 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
Affirm 1
View profile

Avatar
Validate 1
Level 2
gurub46428210
Level 2

Likes

9 likes

Total Posts

19 posts

Correct Reply

2 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
Affirm 1
View profile
gurub46428210
Level 2

09-08-2017

great, thanks Jorg.

Avatar

Avatar
Validate 1
Level 2
gurub46428210
Level 2

Likes

9 likes

Total Posts

19 posts

Correct Reply

2 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
Affirm 1
View profile

Avatar
Validate 1
Level 2
gurub46428210
Level 2

Likes

9 likes

Total Posts

19 posts

Correct Reply

2 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 1
Affirm 1
View profile
gurub46428210
Level 2

09-08-2017

Thanks Jorg, for the quick response.

Is it possible to log to log file & to  a program? so I can keep the current flow & in my program I'll just extract specific dataset.