AEM Dispatcher: Random 403 Forbidden response during concurrent access
- February 9, 2016
- 3 replies
- 4325 views
I've faced with the issue, that sometimes Apache Web Server (with dispatcher module 4.1.11) returns '403 Forbidden' response when not cached page is concurrently requested by several threads.
I have the following environment:
- virtual host running in a private cloud
- OS: CentOS 7.1
- Web Server: Apache HTTPD 2.2.31
- AEM Dispatcher: dispatcher-apache2.2-linux-x86-64-4.1.11
- AEM 5.6.1.SP2
I can reproduce this issue more or less constantly with jmeter script which initially performs cache cleaning on dispatcher and after that requests one of the geometrixx page by 30 threads in parallel. In CQ access logs all requests passed from dispatcher have HTTP Status Code 200, so I assume that the issue is somewhere on dispatcher level. Also there are no errors in dispatcher logs. The only error is in Apache httpd error_log: (2)No such file or directory: file permissions deny server access: <path to the page cache file on file system>
Apache httpd error_log:
[Tue Feb 09 09:17:46 2016] [notice] Apache/2.2.31 (Unix) Communique/4.1.11 configured -- resuming normal operations
[Tue Feb 09 09:17:46 2016] [info] Server built: Feb 8 2016 10:05:02
[Tue Feb 09 09:17:46 2016] [debug] prefork.c(1023): AcceptMutex: sysvsem (default: sysvsem)
[Tue Feb 09 09:18:15 2016] [error] [client 127.0.0.1] (2)No such file or directory: file permissions deny server access: /app/dispatcher-p/htdocs/content/geometrixx/en.html
[Tue Feb 09 09:18:15 2016] [error] [client 127.0.0.1] (2)No such file or directory: file permissions deny server access: /app/dispatcher-p/htdocs/content/geometrixx/en.html
Sample from Apache httpd access_log:
127.0.0.1 - - 09/Feb/2016 09:18:14.703 +0300 15774 "GET /content/geometrixx/en.html HTTP/1.1" 200 19965
127.0.0.1 - - 09/Feb/2016 09:18:14.716 +0300 15720 "GET /content/geometrixx/en.html HTTP/1.1" 200 19980
127.0.0.1 - - 09/Feb/2016 09:18:14.702 +0300 15723 "GET /content/geometrixx/en.html HTTP/1.1" 403 228
127.0.0.1 - - 09/Feb/2016 09:18:14.702 +0300 15728 "GET /content/geometrixx/en.html HTTP/1.1" 403 228
127.0.0.1 - - 09/Feb/2016 09:18:14.716 +0300 15751 "GET /content/geometrixx/en.html HTTP/1.1" 200 19979
127.0.0.1 - - 09/Feb/2016 09:18:14.701 +0300 15748 "GET /content/geometrixx/en.html HTTP/1.1" 200 19967
You can find full dispatcher log attached. I would appreciate any help in resolving this problem.
Upd: I've checked additionally for the issue described here and it is not my case. I have the following loaded apache httpd modules:
Loaded Modules:
core_module (static)
mpm_prefork_module (static)
http_module (static)
so_module (static)
dir_module (shared)
mime_module (shared)
dispatcher_module (shared)
log_config_module (shared)