AEM Dispatcher: Random 403 Forbidden response during concurrent access

[Siarhei_]

I've faced with the issue, that sometimes Apache Web Server (with dispatcher module 4.1.11) returns '403 Forbidden' response when not cached page is concurrently requested by several threads.

I have the following environment:

• virtual host running in a private cloud
• OS: CentOS 7.1
• Web Server: Apache HTTPD 2.2.31
• AEM Dispatcher: dispatcher-apache2.2-linux-x86-64-4.1.11
• AEM 5.6.1.SP2

I can reproduce this issue more or less constantly with jmeter script which initially performs cache cleaning on dispatcher and after that requests one of the geometrixx page by 30 threads in parallel. In CQ access logs all requests passed from dispatcher have HTTP Status Code 200, so I assume that the issue is somewhere on dispatcher level. Also there are no errors in dispatcher logs. The only error is in Apache httpd error_log: (2)No such file or directory: file permissions deny server access: <path to the page cache file on file system>

Apache httpd error_log:

[Tue Feb 09 09:17:46 2016] [notice] Apache/2.2.31 (Unix) Communique/4.1.11 configured -- resuming normal operations
[Tue Feb 09 09:17:46 2016] [info] Server built: Feb  8 2016 10:05:02
[Tue Feb 09 09:17:46 2016] [debug] prefork.c(1023): AcceptMutex: sysvsem (default: sysvsem)
[Tue Feb 09 09:18:15 2016] [error] [client 127.0.0.1] (2)No such file or directory: file permissions deny server access: /app/dispatcher-p/htdocs/content/geometrixx/en.html
[Tue Feb 09 09:18:15 2016] [error] [client 127.0.0.1] (2)No such file or directory: file permissions deny server access: /app/dispatcher-p/htdocs/content/geometrixx/en.html

Sample from Apache httpd access_log:

127.0.0.1 - - 09/Feb/2016 09:18:14.703 +0300 15774 "GET /content/geometrixx/en.html HTTP/1.1" 200 19965
127.0.0.1 - - 09/Feb/2016 09:18:14.716 +0300 15720 "GET /content/geometrixx/en.html HTTP/1.1" 200 19980
127.0.0.1 - - 09/Feb/2016 09:18:14.702 +0300 15723 "GET /content/geometrixx/en.html HTTP/1.1" 403 228
127.0.0.1 - - 09/Feb/2016 09:18:14.702 +0300 15728 "GET /content/geometrixx/en.html HTTP/1.1" 403 228
127.0.0.1 - - 09/Feb/2016 09:18:14.716 +0300 15751 "GET /content/geometrixx/en.html HTTP/1.1" 200 19979
127.0.0.1 - - 09/Feb/2016 09:18:14.701 +0300 15748 "GET /content/geometrixx/en.html HTTP/1.1" 200 19967

You can find full dispatcher log attached. I would appreciate any help in resolving this problem.

Upd: I've checked additionally for the issue described here and it is not my case. I have the following loaded apache httpd modules:

Loaded Modules:
 core_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 dir_module (shared)
 mime_module (shared)
 dispatcher_module (shared)
 log_config_module (shared)

[Lokesh_Shivalingaiah]

Are you facing the same issue with the prev version of dispatcher 4.1.10 ? while you try with other version, please raise a support ticket here https://daycare.day.com/home/createaticket.html

[Siarhei_]

Thank you for your response.

I was able to reproduce this issue with dispatcher 4.1.9. Also, I'v tried different httpd versions and result is the same. I will update my initial post with more details a bit later.

[Siarhei_ · Accepted solution]

Looks like the issue was related to CQ dispatcher bug 'DISP-576 - Dispatcher sometimes returns 404 after flush'. This bug has been fixed in dispatcher version 4.1.12 (Release Date: February 19th, 2016). Dispatcher upgrade has solved the issue.