Timeout gateway error - 504 and connection is failing at the intermediate dispatcher server

[vikesh1]

Hi,

For AEM Author stage instance we have it AWS and on request made is passed through Dispatcher Server -> Intermediate server -> AEM server.

But the application is throwing 504 gateway timeout error and is failing to pass the request through intermediate server and the following logs are displayed from the dispatcher logs.

Mon Jul 15 10:17:21 2024] [D] [pid 17125] Found farm author for st-cms.power.ge.com
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] checking [/libs/granite/csrf/token.json]
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] cachefile does not exist: /data/aem/cache/author/libs/granite/csrf/token.json
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] try to create new cachefile: /data/aem/cache/author/libs/granite/csrf/token.json
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] cache-action for [/libs/granite/csrf/token.json]: CREATE
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Reusing connection: 10.229.225.249:80
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Connected to backend rend01 (10.229.225.249:80)
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: X-Forwarded-Proto
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: X-Forwarded-Port
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: Host
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: X-Amzn-Trace-Id
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: X-Amzn-Oidc-Data
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: X-Amzn-Oidc-Identity
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: X-Amzn-Oidc-Accesstoken
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: sec-ch-ua
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: sec-ch-ua-mobile
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: user-agent
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: sec-ch-ua-platform
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: accept
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: sec-fetch-site
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: sec-fetch-mode
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: sec-fetch-dest
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: referer
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: accept-encoding
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: accept-language
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: priority
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: cookie
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: jwt
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: Via
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: X-Forwarded-For
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Adding request header: Server-Agent
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] Unable to parse response: no more bytes available (remote peer closed connection), state = 0
[Mon Jul 15 10:17:21 2024] [W] [pid 17125] Failed parsing response: no more bytes available (remote peer closed connection).
[Mon Jul 15 10:17:21 2024] [D] [pid 17125] initializing retry, closing connection
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Found farm author for 10.229.224.53
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] checking [/]
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] request URL has no extension: /
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] cache-action for [/]: NONE
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Reusing connection: 10.229.225.249:80
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Connected to backend rend01 (10.229.225.249:80)
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Adding request header: Host
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Adding request header: User-Agent
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Adding request header: Accept-Encoding
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Adding request header: jwt
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Adding request header: x-amzn-oidc-data
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Adding request header: Via
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Adding request header: X-Forwarded-For
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Adding request header: Server-Agent
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] Unable to parse response: no more bytes available (remote peer closed connection), state = 0
[Mon Jul 15 10:17:24 2024] [W] [pid 17840] Failed parsing response: no more bytes available (remote peer closed connection).
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] initializing retry, closing connection

Please suggest for any solutions. In the community i have found https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/aem-service-freeze-unable-... a similar ask but for me it did not help after the restart of the servers.

Thanks!

[DPrakashRaj]

From the log it appears the request url is not properly read as it’s making a request to / with no path in it.
on the dispatcher as you can see below entry on the log

Mon Jul 15 10:17:24 2024] [D] [pid 17840] checking [/]
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] request URL has no extension: /
[Mon Jul 15 10:17:24 2024] [D] [pid 17840] cache-action for [/]: NONE

[vikesh1]

I don't this is causing the issue as this is for cached content and the logs are from author instance for which no caching is configured in the environment.

[PGURUKRISHNA · Accepted solution]

Hi @vikesh1 

The 504 gateway timeout error is usually caused by a slow or unresponsive backend server. In your case, it seems that the intermediate server is not responding or is taking too long to respond, causing the dispatcher to timeout.

Here are a few potential solutions you can try:

1. Check the intermediate server logs: Investigate the logs of the intermediate server to see if there are any errors or issues that could be causing the slow response or connection closure.
2. Increase the timeout value: You can try increasing the timeout value in the dispatcher configuration to give the intermediate server more time to respond. This can be done by adding the following lines to your dispatcher configuration file (dispatcher.any):/timeout
   /connect 30
   /socket 30

   This sets the connect and socket timeouts to 30 seconds. You can adjust the values as needed.

   1. Check the network connectivity: Verify that the network connectivity between the dispatcher and the intermediate server is stable and not experiencing any issues.
   2. Check the intermediate server resource utilization: Ensure that the intermediate server has sufficient resources (CPU, memory, etc.) to handle the requests. If the server is overloaded, it may cause slow responses or connection closures.
   3. Implement retry logic: You can implement retry logic in your dispatcher configuration to retry the request if it fails due to a timeout. This can be done using the retry directive in the dispatcher configuration file.

[kautuk_sahni]

@vikesh1 Did you find the suggestion helpful? Please let us know if you require more information. Otherwise, please mark the answer as correct for posterity. If you've discovered a solution yourself, we would appreciate it if you could share it with the community. Thank you!