AEM dispatcher not serving page from publisher

[RitendraS11]

Hello everyone,

I am facing an issue where the AEM dispatcher is not serving the page from the publisher.

AEM setup is on-premise and non-prod. I had AEM 6.3.3.8 running with the dispatcher(4.2.x) on the Apache server, where the publisher and dispatcher are hosted on separate servers. 

I had performed an upgrade of AEM from 6.3.3.8 to 6.5.9 (Because, the application project is built using JAVA JDK8). Now, the dispatcher is also updated to 4.3.3 as per Adobe's documentation.

The existing dispatcher configurations were not touched, I have replaced the mod_dispatcher.so file and restarted the dispatcher(httpd service restarted).

When requesting a page via the dispatcher, I always get a 502 gateway timeout. When I switched to debug mode, I found the below logs in the dispatcher.log file -

[D] [pid 3968399] Found farm website-farm for <website_name>
[D] [pid 3968399] checking [/test-page.html]
[D] [pid 3968399] cachefile does not exist: /var/www/html/website/content/test-page.html
[D] [pid 3968399] try to create new cachefile: /var/www/html/website/content/test-page.html
[D] [pid 3968399] cache-action for [/test-page.html]: CREATE
[D] [pid 3968399] Creating new connection: <publisher_IP>:4503
[D] [pid 3968399] Connected to backend 0 (<publisher_IP>:4503)
[D] [pid 3968399] Adding request header: Content-Type
[D] [pid 3968399] Adding request header: User-Agent
[D] [pid 3968399] Adding request header: Host
[D] [pid 3968399] Adding request header: Via
[D] [pid 3968399] Adding request header: X-Forwarded-For
[D] [pid 3968399] Adding request header: Server-Agent
[W] [pid 3968399] ignoring unexpected character 
[W] [pid 3968399] ignoring unexpected character 
[W] [pid 3968399] ignoring unexpected character 
[W] [pid 3968399] ignoring unexpected character [Thu Nov 14 00:08:05 2024] [W] [pid 3968399] ignoring unexpected character 
[W] [pid 3968399] ignoring unexpected character 
[W] [pid 3968399] ignoring unexpected character P
[D] [pid 3968399] Unable to parse response: no more bytes available (remote peer closed connection), state = 1
[W] [pid 3968399] Failed parsing response: no more bytes available (remote peer closed connection).
[D] [pid 3968399] initializing retry, closing connection
[W] [pid 3968399] ignoring unexpected character 
[W] [pid 3968399] ignoring unexpected character 
[W] [pid 3968399] ignoring unexpected character 
[W] [pid 3968399] ignoring unexpected character [Thu Nov 14 00:08:05 2024] [W] [pid 3968399] ignoring unexpected character 
[W] [pid 3968399] ignoring unexpected character 
[W] [pid 3968399] ignoring unexpected character P
[D] [pid 3968399] Unable to parse response: no more bytes available (remote peer closed connection) (no more bytes available (remote peer closed connection)), state = 1
[W] [pid 3968399] Failed parsing response: no more bytes available (remote peer closed connection) (no more bytes available (remote peer closed connection)).
[D] [pid 3968399] No socket reuse: response body not fully consumed
[I] [pid 3968399] "GET /test-page.html" 502 miss [website-farm/0] 9ms

Also, when I performed the CURL command from the dispatcher - I noticed that when the URL contains HTTP protocol, I get an output as the letter "P" and when the URL contains HTTPS protocol, I get the output with the desired HTML.

Any pointers, where I should check further to solve the issue?

__PRESENT

[EstebanBustamante]

Hi, 

It sounds like there is a typo somewhere, can you check the redirects and make sure there is nothing that could be causing the issue? 

[A_H_M_Imrul]

Hello @RitendraS11 ,

I assume the page is being served properly through the publisher. Still can you please do the following:

• Clear Cache and Check Permissions:

  • Clear the dispatcher cache for /test-page.html.
  • Ensure the dispatcher has the correct permissions to read/write in /var/www/html/website/content.

• Test with another page simple page:

  • Create a simple page on the publisher and request it via the dispatcher. If it works, the issue might be specific to /test-page.html.

Please let us know your findings..

Thanks

[RitendraS11]

@A_H_M_Imrul— Thanks for looking into my query! As I mentioned in the problem statement, the dispatcher connects to the publisher. But the response doesn't contain anything except "P." It may have connected to the publisher, but the connection was closed before it served a full response. I am not 100% sure about this, but it looks like some network-related issue to me. 

I have asked my question in this forum to get some ideas if someone has faced similar issues and can guide me to fix them.

Whatever you mentioned is already tried and tested!

Best regards!

[A_H_M_Imrul]

@RitendraS11 

Can you please try changing (increasing) the timeout period? You can find it in the *_renders.any file for publisher under /conf.dispatcher.d/renders directory

Thanks

[RitendraS11]

@A_H_M_Imrul, @kautuk_sahni , @arunpatidar  I have received information from the server team that they have upgraded the Linux server from RHEL 7 to RHEL 8.10. Do we need to perform any configuration changes if the Linux server is upgraded? I was trying to check this in any available documentation, but no luck!

Current setup: Linux server - Red Hat Enterprise Linux 8.10

AEM dispatcher version - 4.3.3 

Apache 2.4

In dispatcher.any file, 

/renders
{
    /0 { /hostname "publish1" /port "4503" /timeout "10000" }
}

If you can check if this Linux server upgrade could be causing the issue? The only issue is, I cannot downgrade the server back to RHEL 7. 

Best regards!

[arunpatidar]

when we had the RHEL 7 to RHEL 8.10 upgrade, we did not performed any dispatcher chances and we did not faced this issue.

Adobe might have shared with you the impact or change due to this upgrade.

[RitendraS11]

Thanks @arunpatidar, I tried to hit the URL via curl command from the dispatcher server directly and received the below - 

curl -v http://<SERVER_IP>:4503/content/test-app/en.html

*   Trying <Publisher_IP>...

* TCP_NODELAY set

* Connected to<Publisher_IP> (<Publisher_IP>) port 4503 (#0)

> GET /content/test-app/en.html HTTP/1.1

> Host:<Publisher_IP>:4503

> User-Agent: curl/7.61.1

> Accept: */*

> 

Warning: Binary output can mess up your terminal. Use "--output -" to tell

Warning: curl to output it to your terminal anyway, or consider "--output

Warning: <FILE>" to save to a file.

* Failed writing body (0 != 7)

* Closing connection 0

So, the dispatcher connects to the publish instance. But, didn't get a response from the publish instance. Any idea, if I can check anything on publish instance?

NOTE: The SSL certificate was already installed on the publish instance. Since I have upgraded the AEM instances from 6.3 to 6.5 - I was checking if it could be an error because of the SSL certificate, I tried to install a new SSL certificate on the publish instance and it failed. Can it be related to it?

Best regards!

[arunpatidar]

Hi @RitendraS11 

Are you able to execute the same command from your local machine? what response do you get?

Is the same or connection failed?

I would suggest to raise a ticket with Adobe.

[kautuk_sahni]

@RitendraS11 Did you find the suggestions helpful? Please let us know if you require more information. Otherwise, please mark the answer as correct for posterity. If you've discovered a solution yourself, we would appreciate it if you could share it with the community. Thank you!