Recently we had VM migration from an old location to another one. The migration cloned one branch of the whole architecture as the following: author -> publisher -> dispatcher. Nothing changes except the new clone has a new domain name. Now we found a problem, the dispatcher doesn't cache.
We can authorize contents from author, the contents will be replicated all the way to the dispatcher, but the dispatcher always skips itself and tries to get contents from the publisher. For instance, if we create a file abc.html directly in the dispatcher, it will report a 404 error because it doesn't exist in the publisher. If we delete a cached file in the dispatcher, say xyz.html, we can get the content directly from the publisher but no xyz.html created in the dispatcher. At last, if we shut down the publisher, we will see "The proxy server received an invalid response from an upstream server.", a Bad Gateway message.
You can try the below steps to overcome this issue.
Check if the apache user has the write permissions and ownership of cache docroot in the dispatcher. if the apache user is not the owner and doesn't have the write permissions to this directory, the dispatcher will not cache anything because as a part of cache creation, the dispatcher will create files inside the cache docroot directory.
Check the dispatcher cache rules and check what has to be cached or not.
When you are requesting for a file which is not cached in the dispatcher such as abc.html what you are observing in the dispatcher.log? If you can share the log for a single file request from both dispatcher and publisher it will be easier to debug.