Thanks. This behavior was due to a bug in one of our pages.
When I was getting a 500 response from the dispatcher, I logged in to the author, opened the page, and it was rendering fine. Then the dispatcher's 500 error was fixed too. So I assumed something should be wrong with the dispatcher for the initial 500 error.
Turns out opening the page in the publisher was working, because I had logged in as admin and viewing the page was saving something to CRX, which of course the admin had the privileges to do. This saving was done only on the first view of the page.
But when the page was accessed through the dispatcher by an anonymous user, the save was failing.
tl;dr: the issue was not due to the dispatcher caching 500 responses.