Now as per my understanding dispatcher will forward the traffic to either publish-1 or publish-2 server depending on the time taken by the respective publish server.
When I look into this page then I see there is no configuration which makes it possible to direct the traffic from Dispatcher-1 to Publish-2. It is always Dispatcher-1 to Publish-1 or Dispatcher-2 to Publish-2. If Publish-2 goes down then the request landing on Dispatcher-2 will not be processed.
Could you please clarify what is the use of only one-to-one setup and when it make sense to have one-to-many setup?
We would need to further investigate your issue. From your AEM staging / development environment, you can change the apache web server's dispatcher configuration to show "error" logs. After the dispatcher is restarted, subsequent requests will be visible as new entries in the log file; you should be able to see detailed information about errors and requests here.
From a finger in the air guess, your dispatcher might not be able to access one of the publishers, therefore only one of the publishers can be resolved.