If you have a load balancer in-front of dispatcher then you can take 1 dispatcher out of load balancer which inturn removes the publisher accessing for end users. By this we can avoid downtime to the site while upgrading the publisher which is took out of rotation.
Once the changes are done in the publisher we can put back the dispatcher to load balancer and take out the other publisher and do the same.
If you're planning to remove the publish instance from production permanently/temporarily then you might follow the below approach:
If you are using multiple dispatchers in your infrastructure then you can remove the one pointing to publisher that needs to be removed from the load balancer so that all of your requests will go to another publisher.
You need to disable the replication agent on author instance which points to your publisher which you are planning to remove else it will block your replication queue.
Update your dispatcher render configuration to remove the IP of the publisher which needs to be removed.
If you want to remove the publisher temporarily(for an upgrade) then you can revert these changes once it is upgraded.