If you have a load balancer in-front of dispatcher then you can take 1 dispatcher out of load balancer which inturn removes the publisher accessing for end users. By this we can avoid downtime to the site while upgrading the publisher which is took out of rotation.
Once the changes are done in the publisher we can put back the dispatcher to load balancer and take out the other publisher and do the same.
Hope this helps!
If you're planning to remove the publish instance from production permanently/temporarily then you might follow the below approach:
If you want to remove the publisher temporarily(for an upgrade) then you can revert these changes once it is upgraded.
I assume you have two publish instances running.
Then make 1 publish down and upgrade or do what ever you want to do with that mean time second instance will be up.