One of the project managers would occasionally accidentally add a user to a task eg she meant to add John Sample but added John Smith instead. She realizes the mistake and removes John Sample and change it to John Smith. However, John Sample is now shared into the project. The PM could remove John Sample via the project share or the Staffing/people tab. Or she might forget and potentially have 2 people with the same role working on the same project or inadvertently give John Sample visibility to the project when he shouldn't.
I'm not sure how it can be handled as I know some users would need visibility without being assigned to a task but what is being asked for is if the user is not assigned or removed from the task, their visibility to the project is removed as well.
If it happens often enough, it quickly becomes an annoyance.