Hi Michael,
Yes, In your case you need to get rid of all the pending left out task. As you said you can delete.
Why not create groups during workflow execution using process step and assign using dynamic participant step.
Once workflow is completed delete group.
This approach could be more clear and would have less overhead.
Just make sure create a common group which have all the necessary permission to work workflow and create group add user from process argument or from other places based on your use case.
Thanks
Arun