Thanks for proposing the idea but looks like it conflicts with the actual functionality as smart collections uses a search criteria to dynamically populate assets and thats how it is different then a staitic collection so it won't make much sense to add/remove assets manually to smart collection. I guess a better idea to exclude would be to adjust the search criteria.
Let me know if there are any additional thoughts on the same.