Hi Var,
It is expected behavior & you need to unselect each component. Technically when you select a group under design ex:- [1] it just stores a group name & not list of components. When you unselect it removes the group name & you can unselect which ever component is not required.
[1] /etc/designs/geometrixx/jcr:content/homepage/par
Thanks,
Sham