Restrict field visiblity based on group/sub-group

Would love to be able to filter out fields while creating any report based on the group/sub-groups selected. We have many fields that are specific to a particular group and seeing all fields together might sometimes be confusing to the users.