This is an intermittent issue due to the current design of the Audience library UI. There are Asynchronous calls happening backend with "Create_Trait" and "Fetch_segment_size" requests in any random order.
Fix for now is to keep trying to add filter one by one, add next filter only when you don't see the "failed to fetch_segment_size" error upon adding a filter to the audience. If it gives you the "failed to fetch_segment_size" error, try again. Keep on adding filters one by one, till they all have been added, finally save the audience.
I understand that it is a bad experience indeed, however the permanent fix for it will be a big design change which has been reported to the engineering team and is under maintenance by the Product Team further.