Possible to exclude a segment from an email campaign?

Feels like there should be a way to do this, but I can’t figure it out. 

Scenario: We have a handful of groups where we want to exclude the group members from receiving our community newsletter. 

I’ve created a segment with these members, however there doesn’t seem to be a way to say “Email Campaign, don’t send to this segment”, or better yet, to create a segment that excludes another segment. 

And while I can use filters to say that I don’t want people in X, Y & Z groups, that only returns members who are in other groups, just not the ones that I indicated. Excluding the bulk of our members who are in no groups. 

Any thoughts or ideas are greatly appreciated!