It’s not possible to exclude (parent) categories from showing up, unfortunately. The widget (as well as the main navigation) automatically displays all parent categories and categories that the current user has the permissions to view.

Thanks @bjoern_schulze aware of the basic configuration options and how it works, but was interested in finding a way to hack it. In our case we have an introductions category and depending on volume it can result in the Recent activity tab being intro heavy. My hope was maybe there was a way to exclude a particular category from showing up there or to be able to move those to a separate tab.