Ensure CB Auto Actions is installed then within CB Auto Actions > System Actions publish "CB Groupjive - Stream" for group specific activity streams.
I have some groups for invite only but in my activitystream i see all the private comments.
Not sure what you mean by this. Group activity should only be accessible to the members of that group. The exception being if you are a CB Moderator then you have access to all groups.