Uninstall all CB related extensions in Manage > Extensions then delete all database tables with the following table prefixes.
_comprofiler
_groupjive
_cbsubs
Next be sure to clear out any Joomla menu items to CB as they won't work anymore.
Would the users still remain in Joomla?
Yes.