I'm getting "Mail Delivery System"-mails when members are trying to get a new password...
You mean your mail server is reporting delivery failures for emails? Check your mailer settings within Joomla's global configuration as we use those. Also be sure the registration email address in CB > Configuration > Registration is either empty (will then use Joomla's setting) or set to a valid email address that can send emails from your mail server.