You are likely experiencing Opportunistic TLS functionality in the version of PHPMailer library we're using (specifically 5.2.22). This checks if the server is advertising TLS and if it is a secure connection is used. If the certificate is invalid this will fail. This means your mail server is saying "yes, use a secure connection, but our certificate isn't valid"; that's a security vulnerability and it's failing for good reason, which is that the connection is not secure. The server either needs to stop advertising TLS or provide a valid certificate.
It works for Joomla because Joomla specifically implemented a workaround for this feature introduced in PHPMailer 5.2.10.
github.com/joomla/joomla-cms/pull/9528
Have added a feature ticket to review updating our PHPMailer library to latest where this maybe better handled. There are no plans at this time to implement workarounds for, likely, incorrectly configured servers.
forge.joomlapolis.com/issues/6981