I admit, I’ve never had the need to setup a mailserver as I’ve always used a 3rd party service.
If I recall correctly, I’ve seen a mixed bag of experiences with the Mageplaza SMTP Module. You’ll often find that their free modules can be found on github, which is usually good because you can see issues people have bothered to report that might relate to your own problem. Here you can find their Github Repository.
You mention that the module works fine with Gmail, but not with your own exchange. Out of curiosity, what solution are you using for your own email exchange?
As I mentioned, I don’t have any experience running a dedicated exchange server… However, that message would make me want to check that openssl and curl is up-to-date on both my Magento Server and Exchange Server. Hopefully, it’s a simple as getting the latest version. Or perhaps SSL is mis-configured on the mailserver? I think this should update both applications:
sudo apt-get update
sudo apt-get install curl openssl
Finally, I did a Google Search for “SSL3_GET_RECORD:wrong version number error”, and there are a load of results… But they mostly go over my head. Plus, this error may simply be a side-affect of something else. Sorry, I couldn’t give a more definitive answer.