Modern IMAP Servers / XOAUTH2

[samarai-jk]

Hi there, I forked mailio and added SASL/XOAUTH2 authentication, tested so far with Gmail and Outlook. I've also added a few other goodies like a dedupe hash for messages (SHA256 of the raw RFC822 data) and a uid field in the message class, both to facilitate message processing. Let me know if you want these changes and I send a patch.

But I have a question... For some time now it's not possible to authenticate against Google and Microsoft servers without OAuth2, which makes me think how is it possible mailio does not have this feature yet? Unless you use the library in a corporate, controlled environment, without OAuth2 you exclude a huge chunk of real world email accounts from being accessed with it. Also, once I got the auth working, imap::fetch chocked on Microsoft servers, same issue as in #135, I patched that too. But this again got me thinking, how is it possible I'm the first to figure this out. I guess what I'm asking is: Is anyone still using this library? Is it actively maintained?

Thanks!
J

[karastojko]

Hi,
Thanks for your message. For the Microsoft servers I am aware of this issue but for the GMail I did not experience it. I did some preliminary tests for OAuth2 and I do plan to add it soon. Indeed, there is a PR on this topic but I did not merge it since I think it has some flaws. I would probably start adding this feature to Mailio 0.27 since for the version 0.26 I would like to finish few other topics (like the PR204 and some IMAP issues). So, yes the library is maintained but I do not work full time on it. Thus, I welcome your contributions.

[samarai-jk]

Yes, I saw #207 but I don't think that would have worked. Mine works, at least against Google and Microsoft IMAP servers. But my patch does not include the OAuth2 process itself, that's a whole different problem and I cannot provide that for mailio as we're doing that using a curl-like implementation based on oatpp, especially the browser-callback is rather involved.

I will do a few more days of testing and hardening and then send a patch. On that note, are you interested in other updates as well, so far I've added to our mailio fork:

• XOAUTH2 authentication
• UID-order patch, fixes fetch from Microsoft IMAP server
• SHA256 dedupe hash in message, over the raw RFC822 message data. We use this to identify messages that have been moved between mailboxes or when mailboxes are recreated
• UID field in messages. We use this to bulk import mailboxes in batches, if a batch fails for any reason we can restart without having to download all messages again. Very handy for big mailboxes
• Sequence-number field in message, just for completeness
• Noop() IMAP function, to test if a connection is still alive, used for easier detection and auto-reconnecting of dropped sessions
• List function for special-use folders
• Error simulation methods in imap and dialog class, used to simulate networking and server issues, for code hardening

I'm still working on our implementation, so there might be a few more additions to come. Let me know what you're interested in. And kudos for making this library and keeping it alive! Seems mailio is the only viable option for open source c++ projects that can't use GPL libs. Thanks!

[karastojko]

Thanks for the offer, I am interested to merge them. Just please make separate PRs for different topics, so I could follow the changes easily. I have additional semi-automated protocol tests which are not published on GitHub, so please be patient until I run them against your changes.

[samarai-jk]

Hi There, just wanted to give a feedback. I haven't forgotten about this, in the contrary, I've spent the better part of the last month working on this. The result is a heavily changed fork.

Here's some of the major issues we encountered (and solved):

• the biggest 2 email providers are not supported, no OAuth2 support. We added this
• message parser is much too strict. For example, I have 50k emails in a history going back to the 90s. When I tried fetching these (in non-strict mode already), a big number of messages threw errors because they do not conform to RFCs that many email servers do not enforce. From a strict RFC-POV correct, but the real world is messy. It's a hard sell to our users if their messages are missing when every other email client ignores the problems and does best-effort approach. So we had to massively widen the non-strict mode and change general message/exception handling when fetching
• the fetch command throws at every little problem it encounters. Again, from RFC-POV correct, but hard to use in the real world. How am I supposed to fetch 10k messages when my whole 200-message batch fails because one of the 200 messages has a parsing issue? We completely rewrote and extended the fetch method, also adding many message details that were missing
• multi threading. IMAP is by design synchronous, from that POV that's not an issue. But when building a complex client on top of the lib, there are multiple threads that need to access the lib, even if just to disconnect on shutdown. Which made mailio crash. We had to rewrite big parts of the dialog class
• the most complex feature that was missing is idle/push messages. This turned out to be a major headache. Biggest issue was of course the protocol itself, that's a b to implement. But even so, not an easy library feature to use because this needs to be multi-threaded by design, it's pretty tricky on the client side too. Idle/Push is not a trivial implementation at all. But we got it to work, at least in alpha-state for now
• multiple IMAP sessions in the same process, which mailio wasn't happy about at all. necessary for idle/push to work but also for multi-account support. Very closely connected to above general multi-threading issues. It works now
• lots of smaller missing features and improvements that are necessary to synchronize full accounts with 10s of thousands of messages, too many to list here
• general debugging/tracing/testing features as the lib was a black-box which made IMAP protocol debugging a huge headache (imagine to debug a multi-session idle/worker scenario when you have no idea what's happening in the IMAP sessions). We had to go so far as to install hooks into OpenSSL to be able to decode the TLS traffic in Wireshark to solve a particularly nasty issue with Gmail, which ended up being a big-fat, long-standing bug in Gmail itself

So... I'm not sure what to propose. I am still not done with our email implementation, I still need to implement change-synchronization and SMTP sending, that'll keep me busy for a while. I would very much like to contribute to the mailio library but I am not sure if I practically can. The changes we made are big and many, of which I am sure many you will not be happy about, we bent the library to our needs, which is not necessarily where you want the library to go. And it's now very difficult to extract single features from the overall changes, just because we touched so much code.

The app I'm working on is also open source, email is a comparatively small part of the overall product, it's not an email client, email is "just" one of many modules. We're a startup with a small team and a lot of chaotic pressure, as soon as I'm done with emails I need to move on to the next construction site.

I can imagine that many of the feature we implemented might be of interests to others. Just not sure how to get them back into your repo. We can always just keep our our fork as reference but to merge single features back won't be easy. We have no interest in maintaining our mailio fork as a "standalone" library and will always just maintain the features that we actually need, so our fork won't ever be a replacement but become part of our bigger project, eventually we might even strip out the features we don't use.

What do you think? Happy to chat in person if that helps.

J