The Monstrosity Email Has Become
by Ploum on 2021-10-20
Lot has been said about how the web evolved to become a kind of monstrous entity. If you are on Gemini, you probably see what I’m talking about. The mail protocol has followed a similar evolution but it’s a bit more subtle and has often been summarised as « too much email. ».
I’m currently thinking deeply about Offmini, a protocol which would be to email what Gemini is to the web. This prompted me to write about what was bad with email. This post was written as part of another one where I described how I’m building Offmini. It became so long I figured it should be an independent post on its own.
Sending Problem
The first and obvious problem with email is that it has been developed 40 years ago as a receiver-only protocol. 15 years ago, most of the mail traffic was random spam. By random spam I mean that spammers were really generating random mail addresses (or scrapping them over the web) and sending trillions of emails to every possible address. You could receive spam in a language you didn’t even speak.
Complex protocols have been added on top of SMTP, relying once again on DNS (SPF, Dmarc,DKIM), to try controlling the spam by making email a sender-and-receiver protocol (a notion I will describe in a subsequent post). This had the side effect of making it harder and harder to set up your own mail server.
As homemade mail servers were harder to build and less reliable, email started to consolidate into a small oligopoly: Hotmail, Gmail, Yahoo and a handful of others. Having lots of power, those huge monopolistic beasts could easily reject mails from independent servers as spam, making their service even more attractive to customers. It might not have been on purpose but it was the case (source : I maintained multiple independent mail servers between 2000 and 2010).
One clear consequence is that you can’t send email from your computer anymore as it was originally intended. You need a mail server properly configured with a permanent connection to act as a sender identity. And even that is though as Cory Doctorow learned.
Format problem
The impact of monopolies on email impacted the format of emails themselves. Emails are encoded in a very impractical format called MIME 1.0. By today standard, this format is very hard to parse and has been awfully abused with HTML emails, trackers, etc. Basically created on a napkin by two guys, MIME never went further than 1.0 because nobody agreed to upgrade it. It was too successful too quickly.
There was only one major addition to MIME. And it was not a planned one: the infamous winmail.dat format.
At the time, there was a bug in Microsoft Outlook, the main mail client on the market, that transformed outgoing emails in a cryptic file called « winmail.dat ». The problem was that Outlook itself was able to decrypt winmail.dat files. As Outlook had a dominant position, only people not using outlook were having the problem of receiving empty emails with only one attachment that they could not open called « winmail.dat ». Users of independent mail servers relying on the open source Squirrelmail webmail interface started to blame their mail administrator (myself, for ten years).
The problem became worse when Google reverse engineered the winmail.dat bug in order to transparently support it in Gmail. At that point, the winmail.dat Microsoft bug became part of the MIME standard without any specification having been ever written.
It may gradually fade out but most mail client still have the code to deal with the winmail.dat never documented format.
Email has a bad protocol for sending, a bad file format but what about receiving? It’s not better.
Receiving Problem
As Vint Cerf acknowledged, the whole IP stack was built at a time were memory was really expensive. There was a feeling that storing data would always be more expensive than sending them directly. That’s the reason why we envision the Internet as real-time connection only. As soon as you unplug your computer, you are outside of the Internet. Each software has to deal with disconnection independently. Usually it’s by popping up an error in the face of the user and telling him to check his connection.
As it was clear that most personal computers were not connected all the time, yet another protocol was created to retrieve emails from mail servers and store them on a non-mail server computer: POP3.
The protocol was bad enough not to allow any synchronisation for stuff like folders or marking mails as read. Once again, the goal was to save memory on the server: users could download mail once for all and they would be removed from the server (there was an option to leave them on the server but this was rarely used as space was limited and, with some clients, you had to regularly redownload all the emails).
IMAP was introduced to fix POP3 flaws. IMAP is a very complex protocol with fuzzy part open to interpretations. It’s well known by mail administrators that some particular IMAP servers are not compatible with some particular clients. IMAP was so severely abused by providers that most mail client handle some mail providers separately (like Gmail or Outlook).
IMAP was also created with a permanent connection in mind and most mail clients expect a connection, throwing an error at each offline action. If an offline mode exists, like in Thunderbird, it must be manually configured and is clearly an after-thought.
Worst of all: every IMAP client store emails in its own particular way. There are multiple standard, like Mailbox and Maildir, but each client seems to have its own interpretation of it. If you ever browse Stackoverflow, you will find lots of people asking a naively simple question : « How can I access my email locally with Mutt on the command line and with a graphical mail client ? » Sounds reasonable, right ? After all, you did all the increasingly obscure work of configuring mbsync/isync to get your mail on your local computer (which randomly stop working from time to time until you realize that not receiving any emails in a few days is not normal) in a standardised maildir format, not to mention the cryptic config file you had to copy/past to be able to send email through Postfix, why not accessing them with something other than Mutt ?
Right ?
Well, you can’t. Or, as all Stack Overflow answers will tell you, you « only » need to install an IMAP local server and make the graphical client point to it using your local 127.0.0.1 IP.
Simply have a look at how to use Himalaya, a neat and fresh CLI mail client, offline.
Email had become a monstrosity beyond reasonable comprehension while still having inherent flaws such as plain text sending. Every email out there is sent and stored in plaintext (we can easily agree that PGP/GPG use is anecdotical) and, through HTML and inline pictures, most of them are trying to track you to know when you open the email.
The whole ecosystem is becoming even more and more centralised with some modern mail providers not offering the ability to get your mail out of the service at all, arguing, with reason, that IMAP sucks and does not permit some features (the hipsterish Hey! or the privacy-oriented Tutanota only provide you access to your email through their own proprietary webmail). You can’t even read your mail offline by design and nobody blink an eye.
The spam problem
But, at the very least, we have solved the spam problem, haven’t we?
According to my own statistics, we indeed solved most of the random spam. The spam that was plaguing the network 20 years ago seems to have been greatly reduced or, at the very least, is easily blocked.
But, instead, we are now receiving 10 times the amount of what I call « corporate and notifications spam ». Unsolicited emails that come from real identified companies and people. They send you thousands of emails that, they think, should interest you. Most of the time, you can quickly identify why you are receiving this email. It’s linked to one of your accounts somewhere. At worst, your email has only been sold to « trusted partners ». They always provide you with the option to unsubscribe even if they are very sad to see you leaving. Unsubscribing only works for a short time because they basically create a new mailing list for every mail sent and this one should really interest you. The tracking of users is there by default in most mailing-list tools and show marketers that most of their emails are never opened. Which prompts them to send even more emails, arguing that, at some point, you will be tired of not opening them. Providers like Gmail heavily spy on how to use email. It is widely known in the mailing-list community that Gmail learns to mark as spam mails from senders which are rarely open. Prompting marketers to change regularly their newsletter address and to try to make a catchy title. Sounds familiar?
This effect is worsened by the fact that email has become the lazy default for everything. If anything is happening on a service, even a non-commercial one, mail is sent. Facebook and Linkedin are quite infamous for regularly adding « notification categories » where you are subscribed by default, even if you previously find the hidden setting « unsubscribe me from anything ». Besides lazy engineering, as Szczeżuja points out in the link below, it is obvious that it is a cheap way to remind their users that they exist.
We are now forced to rely on hundreds of very centralised web services for everything and each of those services, by default, fill your inbox. When you enter a new job in a big company, the first action of HR is to subscribe you to a bunch of mailing list. It’s even worse in the academic sector. From the « Weekly news » you don’t care to the « There’s no paper left in the printer from the second floor », email has become a centralised broadcast network. You are forced to be on the receiving end while every central authority known to man tries to broadcast as much as it could.
Your mail inbox is becoming a battlefield where everyone with a small authority fight for your attention, trying to fill your mental space even if you don’t open the mail. I’m an Inbox 0 Taliban and I’m mortified each time I get a glance at a « normal person’s inbox ». It’s basically a long list of companies (lots of Facebook but also local companies) displayed in a long list where only one mail out of ten has been ever opened. Ever wondered why Gmail doesn’t display advertising in its interface? Because it does! All that mail neatly lined up is basically cheap advertising. What’s Google’s benefit? The clumsier your inbox is, the more their automatic triaging rules look appealing. Google is already deciding for you what to write (by suggesting you how to reply) and what is important to look at (with their « smart folders »). It’s not far-fetched to imagine that, at some point, you will need to pay Google for your emails to be important and not considered as spam. Maybe there are already doing it through some kind of « trusted partner program ». It’s, after all, the reason why Facebook created its feed: choosing for you what is important to see and monetising this access to your brain.
There should be a better way.
I’m an inbox 0 extremist. I unsubscribe from everything that contains an unsubscribe link. I spent the last two years sending GDPR removing requests to every company sending me an unwanted email. The first three months were completely exhausting but once I did the first bulk, it became more rigorous hygiene.
I can tell many anecdotes about how companies handle GDPR requests, how I found that I was in some commercial databases, how I tracked down the owners of those databases. How I permanently removed more than 300 online accounts and how I sometimes receive a very unexpected email from companies one year after they told me every information about me was removed. Or how I got stuck in a loop where being unsubscribed from a newsletter required posting on their support forum but they automatically subscribed everyone posting on their support forum. Latest story in town: Lying companies telling me that they had removed my data while I still can log in! (they simply renamed my user account as « removed » but didn’t delete anything).
Funny stories about human stupidity and dishonesty but, in summary, it mostly works. The effort pays off.
My inbox rarely has more than 5 emails at once. In the last year, I received only 20-25 unwanted emails every month, including every random spam and phishing attempt. Half of it is spam from marketers that want to advertise on my website.
But how many people are able to put this time, effort and dedication only to get a less polluted inbox?
Not to mention that, despite all my motivation, I have not yet been able to build an offline setup where I could read/reply to emails offline, having everything synchronised once I connect but still use the webmail if I want. Mbsync configuration is cryptic and randomly stop working without reason. Some mails are, for reason, never downloaded. I know it should possible but it is very hard or too convoluted (no, I’m not installing a local IMAP server on my laptop).
I spent so much time on this because I love emails. But there should be a better way.
As a writer and an engineer, I like to explore how technology impacts society. You can subscribe by email or by rss. I value privacy and never share your adress.
If you read French, you can support me by buying/sharing/reading my books and subscribing to my newsletter in French or RSS. I also develop Free Software.