> If you're not really that curious, suffice it to say that the people who design the protocols and systems understand the attraction but have excellent reasons not to do it.
For those who are that curious, what are the reasons?
The sentence directly before that, from Mark's original e-mail, which I elided, is now horribly sad given your question and today's context.
"""If you're just curious as to the reasons, and don't wish to argue it, I'll gladly explain in private email (offline from the list)."""
To be clear, that was all a quotation from an e-mail written by Mark Crispin: I do not really claim to be an expert in an of this yet, as there is still much more reading I have been meaning to do regarding Lemonade.
I guess I can try, however, to find more Mark Crispin quotes from various mailing list posts to help answer the question; he seemed heavily involved in the 2003 discussions of the Lemonade Submit proposal.
Things seemed to come to a head in 2005 with an argument between Marc Perkel and Mark Crispin. This argument actually resulted in Mark Crispin petitioning for Marc Perkel's removal from the mailing list.
"""> IMAP can move messages to and from the server. So if you have a protocol that can deliver messages to a server then why not let IMAP hand off outgoing email to SMTP?""" -- (Marc Perkel)
"""The fact that two protocols have related functions does not mean that those two protocols should be merged into one protocol. Otherwise, everything would have been TELNET protocol in the 1970s, and HTTP protocol today.""" -- http://www.ietf.org/mail-archive/web/lemonade/current/msg014...
Concurrently to this (possibly causing it) was a new thread started by Mark Crispin with the subject "SMTP-over-IMAP harmful" that has a long list of reasons for why IMAP should not be extended to allow for directly sending mail.
It is probably most appropriate to read that message and that thread to get a summary of the kinds of issues being discussed, so you can get statements "straight from the horse's mouth".
(For clarity to those not familiar with the various e-mail specifications, I will point out that the "submission" protocol mentioned is the port 587 "for users, not e-mail servers" variant of SMTP, not the IMAP Submission specification; though, watch me actually be wrong ;P.)
This isn't the complete context of the discussion, though; that would require spending much more time digging through these mailing lists posts. From the chair of the committee:
"""1. For the folks that have been here from the beginning, please have a little patience when someone comes in with an idea that is 10 years old, got a year of intense scrutiny, and got rejected. Not everyone comes in with the amount of collective wisdom that you have. [Note appropriate use of "you". I mean "you" here.]""" -- (Eric Burger) http://www.ietf.org/mail-archive/web/lemonade/current/msg014...
Mark Crispin actually wrote a memo, called "Message Submission" which provided alternative ways to do this, and which involved a bunch of things I have not looked much into yet (such as BURL). However, this memo also came well before the comment I quoted in the message to which you replied with this question.
I was part of the Lemonade group for a time, and one of the big arguments for a new submission technique, when it was proposed during my membership, is that ISPs were doing more and more blocking of port 25 and 587 from user connections to fight spam, and some mobile carriers had even more strict filter. I worked for Oracle at the time and we were actually proposing SUBMIT-over-HTTP because, drum roll, HTTP routeability was less complex and more predictable, and far less likely to run into end user customer support issues. I don't know who proposed the benefit as 'tcp connections are expensive' because sending mail is a rare event. We were much more concerned with traversing 3g networks, ISPs, and corporate firewalls.
There's also cases like, if you want to forward an email, on a mobile device, you'd have to download the email + all attachments, and then reupload the message through SMTP. The Lemonade group came up with hacks to do forward without download (BURL + IMAPAUTH) or CATENATE, but my guess is, if you have an advanced mailstore backend shared by the IMAP and SUBMIT servers, it could be a lot simpler if the servers were unified.
I do think the IMAP group was often too pragmatic, too stuck on maintaining compatibility with ancient systems, and not as concerned with end user / customer issues, and more concerned with the feelings of a few implementors.
When you consider someone like Google implementing mail on a BigTable cluster, or Oracle on some of their RDBMS (Oracle Collaboration Suite), putting forth arguments that new features must somehow be relevant or not upset ancient mbox based systems seems silly.
I think if one were designing a mobile email client and scalable server today from scratch, you'd be crazy to design a protocol like IMAP.
> ISPs were doing more and more blocking of port 25 and 587 from user connections to fight spam, and some mobile carriers had even more strict filter
Honestly, I personally (and FWIW, I understand that my personal opinion is not terribly valuable here ;P) feel like being part of an arms race against the ISP to not get blocked to be a battle that isn't ever going to end: I mean, even if you get to the point where they can't look at the traffic stream and see that it is SUBMIT-over-HTTP (as maybe it is SUBMIT-over-HTTPS) they can always block by endpoint. Is the next step SUBMIT-over-TOR? The devices that people are using are getting more and more locked down: they can just start checking the data streams before they are encrypted.
> we were actually proposing SUBMIT-over-HTTP because, drum roll, HTTP routeability was less complex and more predictable
Yeah, if a design criteria is "not get blocked", then "over HTTP" seems like a better option. For outright simplicity, you could even just have a very tiny shim protocol (along the lines of the first step of WebSocks) that allows an HTTPS connection to upgrade to an already-authenticated SUBMISSIONS connection that is nothing more than a direct proxy through to a SUBMISSION server. This could be specified in just a few page document and, as it doesn't allow access to the SUBMISSION server until you've already authenticated against the HTTPS one, probably adds fewer security issues than might otherwise be imagined.
(Although, if you want to deal with HTTPS blocking, you'd need something more complex; but if you drop all the way down to HTTP you only gain yourself a couple years before your opponent just starts blocking you using deep packet inspection: if the ISP wants to block you, and they have made it clear that they do by explicitly and knowingly blocking port 587, and you feel that the correct way to fix these blocks is using technology, then if you don't go the full HTTPS route it seems like nothing more than a waste of everyone's time; and once you go HTTPS, you get the advantage of guaranteed-to-be-proxied bidirectional sockets.)
> if you want to forward an email, on a mobile device, you'd have to download the email + all attachments, and then reupload the message through SMTP
Yeah, this is definitely a good case for having some kind of integration. I will again state that I haven't yet looked into BURL, but I intend to do so to get a better understanding of the issues involved here.
> When you consider someone like Google implementing mail on a BigTable cluster,
The main thing I've run across that is difficult to implement in this context is that of message sequence numbers, especially with regards to how they interact with EXPUNGE (but also somewhat in general). I almost brought this up to imap-protocols a couple months ago, but I found a solution (as yet unimplemented: right now I'm still pretty-much-linear for that one case of EXPUNGE as I rebuild my MSN cache; it turns out there are some cute algorithms that should allow me to store a fairly small block of data in a single row and update it fairly efficiently during EXPUNGE) that tided me over long enough to now see Brandon Long bringing up the complaint himself.
If anything, I'd be then looking for just one small extension that I call (temporary name) UIDFTW, which when ENABLEd removes the client's ability to ever again use a message sequence number, forces the return of UID in untagged FETCH responses, forces the return of VANISHED instead of EXPUNGED, and replaces the message sequence number in an untagged FETCH response with "*" or "-" or "NIL" or "0" or some other "blank" token. This feature would be trivially implemented by existing e-mail servers: in fact, the first and final requirement could be made optional (as the client now also knows to ignore them) making this somewhat equivalent to the requirements of QRESYNC for servers that have no trouble with message sequence numbers.
However, for servers that do have a serious issue attempting to maintain message sequence numbers, they could then "relieve their burden" once this extension is activated, and not have to generate or keep around a message sequence number map for sessions that ENABLE this feature. If it is then supported by just a couple major clients, that will already be a help for servers that were willing to implement message sequence numbers, but just hated it from a resource-management and performance perspective. But, you could go further: one might eventually see clients that simply refuse to talk to a server that doesn't have it, or vice-versa. This, though, seems like a highly-minimal modification to the existing system that provides a very nice forward-looking path to something that has no problems with "big data".
Otherwise: IMAP seems fine, but I certainly have nowhere near as much context as you do; I would thereby find it utterly fascinating if you'd be willing to elucidate any (or even just one) other issues (than the message sequence number problem) that I haven't yet noticed that are going to cause me serious problems as I polish up my "big data" implementation of IMAP or make more progress on a mobile IMAP clients (not certain if I will actually be releasing either publicly, but certainly intend to switch to the server for my company's needs in the near future, and would start using the client myself).
For those who are that curious, what are the reasons?