From a client standpoint, POP is not actually trivial.
The main problem with POP is that unless you do something clever, determining the changes in the mailbox from time t0 to time t1 is both conceptually difficult and computationally expensive. This is because generic POP has no concept of a message UID, so there's no principled way to diff between states. In a very real sense, this means that POP is somewhat broken for the most important use case for the client: syncing to server changes.
The UIDL extension adds a "UID" but it's just an MD5 hash of the contents -- meaning that multiple copies of a message appear to be the same message. And you can ask for just the headers -- which means you can get the message-id header -- but this is still very expensive to do repeatedly (say, every 5 minutes) on a 100,000 message POP store. And you can't ask for just the Message-Id header, which would fix the problem.
Even if you have valid UIDs -- which you won't -- you still have to run a diff algorithm. Typical dynamic programming algorithms are O(N^2), which obviously sucks big time for a 100,000 message POP store.
For Inky (http://inky.com) we use a clever linear-time diff algorithm based in part on [Meyers 86]: An O(ND) Difference Algorithm and Its Variations. [Burns & Long 97] A Linear Time, Constant Space Differencing Algorithm is also a good treatment. But I know Outlook and Thunderbird both use non-linear-time algorithms to diff, so "leave messages on server" gets increasingly (non-linearly) expensive as the mailbox size grows on the server.
A few other points on comments made in this thread:
- POP is still widely used. In the US, for example, Comcast has finally migrated to IMAP, but Verizon is still POP only.
- POP is, from the server standpoint, a very simple protocol, and it is highly amenable to automated testing, as others here have pointed out. For our own testing we generate both patterned and random mailbox modification sequences, then have the test client cooperate with the test server to ensure that the client has (independently) correctly determined what's happened to the (test) mailbox. This is a perfect example of a situation where investing significant effort into automated testing pays off -- and where a TDD approach to development would also work well.
The UIDL extension adds a "UID" but it's just an MD5 hash of the contents -- meaning that multiple copies of a message appear to be the same message.
That sounds like a server problem; to quote the RFC,
The server should never reuse an unique-id in a given maildrop, for as long as the entity using the unique-id exists.
Why would you hash the contents? The arrival time should be unique, assuming no two messages could arrive at exactly the same time. That doesn't require any hashing.
I think POP could've been far better designed, without growing into the complexity of IMAP, with just a few little changes like this.
That's just good advice for dealing with modern software, for which fixes, breakage, and feature churn are all mixed in a single awful stream. "Newer" does not mean "better."
I use near weekly updates of a custom Android 6.0 ROM and daily use Arch Linux which is a rolling release. My work is on the web with npm keeping us bleeding edge. Other people often ask why my system does this little extra thing that's useful. Usually the answer is found by --version.
Life with up to date tools can be real good. So unless I'm simply the only one using good tools, updates can be good.
I think it's a mixed bag, to be truthful, and it's a circumstance that's very well described by "pick your poison."
My primary OS is also Arch Linux, and while it's certainly stable, it's not without its warts. Failure to update regularly on a rolling release distro can have absolutely disastrous consequences (though you only have yourself to blame), and a healthy dose of caution is strongly recommended whenever a major update to important software is in the pipeline (think KDE4 to KDE5 transition). I think this sort of bug (Outlook) illustrates the importance of having an abundance of caution with new software where the failure modes may not be well understood by merit of its relative youth. But with rapid releases, I think the problem is a bit more focused on the end user: Someone who is unable or unwilling to take the risk of updates causing material harm to their workflow or consuming time they can't afford in order to fix potential problems should look for more conservative release cycles. I don't think it's really a matter of "good" versus "bad" tools; that may be part of it, but I can't help myself from thinking it's a matter of misplaced expectations.
That is, it's easy to fall into the mindset of erroneously believing that faster, more rapid updates is always better without fully appreciating their impact. (I've done this more times than I'm willing to admit.)
I do think, and maybe I'm wrong (which I usually am), that those of us who tend toward using rolling release distros have a bit of a bias and a rose tint to our glasses. We almost innately know what the risks are, and I think we take that for granted by assuming most others will freely accept such risks and appreciate the occupational hazards that go hand in hand with change. Not everyone has the same degree of patience, nor the same goals or motives. I think our optimism for and evangelizing of software that pushes rapid releases (like Arch, as an example) can help create an aura that lulls those we influence into expectations that don't mesh well with their use case, their personality, or their constraints. Is that a bad thing? I don't think so (evangelizing is important), because I do agree with you: Updates can be good. I just think we're all too happy to espouse advantages while sometimes glossing over potential drawbacks (guilty again as charged!). :)
Anyway, I should apologize: I didn't mean to wax philosophical. It's late in my timezone, and I saw another Arch user who provoked me into a short essay. I agree with you and username223, but I don't have any real answer. I do think that sometimes we ought to be more cautious with our advice and perhaps weigh context more heavily than our excitement allows. (I made the mistake once of suggesting Arch to someone who really ought to use something with sturdier training wheels. My only saving grace is that he never got around to installing it.)
It depends. With rather new still evolving ecosystems, every update is welcome. Also the security updates are important for OS and browsers.
But with stagnating platforms, like Windows where tipping point was Windows XP (or Windows 7), it goes downhill with every update since then, the bubble bursted, now the user is the product, and your files get screened and searched by the platform owners. You can run software from 1985 (Win 1.0) on Windows 7 (32bit, but it's just an arbitrary limitation to prevent 16bit applications from running on 64bit OS, the competition can do that see Wine and ReactOS). Almost every developer already moved on to the web or emerging new platforms like the market share dominating Android or the second most popular one, iOS/OSX. End users are smart, they don't buy into old antique burned platforms of yesteryear anymore. They application landscape is changing as well. And for everyone who is still happy with their old software, there is little reason to update, and it won't get better on sinking platforms.
No, its not an arbitrary decision - there are some pretty decent reasons why Microsoft declined to support running 16 bit apps via NTVDM/wowexec - virtual 8086 mode isn't supported in long mode - now certainly, they could have ported the functionality from NT 4 they used to get NTVDM working on non x86 processors (it included an i486 emulator) - but to what end, realistically, how many people are still running 16-bit only software?
Another thing I'd point out - Microsoft almost backed themselves into a corner worshiping at the shrine of backwards compatibility - to the point it was difficult to move their platform or their ecosystems software forward to use more modern, more secure and more reliable methods - so unless you've been very forward looking from the start (see IBM System Z) there is a real, fundamental and painful engineering cost to maintain a line to yesterday without great sacrifices to tomorrow.
I'd argue that the PC (be it Windows, OSX or Linux) is here to stay for the foreseeable future - it may not be the platform for everyone - but for many workloads and applications the web, or mobile simply will not do.
> unless you've been very forward looking from the start (see IBM System Z)
Can you elaborate? System Z always looks curious, but I don't think that many people who aren't involved with mainframes professionally had a chance to even look at it
> Almost every developer already moved on to the web or emerging new platforms like the market share dominating Android or the second most popular one, iOS/OSX.
I (Moto X 2014) got a different set of bugs on M. Mostly problems with basic functionality:
- When I get called, half of the time I don't see the name of the person when they are in my address book.
- Music controls (which were already very basic) usually stopped working a certain amount of time after the last restart.
- Music randomly pauses when browsing the web at the same time (as in, I have to open up the music notification card and press 'Play' again).
- E-Mail notifications have been flaky for some reason.
- Active display music controls are worthless again. Google decided to add a 'favorite' button. Motorola just picks the first three buttons from the notification card. Now you can't go to the next track anymore.
Security updates were also still at the 1 November 2015 level. After being stuck for half a year on a buggy 5.0 release before getting 5.1, I know this is going to take months to fix, if ever (Lenovo probably doesn't care about the 2014 anymore).
I am now back to an iPhone after a Nexus 4, Moto X 2013, and Moto X 2014.
Now that you need to bring your own phone to most carriers, there is no reason not to stick with Nexus/pure Android. The updates work there for the most part.
I almost totally agree. If it wasn't for security fixes, there would be no value in updating almost any software. If it's already working OK, just leave it alone.
15 years ago when people had problems with their graphics cards, the standard "fix" was to update the driver. Now we're still updating drivers. Weren't the problems supposed to have been fixed many years ago? It seems they introduce as many new bugs as they fix, making the net effect of updates useless as far as bugs go.
For security related bugs, stop using C++ for internet facing software.
"Now we're still updating drivers. Weren't the problems supposed to have been fixed many years ago?"
Your statement makes sense if you believe that graphics card manufacturers were still working on drivers for 15 year old cards and 15 year old APIs.
Manufacturers are always putting out new cards, and there are new APIs to support. Writing drives for dozens of chips spanning multiple generations of graphics architectures to work on multiple OSs and supporting multiple APIs is incredibly challenging.
Even if there was just one chip and one OS and one API to support, getting maximum performance across multiple applications is non-trivial. Drivers have a lot of heuristics to try to maximize performance, and manufacturers invest heavily in tuning those heuristics even to specific games to get the best results.
If you aren't a gamer, then, yeah, you can run three year old drivers and it won't affect the speed of scrolling the text in your browsers or whatever.
Also, most games have unbelievable bugs that currently get caught and mitigated by drivers. So whenever a new game is released (or updated) a new driver has to be released.
It's hard to explain to people, who aren't techies, why I, as a programmer, don't have the latest and greatest software installed asap.
I still use Visual Source Safe for my personal projects. It's never failed me in 20 years. Right click, check out, right click checkin. Perfect every time.
> I still use Visual Source Safe for my personal projects. It's never failed me in 20 years. Right click, check out, right click checkin. Perfect every time.
1st reaction: Horrified that you are still using VSS.
2nd reaction: Was that if it does what you need it to, well, then, who am I to judge? It doesn't matter whether the tool was created yesterday or 20 years ago.
3rd reaction: Does it really do what you need it to do? I would guess that maybe svn/git has some simple featured that would significantly improve your life.
4th reaction: Wasn't VSS known as being super buggy? Is it just luck that you've escaped failure in 20 years?
I've used git on several hosts and subversion back in the day. I have no problem learning new things. Tortoise, bitbucket, github, beanstalk to name a few.
I think your reaction is more horrifying than me not having a compelling reason to change the way I've been doing things without flaw for decades.
So long as it works for you, thats great - I just hope you have a migration path if/when it stops working.
I do however get the objection to newer is better, I keep getting besieged to move my projects from svn to git - to which I usually respond "Why, tell me what feature we need in git, that svn doesn't do?" I've yet to get an answer.
If you "keep getting besieged to move [your] projects from svn to git", I can guess that who is asking is probably downstream consumers of your projects. What they might actually be wanting:
- Familiarity. Everyone uses git nowadays; like it or not, svn projects are the odd ones out.
- Fast, offline querying of the project's history. With git, they have a full copy of the project's history in their local computer, while with svn, any query has to go to the server. This helps a lot when chasing regressions, or just when browsing the changes between one release and the other.
- Easy branching. Branches in git are more lightweight than branches in svn, and git's merge functionality is quite good. When they want to propose some change to your code, they can just create a branch in their local copy, make the changes they want, publish the branch somewhere, and ask you to merge it; this is made even easier by sites like github. With svn, unless they have an account in your svn server, they have to do it the old-fashioned way.
I'm not trying to be mean. If one is going to stick with an obsolete source control system--as an important part of one's work process--there should be a good reason for doing so; however, one cannot make such a justification based on ignorance of the alternatives unless the two hours to learn, e.g., git, cannot be spared and would not be recouped in future productivity.
I don't know if they fixed VSS in later versions, but I used it in a team environment from 2000-2002, and heard of other people using it similarly even later than that (including for art assets as well as code...), seemingly without any problems.
As well as the oft-unjustly-maligned check in/check out model, two other things it has going for it are that administration is pretty easy and it has both a command line client and a GUI client. (Which might sound like a ridiculous thing to say, but I have used some systems that have one but not the other - hopefully very rare these days though.)
This isn't really a recommendation for it, though, unless they've put a huge amount of work into it over the past fifteen-odd years (which somehow I doubt).
To get the same check in/check out workflow, you could try Perforce (though I heard from somebody that did it on AWS that the initial setup can be a bit fiddly) or SVN (though check in/check out evidently isn't quite how it's designed to be used, because (a) this is not the default, and (b) the performance can be pretty crappy).
Yeah, for real. People think that, like grease monkeys, people who work with computers, and like them, should have the latest and greatest whatever the hell app is going around. Because you know it's going around, like measles. Others, like worser diseases.
A friend and I, him no longer on Facebook, me without it at this point, both have stickies on our cameras all day that we peel off to talk on Facetime. And as we were chatting, he mentioned someone who was a power user but didn't know much. He was downright promiscuous with the software he used, just, terrible. He never even thought of reading a EULA, had every app that fit on his phone (while my friend and I want nothing to do with smartphones), knew how to use all the features...Perhaps we're slightly paranoid users, but it's these tiny religions that keep us from spilling our guts to the whole planet. And it happens, and there's no undo. I know what computers can do, and it's not fun to think about what happens when yours is commandeered. Especially if you look at the situation antagonistically and as a programmer.
I think of old applications, and old computers, the way I think of generic pills: you know what you're getting, the luster is gone from the competitor that once made it look like shit based on vaporware promises, you can trust it no problem, and it's super cheap. And you're used to it, you know the rules, you know how it works. New hardware is expensive, but new software is worse: you pay either in loss of data, highly-targeted advertising, price discrimination when you finally buy something, irresponsibility with data, worrying about getting hacked, getting hacked, countermeasures, getting spammed, or in a mundane but expensive way as with the $5 a month that Danish guy said every app should charge. But most apps just can't man up and charge you a fee, or tell you to go screw, instead, they play it sleazy, they want to traffic a tiny part of you.
> I still use Visual Source Safe for my personal projects. It's never failed me in 20 years. Right click, check out, right click checkin. Perfect every time.
Wait! What?! Make sure you do not run out of disk space or do any of the other things that instantly corrupt your VSS repo. Hopefully you were being sarcastic.
I'm amazed that you find VSS reliable. Everybody I've known who used it back in the 90s (including my team) had to dump it because it so easily got corrupted. Aside from git and mercurial, you can use Perforce personal for free.
Wow, the vitriol is thick tonight. I also work in teams and like I said in the other comment, I use different flavors of version control on those projects.
> I almost totally agree. If it wasn't for security fixes, there would be no value in updating almost any software.
Software developers know this, which is why they maximize the scariness of all security fixes, and minimize your ability to separate security fixes from all other software changes. How much churn did you have to swallow to fix "algorithmic complexity attacks" you never saw, which would at worst result in a DoS? Did anyone actually attack your hash tables?
Many years ago I worked on the DPOP POP3 server and DList mailing list server, and did a little work on our company's SMTP and IMAP servers too.
My experience was that most clients sucked.
For example, one (Eudora?) would move IMAP messages by copy and delete (which may have been idiomatic, as I say it was a long time ago). But it wouldn't check the success of the copy operation, so a failure to copy would result in a delete, not a move.
Seems they still suck in 2016, for what seem like trivial reasons.
I mean, surely this sort of protocol interaction is very, very amenable to automated testing of some sort. We had a bunch of automated regression tests for our mail servers, written in C, back in the late nineties.
I'm genuinely uncertain whether to blame incompetence or another attempt at the strategies spelled out in the Halloween Documents: http://www.catb.org/esr/halloween/faq.html
I wasted a couple hours trying to help one of my clients who had this problem last week. What a pain it was. He had 5 devices connected to his account, 2 on pop and 3 on imap. It took 20 minutes just so listen to his explanation of emails appearing and disappearing and re-appearing based on which account saw them first. We spent the next hour turning on each device one at a time until we determined that his outlook pop client was the one deleting them, even though it was configured to leave them.
I was actually more surprised someone still used POP3 for direct client access - not a bad thing really - I just thought the world had migrated to IMAP.
I think greater compartmentalization of software is long overdue.
For a program like this, there ought to be a Sacred Core that Does Very Little and has implementations of key protocols that can’t be touched without the blessing of about 5 senior engineers and the personal seal of the CEO or some such.
In other words, it should be unbelievably hard to screw with parts of the program that are crucial, while making changes that shouldn’t have anything to do with it. (Heck, for all we know, they were adding Windows 10 Tiles™ when this screw-up occurred.)
Look what Microsoft did with *.odt (OpenOffice/LibreOffice) compatibility. Now they show bogus security and compatibility warning dialogs if you open or save such a file.
Look what they do with IMAP support in Outlook 2016. Pre-loading just the "Subject" (a feature supported since 1990s) got dropped, instead of the whole email incl attachments is downloaded.
And now we learn POP3 got crippled too. The common Microsoft tactics to lock-in end-user and enterprise customers to proprietary ever changing protocols only fully supported by their most recent server software - their Exchange and Outlook.com/Office365 cloud eco-system.
Microsoft wants you to upgrade to Windows 10 on PC, adopt it on smartphones (even it has just 1.1% market share), on servers, SQL-Server 2016, Exchange 2016, Office365 (subscription based Office 2016 client, cloud based SharePoint 2016 = OneDrive for Business on Azure), Skype for Business (=rebranded Lync), etc. Oh, and they ask you to install their telemetry services for Office on your clients, to get a "full picture". The telemetry and other phone home stuff cannot be turned off, except in the expensive LTSB license version. And several IPs and URLs are white lists in the kernel mode network layer, in all versions.
It's up to you, to help Microsoft to create another monopoly. Or you out smart them and say no.
"Look what Microsoft did with .odt (OpenOffice/LibreOffice) compatibility. Now they show bogus security and compatibility warning dialogs if you open or save such a file."*
To be fair, oOo and LO always warn you when you want to save a file in MS Office format. Mind you, you can turn the notification off.
Food for thought: one of my employers now deploys OpenOffice as well as MS Office after years of resisting that because of fears of user confusion. Guess what? No confusion apparent.
OpenOffice is dead, sadly refusing to accept it, so people still continuing to install it - check the commits count for the 2015 year. While LibreOffice does a very good job improving (including Microsoft formats support).
I tend to agree and I use LibreOffice myself. Employer is a Microsoft shop (win10/7 clients, SharePoint, outlook/exchange, active directory &c). I imagine someone just went to the oOo Web site and got the current download.
It's so sad, because LTSB is quite a good system. My home system was eligible for Windows 10 upgrade, which I did - then loaded LTSB as soon as I found out about it and replaced it, so my conscience is somehow clear about running it.
It reminds me of good old Windows 7 days and updates aren't breaking the system weekly.
You still have to disable various telemetry crap though.
Well, it has nothing to do with the protocol. Like you said, POP3 is dead simple. This is within logic that sits above the protocol.
That said, yeah, this is something they definitely should be testing. The fact that the article updated yesterday with a new knowledge base article and the status of the problem says they are looking into it, means they've been notified but not yet fixed it. If they haven't stopped that update from rolling out, someone at MS may come back to very unhappy managers on Monday, and have to explain why they thought leaving a data destroying bug live over the weekend was the right call...
Somewhat related: LiveMail doesn't work reliably with Yahoo Mail.
I was trying to fix email for a friend recently. Her UK ISP's email service is a rebranded Yahoo Mail account.
It works fine on her iPad, but LiveMail in Win7 (she doesn't want Win10) regularly chokes on attachments.
Turns out LiveMail can't cope if there's spam in a Yahoo trash account. And this bug has been around for years now.
Not sure if incompetence, corporate dementia, or some nefarious plot to get users to switch to <whatever Microsoft's mail service is called this year.>
I sometimes speculate if it's coming from difference in focus of attention.
I don't use Outlook myself any more, I know some who do. I know a lot of odd behavior in Outlook seems to be caused by where it involves connection with IMAP/POP3 protocol connection. (And I wouldn't think Microsoft employees are dogfooding them, either.)
Empirically, it seems to be less of this problem occurring on the connection to the Exchange server, back when I was forced to use before. (Though at that time didn't have had so much of the connection, but had plenty of issues some involving broken database and such, I wouldn't touch with a ten foot pole myself...)
This is typical of software development in recent years. The emphasis on newer platforms and technology (in the case iMap) results in new user interface limitations, as well as bugs, for existing users of older technologies (i.e. desktop, keyboard access, and Pop3).
I've found eM Client to be a good alternative to Outlook and WLM.
The main problem with POP is that unless you do something clever, determining the changes in the mailbox from time t0 to time t1 is both conceptually difficult and computationally expensive. This is because generic POP has no concept of a message UID, so there's no principled way to diff between states. In a very real sense, this means that POP is somewhat broken for the most important use case for the client: syncing to server changes.
The UIDL extension adds a "UID" but it's just an MD5 hash of the contents -- meaning that multiple copies of a message appear to be the same message. And you can ask for just the headers -- which means you can get the message-id header -- but this is still very expensive to do repeatedly (say, every 5 minutes) on a 100,000 message POP store. And you can't ask for just the Message-Id header, which would fix the problem.
Even if you have valid UIDs -- which you won't -- you still have to run a diff algorithm. Typical dynamic programming algorithms are O(N^2), which obviously sucks big time for a 100,000 message POP store.
For Inky (http://inky.com) we use a clever linear-time diff algorithm based in part on [Meyers 86]: An O(ND) Difference Algorithm and Its Variations. [Burns & Long 97] A Linear Time, Constant Space Differencing Algorithm is also a good treatment. But I know Outlook and Thunderbird both use non-linear-time algorithms to diff, so "leave messages on server" gets increasingly (non-linearly) expensive as the mailbox size grows on the server.
A few other points on comments made in this thread:
- POP is still widely used. In the US, for example, Comcast has finally migrated to IMAP, but Verizon is still POP only.
- POP is, from the server standpoint, a very simple protocol, and it is highly amenable to automated testing, as others here have pointed out. For our own testing we generate both patterned and random mailbox modification sequences, then have the test client cooperate with the test server to ensure that the client has (independently) correctly determined what's happened to the (test) mailbox. This is a perfect example of a situation where investing significant effort into automated testing pays off -- and where a TDD approach to development would also work well.