Ouch. Talk about not thinking it through. Relating back to a common topic here, one of the things that distinguishes between senior engineers and not senior engineers in my experience is the ability to visualize both secondary and tertiary consequences of a particular change to the system. They stop and say "Ok, we change this, that means ..." and then walk forward past the change in their mind exploring the ramifications. In order to do that they have to have a broad understanding of the system involved from both a technical and operational perspective. They come back with "Oh if we do that then we'll get this effect and that would be bad, how can we prevent that from happening?" and if the answer was "We can't." then they spike the feature.
One of the most frustrating things for me as an engineer at Sun early on was bouncing some whiz bang idea off Bill Joy only to get one of his one line zinger 'but that breaks X' type replies. Always annoying but often right as he would take what was proposed, extrapolate two, three, or a half dozen steps and then point out the now 'obvious' flaw.
Facebook certainly has a reputation in engineering circles as following the "move fast and break things" model or touting "eventual consistency" or similar. That works for certain areas, as others have pointed out, but once you start manipulating customer data that they have set up, it crosses a line.
I'm a strong believer in confining Windows to a fixed set of resources -- i.e., a VM with fixed memory and disk space. This incident, though I don't seem to be directly impacted (checked contacts, account settings, etc), Facebook has crossed that line into Windows equivalency for me. It needs to be confined where possible.
Mobile app deleted on all my iOS devices and FB account disassociated on my WP7 device (most used for testing).
The problem is not limited to engineering. I have often had to push back with product guys responding to feature requests from users to address edge cases.
The trouble with doing so in social networks is that the potential for the law of untended consequences to strike is much greater in it as it is nearly impossible to test for every permutation and combination available.
If, as described elsewhere, the overwriting is due to using a specific and common API, the only unintended consequence is user reaction. No need to test for every possibility there.
Please don't. If they really hurry up to fix things they could just create another breakage ;) Let them time to really think through it and fix the damage, instead of trying to do a "quick fix in production".
Sadly Solaris, aka the 'Lulu' project, aka the merging of SunOS and System V, was in fact the brain child of Eric Schmidt and Bill Joy. They stood on the podium and announced it together in October 1989. Folks would have noticed if the stock market hadn't gone into free fall :-)
> Today Facebook also admitted that its API for contact sync on phone and device apps was set -- on which devices and systems we're not told -- to take an individual's most recently added e-mail address and overwrite their correct contact e-mail in everyone's address books with the new e-mail.
and
>> Since Facebook changed the email addresses on contacts, I've actually lost every single one of my email addresses including those for work.
From this article it sounds like actual data loss occurred on some address books, where non-facebook.com email addresses were permanently deleted from contacts. For those people who don't have a backup of their address books, it might now be impossible to recover those lost email addresses.
Isn't this more worrisome than any issues around how Facebook handles incoming messages?
It is. Everything else seemed to me to be whining about how someone who saw you on Facebook will maybe not be able to contact you because your default mail is a Facebook one (but why would anyone contact you by mail if Facebook messages are better? They both a) show notifications on FB and b) send notification e-mails to your real address).
But this, this is real, and worrisome, and it has done real damage, and it will be damn hard if not impossible to reverse.
Facebook messages are better when the person who wants to contact you found you on Facebook. They are better because they show in two places: your normal, original (not the Facebook one) mailbox AND in Facebook itself. So it is at least as good as sending an e-mail, and likely better.
If someone went looking for you on facebook, chances are they know how to simply click and message you.
If they're going the extra mile to copy and paste your e-mail address, chances are they want you to e-mail you, and not send you something through facebook.
Aside from the simple fact that not everyone has the same notification settings, it's already been shown that facebook has been losing people's e-mails. No notifications, no nothing, just gone.
Sometimes it seems to me that FB doesn't have an impact assessment process. This is a very important data change you're making to 900-ish million records on a system that is inter-woven with and syncing to... everything. I understand bugs happening, but really, the cascade effect of this email change had to have been foreseen by someone... no? Bueller? Bueller?
I really don't mean to trash FB as a lot of good people work there. However... when I read about something like this, and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level. (I know, a mis-balanced and irrelevant comparison, but it's just for a 'gut feeling' sense)
I know what Hanlon's razor is, and I can guarantee you FB Engineering knows what a rollback is and how to know whether you need one or not. I'm not sure where you're getting the idea that they're that stupid.
Would applying such a 'rollback' involve replacing facebook.com email addresses with the original external domain addresses in the contacts list on the devices affected by the write-over issue?
If so: Facebook have taken private data and placed it in a central location
Yes, a (hypothetical) "complete rollback" would involve replacing removed data. Sure, data privacy would be a concern, but it's beside the point that rolling this back would necessitate un-overwriting.
Only Violet Blue has said that they don't have a complete rollback as best I can tell from the CNET article.
I would guess the only thing they can't undo would be the time that has passed. If any data Facebook ever had on any of their users (such as their primary email address [edit: at any point in time]) was ever lost, they wouldn't have as much to sell to their customers!
She didn't say they didn't have a rollback plan, she was saying there's no way to undo the damage they've already done. She's not privy to their internal plans, it was a metaphor.
Rolling back the change won't get people's time or email back, or undo the damage to facebook's reputation.
That would make sense giving her the benefit of the doubt, thanks. The transition to metaphor was pretty blurry in the article title.
As I pointed out elsewhere, they can just forward the email to the intended address as an unwelcome MTA, so I disagree with your statement that they can't get people's email back. I agree on the time and reputation points though. However, I don't think any rollback has ever gotten those back, so 'rollback impossible' because it can't do what any rollback has ever done seems disingenuous.
As silly as it sounds, this almost sounds like a job for Git. In that case, you could simply revert a single commit, and have the user resolve the conflicts. For that to work well though, you'd need to keep separate "repositories" for phone/Google/Facebook/whatever addresses and merge them locally, which could get very ugly for end users.
Thanks for sharing your perspective. Just to make sure we're on the same page, you are arguing that FB engineering was 100% aware of the consequences of automatically switching primary email address to @facebook.com for users who had setup that functionality, including overwriting email addresses in phone contact lists and capturing email unintentionally sent via those modified contact lists?
This is certainly within the realm of possibility and indicates that you hold the team's technical ability in high regard. If it is indeed true, I would wonder what percentage of the folks involved in the implementation actually thought about it and how high/wide the issue was escalated. I have certainly been in situations where after-the-fact I found out that something I had been a part of shipped with known bugs/features I didn't know about that I would have raised a lot more concern over.
My line of thought is that they knew it would change phonebooks and that they weren't putting whatever resources necessary into creating a process for rollback. Given the nature of Facebook and it's technology, I'm not even sure there is anything they could have done, but it's de rigeur to know if this is the case. It's not even about high regard, but by minimum standards. If you know you can't rollback you go full steam ahead.
If you've never worked someplace where they have dedicated staff for source control, that fact alone might give you an idea at the level of expertise involved here. It's nobody's first time at the rodeo, and setting the sync API on the change (as described below) wasn't just slipped in by an intern.
You finding out after-the-fact and still having an opinion about bugs shipping in code you contributed to only reinforces my point: someone made the decision to go ahead anyway.
> and then I see a company like Google figuring out a time smear function to seamlessly account for a leap second in their servers to avoid cascading system changes... it's hard to feel the two companies are functioning on the same level.
Sorta makes you wish Google would stop chasing Facebook and just do their thing.
c) Associate your real email address with your FB account
Facebook gets an email address I don't use anywhere else. No one has me listed in their contacts under the only email address facebook knows for me (well, I presume. It's not hard to guess the address I use for facebook if you know my real address, but there's no good reason anyone but facebook would use it).
I've done that on general principle for more than a decade any time I have to give my email address to anyone other than an individual. This instance is the first time in a while where it's actually felt worthwhile. Originally I did it more for spam tracking. GMail's excellent spam filter fixed that problem for me years ago. But I never stopped doing it.
And I'll also add:
d) Never associate your real name with your FB account.
I use my first name plus a family name that matches my middle initial. It's generally recognizable and accepted by friends.
When I first set up my account this way (items A through D) it did seem a little "tinfoilly", and this last measure still does to some extent. But since Facebook doesn't actually let you delete your account, I'd rather err on the side of prudence. It will be nice to at least approximate a hole in their databases if their policies ever get too onerous.
If you use the FB app, it will associate your mobile phone number with your account (and make it publicly visible unless you change this default). Remove it, and then a few weeks/months later it'll magically appear.
Can anyone point to a Facebook representative saying that rollback is impossible? (I don't care at all what the CNET author says.) It seems like they should just put everything back to the email address they had before it was auto-updated to @ facebook.com. Then their apps would put them back to what they were on people's phones.
Lastly, if they were truly going to fix the problem, they would pump through all the emails they got a chance to spy on, this time to the correct address, maybe with an attachment or sending the email as an attachment with their official apology.
Can't rollback the delay and that impacts a lot, but email is theoretically unreliable anyway. Hurry up guys and gals... show me someone at Facebook actually cares about the product (their users) they are selling to their customers (their advertisers).
A couple issues pop to mind. My understanding was that this update took a week(+?) to roll out and may still be in progress. It could take them awhile to even start to roll back. Then if you do roll back what do you do about all the people that have changed their settings in the meantime? They have updated their email addresses, perhaps with different ones, so do those get over-written with the old data in all cases? I'm sure it could be figured out but how timely it would be is questionable... not to mention that FB may still be trying to salvage the bump in use for @Facebook.com they were aiming for in the beginning.
I agree it's not a simple fix. Since they have everything stored, they could act as though the address had never been changed. (Primary email should be most recent non-auto-change. Email should be sent to value of primary email address [without auto-change] as configured when email arrived.)
This mistake/bug/zuck-style enhancement has had a colossal negative impact on many people. If Facebook tries to salvage anything from it then it's even more clearly time to walk away. I am not cheerleading for Facebook at all on this, I just want people's contacts back to normal and their emails delivered to where they want them. I think we agree this is possible, in fairly direct contradiction to the title of this article.
There's no reason to assume that the old addresses in people's address books were the same ones Facebook had as primaries. If my co-worker was in my phone as tom@patientslikeme.com, but signed up for Facebook as tom@gmail.com, and Facebook tries to roll back? Data loss still, and possibly even a privacy leak.
I was basing this on the fact that the facebook app/sync would have pushed tom@gmail.com into the contact info in the same way that it is now in trouble for overwriting with @facebook.com.
Thanks for bringing this up as it would not work if they only push changes, and no change had been made facebook-side prior to the 'bug'. In that case, they should only rollback those that had been pushed out and the majority of email addresses would be lost. However, since users can go into the settings and restore their primary email address, it seems like it would be best for Facebook to encourage them to do so in the same way they've nagged me to switch to timeline forever.
It's a metaphor. She's saying they can't undo the damage they've already done, not that they can't literally change the code (or email addresses) back.
"Move fast and break things" should still require that you think through what you're doing. The problem seems to be that the complexity of Facebook's system is beyond the understanding of anyone who reviews these changes.
I'm pretty sure that FB knew exactly what they were doing. They did think things through and decided the the short term PR loss was worth the benefits of gaining access to the email of 90% of their user base.
Let's say 1% of all their users is fed up because of this and 0.1% actually closes their account. This leaves Zuckerberg with 98.9% control over e-mail traffic from 900+ million users. Well played, Zuck, well played.
Couldn't this be construed as an abuse of a monopoly(antitrust)? They have a monopoly on social networking(there are alternatives, but they are very very small in comparison) and they are leveraging that to expand massively in another sphere, namely as email providers.
The problem here is not one of competence or failure to think through implications. Facebook know exactly what they are doing, and you can be sure the potential implications of these changes were debated at length within the company at a high level. They were taken precisely because of their implications for non-technical users, not in spite of them - FB has replaced users contact details because they want Facebook to be the only way most people can contact others online.
Facebook have always wanted to replace the internet entirely with a system they control. Hence the walled garden approach with incessant prompts to signup to view content, the use of address book import to spread virally, the use of beacon to track and announce users activities, the use of like button js to track activity, the launch of Facebook apps and single sign-on to attempt to corral web developers within their ecosystem and make them dependent, the launch of a competitor to email/IM etc etc.
This latest move with email is simply the latest in a long line of moves consistently attempting to steer users into spending all of their online time within the confines of Facebook, it is entirely consistent with their past behaviour and we should expect similar moves in future.
So the arguments about how they could have done this differently, or whether Facebook has made some mistakes here are largely irrelevant - they will always push to own your data, email, photos, and digital life, because it's in their DNA, this is not a mistake, it's a pattern of behaviour - they can and will control your digital life if you choose to hand it over to them. Consider the quote from FB on the way their FB email works:
Regarding the "email loss" this may actually just be confusion around the Messages Inbox: By default, messages from friends or friends of friends go into your Inbox. Everything else goes to your Other folder.
All messages which do not come from FB contacts go into a folder which is effectively a junk folder - they want you to use their email for everything (hence the recent changes), they want your address book to contain nothing but FB contacts, they want you to forget email even existed, and they want you to encourage your friends to join FB so that you actually receive their messages. That won't work for all their users of course, but their hope is that they keep enough of the user base on board that they can dominate the internet and replace email with FB and the web with a network of sites where they control login and user info and all roads lead to FB.
So replacing email with FB mail wherever they can, hiding other contacts and trashing user address books is not a mistake, it's part of a broader strategy, one which is not in the best interests of their users, but which they hope most will go along with anyway.
I deleted the Facebook app from my phone a long time ago and I only use it from an incognito browser window. So that's how much I trust their reach. But I find it a little incredulous that Facebook, in their quest to control all their users's mail would deliberately overwrite email addresses in users' contact lists. It seems like the kind of dumb plan for world domination that only a caricature of an evil villain would execute.
I used to have an Android device, and let the thing sync my Facebook contacts into my phone. What I wound up with was a mix of people I'm actually friends with, and also a big collection on "Facebook friends" ... those people I'll never ever call, but knew from back-in-the-day and wanted to connect with on Facebook. I don't need them in my phone today because the extend of my correspondence with them will be what I consider "two-way read-only". They can look at my FB stuff. I can look at their FB stuff, but that's pretty much the extent of our relationship.
I'm on an iPhone now, and know that iOS6 will give users the same option ... to link your device w/ Facebook ... sucking in all your "FB friends". I won't be enabling that feature, and while the software apparently indicates you can disable this part of the connectivity, I'm reluctant to even link it for fear that it'll immediately hose my contacts in a way that requires me to manually clean out that kid I went to middle-school with many years ago.
There were so many obvious problems with associating a @facebook with everyone's account. As soon as I heard the news, I thought.. hmm.. someone could easily scrape 1 million facebook usernames, and be able to send 1 million emails to all of them using their facebook email address.
And make it all uber-personal because they know their full name AND all their friend's names AND what they liked AND their hometown AND the college they went to, etc, etc,. That's a spammer's wet dream.
The @facebook was always there and could always have been used to reach a person's FB messages via email. Also, most of that other information would probably be marked as only available to friends or friends of friends. So you wouldn't be able to scrape it.
Don't get me wrong, this is a horrible move by facebook, but spam isn't the reason why.
Then I changed my email address back to what I want it to be, and I have ceased caring. I will continue using facebook to coordinate bar meetups with my friends.
I think you kind of hit the nail on the head with your last sentence. You will continue to use the service for fairly meaningless tasks. It seems Facebook wants its users to use their service for more than that. So while this debacle may not make many people delete their accounts, Facebook may find its apparent goal more difficult to achieve going forward.
I've always had a very healthy mistrust of facebook, based on their past actions, and I never let facebook sync up with anything else I use (and I don't use services that only authenticate via facebook). They simply can't be trusted with anything important; not because they are incompetent, but because they have no moral compass that prevents them from screwing users over.
This is just one more reason for me to continue to silo facebook off into its own world, where it knows nothing about the rest of my life, particularly the really important stuff.
No, I won't leave Facebook entirely over this, but I did just uninstall their application on my Android device. I simply don't trust them to have access to my contact list.
Frankly, I didn't trust them much before: I'm sure they were mining it, which is not something I would explicitly consent to. However, now I no longer trust them to not break it, so I've removed their access to it entirely.
I see they actually created a way to remove the FB app from an Android device. Back when they hoovered up phone contacts into their database, I tried to break the connection, but all I could do at the time was delete my entire facebook account. Been pretty happy not wasting my time there, as I see it devolving into pseudonymous games, meme one-upmanship, and picture albums kids will mostly later regret.
If you installed Facebook yourself, you can always remove it. You can't remove it if your carrier or manufacturer put it there (at least not withput rooting). AFAIK Android 4.0 allows you to deactivate these apps, effectively removing them witgout deleting the files.
That assumes you CAN uninstall it. Many devices come with Facebook preinstalled and not removable (because Facebook worked a deal with the device manufacturer or the cell company).
I don't have the entire Facebook userbase added as a friend, but I haven't seen a single post on my newsfeed about this problem. Yet every second day I see posts about how much someone hates Timeline.
It will be are for people to completely leave Facebook. What Facebook needs to worry about though is that people will stop using the service as much. People may be less likely to integrate Facebook apps on their phones.
All of this could lead people to use FB less, which is is exactly the opposite of what they were intending.
I've been using a great work around to the numerous Facebook issues, including the lack of meaningful interaction with Friends and Family; I use a combination of email, telephone and physical interaction to maintain and form social connections.
So this whole debacle has taught me that I have an "Other" folder, which I never saw before, and it turns out I have 20+ messages from people from the past few months which I didn't know about!
I'm trying to find a way to get Facebook to send me an e-mail notification when I receive a message that goes into "Other", the same way it sends me an e-mail notification when I receive a message that goes into "Inbox".
But I can't find any option for this anywhere. Does anyone know? Otherwise, how am I ever supposed to know I have a message from someone on Facebook who I'm not already friends with?!
So how can you make sure you find your lost laptop, appear on Israeli TV, and respond immediately to job offers? Unfortunately, the Facebook rep informed me, you can’t change your settings to get email notifications for your Other messages, the way you can for your main messages. Your best bet: Make checking the Other tab part of your daily Facebook routine.
So since I never log into Facebook, but use it as a place to receive messages, which I find out about via e-mail... I am doomed to never find out about possibly important messages from people who are not already my Facebook friends.
This is possibly the worst design decision I've ever come across on the Internet. Facebook has created a black hole for messages. Even when you log into Facebook, it doesn't show a number next to "Other", so you can't even find out if you have messages there without navigating to the "Other" page.
I can't wrap my mind around how insanely stupid this is. WTF, Facebook?
"Facebook is indeed intercepting messages -- the e-mail is clearly passing through Facebook's servers, but this seems to be where it stays, as neither sender nor receiver are getting a copy."
Is this really appropriate use of the word "intercepting"? Typically intercepting carries a negative meaning -- like "the FBI is intercepting my postal mail and opening it before it reaches me". However, with their use of the word, every email I send is "intercepted" by multiple servers before it reaches its destination....
Facebook is intercepting messages by replacing the destination address (which email clients often do not readily display [opting instead for full names, even photos, etc.]), and now claims to have done so inadvertently. Multiple servers relay messages before they reach their intended destination, yes, but that is by design.
To continue stretching your analogy, I could claim that the post office x-ray/bomb scanners are 'intercepting' mail sent to me when in fact it is an understood part of the process to anyone who cares enough to research how things work. Someone moving my mailbox [edit: filling out a 'change of address'] and photocopying everything sent there is pretty clearly a reasonable use of the word 'intercepting'.
I can't stop my @facebook.com address from appearing on my timeline. The option shows up but clicking on it does nothing. Similarly, I can't make my @gmail.com address show up on my timeline. The mouse listener is also missing.
"Since Facebook changed the email addresses on contacts, I've actually lost every single one of my email addresses including those for work." -Complaint from iOS 6 user
My suggestion, don't use the very first beta release of a operating system on anything important. Digital darwinism at work there.
Oh that is Facebook screwing up. I'm not trying to pardon that. I just hate people who complain about stuff when they take a well defined risk and it backfires on them.
Given FB is already having a hard time generating revenue from mobile clients, doing things that make mobile users afraid to use their service seems like a really bad idea.
Well, given that they can't monetize mobile usage and they can monetize desktop usage (if somewhat poorly), technically driving users back to desktop usage from mobile would be good for them.
Now what happens if other sites and apps do the same? Will our address books change every day and contacts fill up with dozens of unused email addresses? Many of Facebook's growth techniques have been imitated by other startups so going into the territory of manipulating data that has always been off-limits is a big step. If another social site uses those techniques against to manipulate data on Facebook profiles, will Facebook change it's mind?
i uninstalled FB app about 8 months ago, never used sync on it anyways since i use FB more like a glorified twitter. been using mobile site since then (which is quite good), posting photos by sending an email to a special status address. so much more privacy. if they ever get rid of the mobile site to force me to install the app, byebye facebook on mobile.
I like FB so I won't be doing one of those "hey everyone, look at me I'm quitting FB" acts. But I highly recommend that you don't put FB on your phone...it's amazing how even a five second glance to check for new notifications will ruin your train of thought. And also, not having it mess up your phone account is nice
There is the product life-cycle which has the phases: introduction, growth, maturity, and decline.
The point is that facebook needs to understand that "Move fast, and break things" is an appropriate approach in the earlier phases, but it is quite incompatible with later phases, when people start to rely on a service.
That's actually a really good analogy. And the GP is right. There should definitely be some laws to prohibit these kinds of things that Facebook is attempting (doing), now that electronic mail/communication is becoming standard.
One of the most frustrating things for me as an engineer at Sun early on was bouncing some whiz bang idea off Bill Joy only to get one of his one line zinger 'but that breaks X' type replies. Always annoying but often right as he would take what was proposed, extrapolate two, three, or a half dozen steps and then point out the now 'obvious' flaw.