I keep hearing about this coming up, yet no one ask talked about why the data is irretrievable.
What it implies is that the internal tool(s) that the support staff uses was designed to do deletion instead of status changes. I imagine they have a web interface that fires off some backend script which a) runs sql deletes on an account ID and all tables and b) fires off scripts that physically deletes all photos across all servers.
Taking those implications in hand, it points to an incredibly poorly designed system for account management. Some years ago, there was an internal planning meeting where a decision was made to do this. Some people objected and were overruled. The ramifications of that project manager are still sticking it to people to this day.
I have a hard time believing that Flickr would have purposely engineered their systems in the manner I've described. Not impossible, but unlikely.
So I think its more likely that the data is still there, but that customer service has no method to resurrect it. Its probably a mix of internal policies and the lack of devoting a couple of engineers to fix the problem that is causing all of this.
> "some backend script which a) runs sql deletes on an account ID and all tables and b) fires off scripts that physically deletes all photos across all servers."
Neither should happen. You never actually delete rows from your tables. You mark them as deleted, sure, but the data needs to be able to come back. Space is cheap, there are simply very few cases where outright deletion from a DB table is warranted.
I have a hard time believing Flickr would make a mistake like this. I agree with you - odds are if they actually let an engineer loose on this problem it'll be licked with a DB backfill in 20 minutes.
I bet that in cases of copyright infringement a logical delete isn't legally sufficient. Sure, you can leave the database pointers in place... but you're going to need to actually remove the infringing content. In that case, at least, a true delete tool would be needed. Like any Internet discussion about copyrights or other people's backend scripts, this is all purely speculative.
I don't buy it. Simply making infringing data unavailable has to be legally sufficient, except in cases involving removing secret or confidential data (which would likely require a court case) there's no harm in having the data merely inaccessible. Moreover, it simply must be the case that this is how it works in practice.
Consider how deep this rabbit hole goes. If expunging any copies of copyrighted data were strictly necessary, then:
What of data existing on backup media? Should those be re-hydrated, the target data expunged, and then recreated in abridged form?
What of data existing in various content caches (such as memcached)? Should all of those systems be flushed of any possible contaminant?
What of the actual data on physical media? Deleting a file on any modern form of media does not expunge the data, it merely unlinks the location of the data on disk from the file system directory. It might be necessary, depending on filesystem and drive type, to scan all of the unused sectors on an entire disk to find out if the target data or any part of it existed.
As you see, quite quickly you get into absurdities. Nobody goes to that much trouble to delete merely copyright infringing data.
I can't speak for Flickr, but I can tell you that one of their larger competitors simply does a logical delete. That seems to take care of any legal obligation.
I should note that this not true in the case of child porn. Once the feds give them the signal that is completely purged.
> You never actually delete rows from your tables. You mark them as deleted, sure, but the data needs to be able to come back. Space is cheap, there are simply very few cases where outright deletion from a DB table is warranted.
This comes with a cost, and it's not a small one. Every single query that ever touches this table, in every single piece of code owned by the company, written or maintained by every single developer (consultant or transient or offshore, none of which ever got training sessions on this requirement) needs to be enhanced to respect "where isDeleted <> true".
It's well understandable if the expected cost to recover after the fact from an occasional issue is smaller than the cost of implementing and maintaining that in the first place.
A sane approach would be to do all your table access via views, and the view definitions have the 'where is_deleted = 0' or whatever. Then your database handles it.
Updates to the view work as well, in pretty much every database system, including MySQL even.
That just shifts the problem. Every piece of code and every developer now needs to know to hit that view rather than directly hit the table. Except the code that actually deals with deleting and undeleting which has to hit the table directly (or another view or sp.) And you still have the original problem in writing and managing the views themselves - every developer writing or maintaining one of those views needs to know to respect "where is_deleted = 0".
Using views adds a layer of abstraction. That's often a good idea and a step in the right direction, but does come at the cost of increasing the overall complexity of the system. It's an incremental step to the positive, but no revolutionary solution.
"..but that customer service has no method to resurrect it".
Having seen this kind of effect from a brain drain in the past, I think you're exactly right. Software needs love to survive and this is what it looks like when software and a company becomes unloved.
Indeed. I'd have sympathy for Flickr if this story didn't come up almost monthly in the last few years. Between this and the account deletions of those who spoke loudly over Flickr's other account deletions, more and more Flickr is becoming a hostile system.
The lack of a retrieval system could be from a lack of Yahoo wishing to invest in upgrades to the platform in general. The last few years have seen few changes for users beyond geotagging and tagging persons, and competitors (free and otherwise) have stepped far ahead in terms of personalisation and galleries.
I'm very curious to see if Flickr turns into another 'Delicious' situation in a couple years.
I think that danger is very real. I used to be a Flickr Pro member but I've been taking fewer pictures the last few years (for several reasons) and can't justify the Pro account anymore. However, the side affect of that is that now some of my older pictures are less accessible and my old photo sets aren't browseable. At least to me this seems like a bad business plan for Flickr, especially amidst an abundance of rabid competition.
Flickr's main competition isn't with services that have feature parity with them, it's more with the growing list of free image sharing sites. Imgur, twitpic, facebook, picassa web, even dropbox. Flickr's value add is rapidly becoming smaller and smaller, which could very easily undermine their business model.
Exactly. I used to have a flickr account but since dropbox/facebook came along if I want to share anything it goes in dropbox (public) or facebook (public, but it's easy for my family to see the pics).
It's an engineering decision - Flikr has to host a large amount of data for very little cost, since most users aren't paying. Space isn't cheap by the Petabyte!
The design for a system where you had multiple redundant copies of all the data and could roll back any changes to the data set is different - and would cost a lot more.
Here is what I don't understand. They have multiple redundant copies in case of unlikely scenario of disaster, but they have no copies or protection in case of quite likely scenario of operator error. This seems backward to me.
EDIT: I don't believe that Yahoo! Flickr does not have a way to recover these things. Or I'm completely wrong.
I think it's likely that Flickr has a way to recover the data but the front line support personnel don't, and they don't want to have some dev spend a few hours to fix a single customer's problem, even if it was Flickr's mistake.
Reverting a full db should be straightforward (stop the db server and revert a dump).
For a single account it's more of a pita.
Retrieving all the datas associated with one account, put it in a format readable from the production environment and recommit all the changes while dealing with the potential errors and inconstencies seems quite a huge deal.
Messing unrelated data in the process would also be a nightmare.
Of course, not deleting anything from the start wouod be the way.
It's trivial in Oracle, just do a Flashback Query. You would just need to support that in whatever front end interface the customer support people use.
Are you saying that a freemium service should never delete data—or that a customer service rep shouldn't be able to do it?
I can see why Flickr might engineer things this way. Most of their users don't pay (granted, this guy was a pro user). Deleting images when an account is deleted saves money.
I think this is a huge problem with all cloud services.
As we start to rely on cloud providers to look after our data, we need to either become more educated (and proficient) by creating regular backups ourselves (which kind of makes the idea of managed cloud services defunct imo), or be in a position where we can individually sue for damages incurred by negligence on the part of the service company in question.
So many of these companies have liability clauses which negate all responsibility in situations where negligence leads to loss.
I think the current situation is crazy, considering the amount of responsibility and trust that's involved in using web-services which store and manage irreplaceable data.
A single service, even a cloud, is still a single point of failure. Diversify your backup strategies.
Indemnification is likely going to push prices up further, making cloud non-competitive compared to traditional storage.
Still, I feel a lot of old technology is merely being relabeled. Today everything that stores data remotely is called a 'cloud'. We may yet see small NAS units relabeled as 'private micro-clouds".
I agree - I've recently learned of the importance of diversification first hand.
However, I'd argue the ability to take legal action, would actually encourage service providers to diversify their own backup strategies and procedures. If a mistake becomes too costly to consider - I think less mistakes would be likely to occur.
If a mistake becomes too costly to consider - I think less mistakes would be likely to occur.
This conclusion assumes that the people running the business are capable of resolving such issues. If they are not, then all it does is put people out of business. Not everyone grows and adapts when faced with such pressures. Some of them just go extinct.
"This conclusion assumes that the people running the business are capable of resolving such issues. If they are not, then all it does is put people out of business."
If a company can't be relied upon to look after my data - I want them to go out of business.
Unfortunately, recent years have given me the impression that if you set the bar too high, there will be so many failures that you won't have any services to turn to. That's my concern. I wish that weren't my concern and that I could heartily and unhesitatingly agree with you.
The problem here is that the photographer lost data that he had no way of backing up. His photos are fine, his online collections, uris, and social networking are all gone and only existed on flickr's servers.
I don't get how in todays day and age the person lost their own copies of their photos. All the photographs I have taken with my current camera are still on the memory stick, on my hard drive and in the cloud. Why? Because data storage is massive compared to your average cameras photo quality.
My camera holds about 2,000 pictures on the highest compression setting on an 8GB card. It's not like that amount of data is precious. I have hundreds of movies and thousands of TV episodes stored, why is it that this person couldn't keep their photos?
I hear a lot of stories like this, and being a writer I can only think "are they stupid?" Until recently everything I'd ever written, if stored in RTF and rar'd would still fit on a floppy disk. I think I've expanded to two.
I now have a flash drive for storage. However my basic method is to simply download and rar all my files from Docs, email it to myself through gmail and load it onto the USB drive. It leaves me backed up in 3 places and the original copy.
TFA: "Mirco Wilhelm has the original files saved elsewhere, but the photos from his extensive Flickr collection had been linked to from all over the web, including the official Flickr blog. Those links will now point to deadspace. Additionally, the followers he had accumulated, tags, photo captions and copyright information have been wiped out and may not be restored"
Beyond that, I have well over 2000 photos in my photo library, and I'm by no means a serious photographer. A professional might take several hundred photos in a single day. (Not all will be useable or worth publishing, but the idea that you might just never delete any photos off your memory card just doesn't work for a lot of people.)
You're right in your point about backups. These cloud services present quite a problem in the sense that they're structured to accommodate massive amounts of data. Often times this data is generated within the service itself with no means provided to extract it. So even if the user wanted to be responsible for backing up their data it's often times impossible. Moreover, even if the user wanted to AND had the means to do it, it could be a massive amount of data to sync up -- more than your average user is going to take the time to do.
Consequential damages aren't covered in these services' terms of use because nobody can afford to pay for them. Should Amazon be liable for Netflix's profits on EC2? If so, Netflix isn't paying Amazon nearly enough money.
What a coincidence. My Flickr pro account expires on Feb. 15 and today I've decided to move away from Flickr and not renew my account.
What I've done in the short term is to use FlickrTouchr (https://github.com/tominsam/flickrtouchr) to retrieve my ~7 GB of photos from Flickr and upload them to Dropbox. The Dropbox gallery feature works quite well and has the advantage that you can easily backup/modify the underlying files.
In the long term I'd like to move the pictures to my own server (allowing me to use my own domain, etc.) and hack together a simple frontend similar to Dropbox' gallery.
If you decide to move away from a Flickr pro account (and if you also have non-public photos on your account) don't forget that after your account is switched back to a normal account you can only access the 200 most recent photos. So if you want to delete your photos from Flickr it's best to do so before the pro account expires.
This is fine if you're just using Flickr as photo storage and publishing, but (especially for pro photographers) a big part of the reason to be on Flickr is the social networking features. If you've got people linking to your photos and photo stream, you can't easily duplicate that outside the Flickr ecosystem with your dropbox hosted photos.
>I'd like to move the pictures to my own server (allowing me to use my own domain, etc.) and hack together a simple frontend similar to Dropbox' gallery.
I think Opera's personal browser based server ("Unite") is going to have it's day (or something similar will) - http://www.opera.com/press/releases/2009/06/16/; has a "photo server" built in.
The article pointed out that a lot of services "deactivate" an account rather than delete it. I would suspect this common sense methodology is as obvious as hashing a password, but apparently, and unfortunately for the artist, it is not.
That's the biggest point I took from this. Removing the wrong account can always happen, but actually wiping data instantly without an invisible "grace period", or retrieval from backup, or anything else to get it back, that's just poor. Especially if it's a pro account.
It sounds to me like they didn't invest much time in the internal admin section. They probably delete a lot of accounts every day, and they've never had an issue with the wrong account being deleted? Or an after-the-fact clearing up of a misunderstanding? They don't start a dialog, at the very least only with pro-users, to get the other side of the story?
I hope something else went wrong here as well, and it's not the way they work normally.
> I would suspect this common sense methodology is as obvious as hashing a password, but apparently, and unfortunately for the artist, it is not.
A few days ago we had a big thread on Undo in software development, and a lot of people were suggesting Undo is very very difficult to implement with little benefit to most users.
Then you have articles like this where people are shocked that undos weren't in place.
The guy has his backups. His problem is not the data. His problem is the human-networking side of things: lots of places linked to his photos (including the flickr blog) and those links now point to dead space. That will reflect badly on him, I'm sure, and not flickr.
If I were Yahoo, I dont care how much it would cost, I would have someone, or someones, recreate his account as it was, even if that meant calling in a data forensics team.
Seems like a funny policy to not backup production paying customers photos & metadata. Think of all the photos, comments, followers, etc that just got nuked! Seems like they rely on RAID or having multiple copies of the data spread around for disaster recovery. Sure it protects you from a server/raid/hardware failure but when a user/admin does something dumb it can really hurt!
Doing a quick google search turns up this gem:
Staff - heather says: "When an account is deleted, photos and any metadata associated with account is queued and deleted. If not instantaneous, it's removed within minutes."
RAID is there for system interruption protection. RAID is there to keep flickr cruising along without interruption.
If they delete content from a volume covered by RAID, then all the mirrored copies go poof simultaneously.
What you are suggesting is a higher level content management system. Apparently (from the drift of the comments) that is something flickr doesn't want to get involved with.
It seems like a _very_ good policy from a privacy perspective, if I accidentally uploaded some pictures that I really didn't want online (easy to do with an upload tool pointed at a directory), then I'd feel much more assured if Flickr would tell me that they didn't keep any copies of my pictures floating around "just in case".
Of course, deleting a complete account without any means of rectifying it sounds like a very bad idea and shouldn't have been possible. At the very least there should have been red warning signs flashing when the support person deleted a pro account with 4K+ pictures in it.
The real dilemma is that as a service, you can't do it right. If you keep the data around and just mark it as deleted, the EFF will post articles about you violating the users rights for privacy and control of their data.
And if you delete the data, then $publication will write articles about you being unable to restore it.
Take the Facebook "account disable" feature as an example. In the context of this article here, it was sold as a "good" feature (so as something that Flickr should have too).
But this exact same feature was all over the press a year ago as something really bad and just showing how Facebook just doesn't want to let you go.
So. What is it now?
Personally, just flagging for deleted and living with the privacy concerns might be the better option because there are potentially more people deleting data by accident (or have it accidentally deleted) than there are people too concerned about privacy to use the service.
Also, users concerned about privacy just won't be users whereas users who just lost their data were users and now aren't, so that's a net loss compared to the users not joining.
Thing is, this is what happens to you if you complain at Flickr. They don't like having to actually do work. That's pretty obvious from the fact that after over five years of losing people's photos like this, they still fail to back up anything. The founder even lost all his stuff at that cluster-f of a social network gone wrong. What's funny about those slackers at Yahoo's Flickr is that they think they can pull off hosting a porn site, while pretending it's a family friendly place for your kids' photos. That may be fine for the average user that wacks off at work, but not so good for an advertising platform, as it turns out. You see, major corporations and small businesses alike have an issue with their ads being surreptitiously placed onto hardcore pornographic web pages without being told about that. Yahoo's Flickr can play this game of hide-the-porn while tricking the general public into trusting them. But those lies don't really fly in the advertising world on which Yahoo depends. Everyday, several people get deleted from Yahoo's Flickr as that company desperately attempts to make their porn site not really appear to be one on the surface. The copyright infringement is the same thing on a smaller scale. They count on people stealing your content, that's why they tricked you into placing it all online in an easily accessible catalog of stock images from trusting idiots. Make any kind complaint about the way they are doing anything, and you're booted out mercilessly. That's just the way it goes and Yahoo doesn't care one bit how you feel, because they obviously do whatever they want to. They have the government in their pocket and free reign to push porn into grade schools unlabeled, give your photos away for free to anyone that wants them without liability, and harbor countless sexual predators, pedophiles, and registered sex offenders, whom they cloak so they can be right next to your children without anyone being suspicious. Can't really see anything worthwhile about that website, or Yahoo in general. It's all lies from them, and everyone eats it up with a clueless smile.
I highly doubt the files are irretrievable. It's more likely it'd cost Yahoo a whole lot of time (or a large cost in getting specialist contractors) and they would rather just risk facing bad PR than coughing up the money.
Maybe they'll do the latter after some rethinking!
Although this is a terrible mistake on Flickrs part, and it shouldn't have been possible to do this by accident, am I the only one that is happy that deleting a Flickr account really means _deleting_ it?
Most other services like Facebook, GMail are explicitly designed to keep your data forever, and if you don't want that, well, you should have thought of that before using the service. I would actually be happier and feel safer if we heard more complaints from Facebook users that accidentally deleted their account and couldn't get it back. It would be a clear sign that their system was designed with privacy and "data hygiene" in mind.
This is why I never have a DELETE call anywhere in my app or admin code. I always mark as deleted (there are performance improvements in this for some databases as well)
I've worked at several places where the strategy is 'never delete, just mark as deleted'. Your database tends to grow pretty fast, but you never have to defragment your index or tablespace.
One can only hope that this is not quite over yet.
There's been quite a few well constructed and thoughtful apologies circulating HN lately, one of which was Andrew Mason's apology for Groupon's Japan blunder. Hopefully, Flickr will pick up the torch, and issue a more official, and hopefully more well thought out, apology. After all, it's been less than a day since the incident.
Of course, the deletion in itself is inexcusable. I just hope Flickr steps up.
Yahoo seems to be having trouble in general. First there are rumors that they are shutting down Delicious. Now their photo sharing service is starting to show real problems.
This whole situation points at a market opportunity for a competing service to offer users the ability to backup a real DB dump of their entire user, not just "everything you uploaded", but actually a file that, provided later to the service (or a competing service) contains everything needed to recreate that account.
4 year subscription to Flickr Pro? That's not even a year for every year of photos he has lost. I would think that if you deleted 5 years of a users photos the least you could do would be to give them a life time subscription to Pro and buy them the best damned camera available so that they can take new photos.
This is pretty awful. I have some sympathy for the support staff though, if their tooling allows them to irreversibly delete someones account (and it's history) they need better tooling. The realisation that they just deleted someones 5 year old Pro account and 4,000 photos... oh shiiiiii
"This wasn't much compensation considering Flickr Pro costs $24.95 per year, and Mr. Wilhelm has already received a year's worth of Pro through his participation in some events and competitions.*"
what else could flickr have offered to compensate?
I would have asked for something closer to a full refund and possibly a longer, if not lifetime, term of free usage.
It's obviously expensive for Flickr to offer something like this but I mean they accidentally deleted 5 years of a paying customers work. I think a refund is the bare minimum.
They have done that bare minimum. Flicker has already provided the guy with five years of hosting and linking - that doesn't just disappear because it's in the past. They provided the service that was paid for over those years.
Now they've made a mistake, and have refunded this years payment and added future access for free. They have done more than 'the bare minimum'. It might be good PR to refund previous years' fees, but they were at the time provided in good faith and competant manner; there is no 'fair exchange' reason to refund them.
even a lifetime of flickr wouldn't satisfy the authors request for compensation. a few hundred in flickr dollars is not worth thousands of lost photos.
anyone know of anything like that for downloading the data locally? backing up data in the cloud to another place in the cloud just seems stupid.
i've seen some flickr-specific backup scripts that just dump all of the images to a directory, but they don't preserve titles, dates, sets, collections, tags, etc.
backing up data in the cloud to another place in the cloud just seems stupid
Why?
The difference between backing up your Flickr data to an encrypted drive at, say, Amazon and backing it up to an encrypted drive under your bed is: Amazon has better physical security, and far better redundancy, because (assuming you use S3) the "drive" you save it on there is actually a replicated data store that spans multiple regions and datacenters.
The point of backups is to provide redundant storage that is relatively uncorrelated to the original. Once you move the data to different disks at a different company in (if you like) a different country, you've done a lot to solve that problem.
i don't want to pay someone else to host a copy of my data when i have perfectly good encrypted backup mechanisms in place. i want offline copies of it to store, look at, modify, copy, etc. whenever i want, and the fewer companies that have access to my private data, the better.
also, many of the reasons backupify.com cites for needing backups would also apply to their own s3-backed service. "hackers", storage failure, human error, TOS violations, accidental account lapse/termination, etc.
Backupify allows you to download the data you backup to their cloud servers onto your local PC, and the base level of the service (2GB online backup) is free.
Would someone really trust Yahoo/Flickr to all their images and not have backups? Hopefully they didn't really care much about the images, else they should have copies elsewhere.
I would imagine it's not so much about the loss of the photos as much as the loss of the social aspects of those 5 years, the comments made, the photos favorited, collections followed. Moreover, there's the uploading which would be a monumental effort to redo, including titles and captions.
Not only is what you said correct, it's from the second sentence of the article: Mirco Wilhelm has the original files saved elsewhere, but the photos from his extensive Flickr collection had been linked to from all over the web, including the official Flickr blog.
I am forced to conclude that few read the article, since most of the comments have focused on the - erroneous - idea that he depended on Flickr to store his data and had no backups.
I'm not siding with or against you or Flickr or anyone else here, just pointing this out: I see nothing in the FAQ or Upgrade page that indicates a Pro account ensures your files will be backed up.
(Point being, nothing site-wide in my quick glance seems to indicate a promise of the integrity and forever-ness of your data.)
sure, but I think it's reasonable to expect that. I don't pay for gmail and if it all went pop tomorrow I would be incredibly annoyed but I would accept that you get what you pay for. If I had a gmail pro account (if such a thing existed), I'd feel a little more entitled to special treatment.
I'm pretty sure when I signed up for Flickr Pro, that their TOS said something to the effect that even should Flickr go out of business, they will ensure that your photos are preserved for a reasonable amount of time.
Nope, and neither did the photographer that is being written about.
What isn't backupable, and thus is lost forever is all of the comments he had on the various pictures, all of the groups his pictures belonged to, all of the embed's of pictures on various sites, all of the social aspect of Flickr (such as friends, family, all of those things).
I use Flickr to share my photos with the world, but I keep a local copy of all of my pictures on my local file server and on my laptop and all of the backups I have of that laptop...
Isn't that on google? Or bing, since that's who yahoo uses. That way, you could get back all of the links. It would take some developer time, but that's exactly what's needed for PR.
1. people who've been in the situation of lost data when somebody else responsible for its backup hadn't backed the data up.
2. people who don't belong to the first group.
After graduating from the second group into the first, i've always been classifying my data as either "can loose, fnack it" or "i'm backing it up myself, right now"
What it implies is that the internal tool(s) that the support staff uses was designed to do deletion instead of status changes. I imagine they have a web interface that fires off some backend script which a) runs sql deletes on an account ID and all tables and b) fires off scripts that physically deletes all photos across all servers.
Taking those implications in hand, it points to an incredibly poorly designed system for account management. Some years ago, there was an internal planning meeting where a decision was made to do this. Some people objected and were overruled. The ramifications of that project manager are still sticking it to people to this day.
I have a hard time believing that Flickr would have purposely engineered their systems in the manner I've described. Not impossible, but unlikely.
So I think its more likely that the data is still there, but that customer service has no method to resurrect it. Its probably a mix of internal policies and the lack of devoting a couple of engineers to fix the problem that is causing all of this.