Firstly, this post does not even mention or describe what kind of information was exposed and makes it seem like some inadvertent tracking that's not serious in any way. You can continue with your root cause analysis, but you already know very well about what kind of information was exposed and shared with Facebook. The least you could do is own it and alert people about it.
> We take the privacy of our customers’ data and personal information very seriously...
It's astonishing that there is no apology of any kind on this post. Is that too much to ask for such a grievous violation?
> On March 8, 2021 at 8:39pm Pacific time, a new Facebook campaign was created that started firing a Facebook advertising pixel, intended to only run on marketing web pages. However, it was inadvertently configured to run on signed-in pages.
It went unnoticed for two weeks, with Facebook gathering information about the customers' files! And yet, no apology!
There will never be an apology. A corporate apology is an admission of guilt. An admission of guilt opens it up to lawsuits. Through its litigeous nature, the US has evolved a culture whereby apologies and human decency are antithetical to survival.
Yev from Backblaze here -> I apologize for this mistake. We've updated the post with additional information now that we have it and there's also an apology.
Yev from Backblaze here - with sincere apologies. After we removed the offending code on Sunday night we went into investigation mode and conducted a rca - we wanted to get some kind of blog post out there so that folks could see that we were working and investigating the issue. I will grant you that maybe our initial execution on that was lackluster, but we wanted to make sure we had a better handle on what we were talking about. We've since updated the blog with additional information, you can read those here: https://www.backblaze.com/blog/privacy-update-third-party-tr....
> We are also reviewing applicable third party code on the website.
This is very simple. On all dashboard and similar signed-in pages, there's exactly zero applicable third party code.
If you think otherwise you do not in any way take "the privacy of our customers’ data and personal information very seriously".
As such, you should be much more specific about what you mean with "we removed the offending code".
If you never intended for third party scripts to run on the dashboard page, well then you got some serious explaining to do as to how your deployment process let that slip.
We also subsequently removed Google Tag Manager from the private pages.
This is good. However, the question then becomes how on earth did Google Tag Manager get added to those pages to begin with?
Either you did not consider this a cause for concern, or it slipped past your code review. In either instance, how could I ever trust you with my data?
Yes, this is rather disappointing. It's probably also in breach of GDPR assuming backblaze haven't contacted customers directly, informing them about the nature and detail of leaked data (based on your use of backblaze on X, we see that the following filenames and Metadata was incorrectly handed over to a third party (Facebook)...).
Possibly. But Backblaze has an agreement with the user; and they have taken data they need / are justified in handling (filenames, sizes) and leaked them. Google docs wouldn't be liable if I stole your credit-card and put it in a Google doc.
It may be that fb is doing something illegal here too - eg the whole way tracking pixel work seems difficult to square with gdpr - but the tracking pixel/service looks a bit like the drill used to force a lock - another party (like BB) must deploy it. And arguably the purpose is not to "steal" data.
>a new Facebook campaign was created that started firing a Facebook advertising pixel, intended to only run on marketing web pages. However, it was inadvertently configured to run on signed-in pages.
"We take privacy very seriously" is a sign that they don't take your privacy seriously at all.
They literally sent file names and file sizes over to Facebook. That might very well include very personal or confidential data. The fact that they don't even mention this and make it sound like some random unimportant tracking happened without them noticing is completely ridiculous.
Yeah, like file names of say PDF's, files with unique identifiers that give insight to the users personal life, worse still profile names of PC backups etc. etc.
Security in services is all about liability and risk rather than an absolute guarantee.
I've seen so much anger over this issue that it's left me confused. Questions like how they can ever be trusted now. They could never be trusted. If the information is really that important, then it should be encrypted before being passed to any other service. Companies will screw up, the question is how are you going to be on the hook for it. The great thing about services is that you can pass the blame the service than if you had dealt with it in-house.
That said, this has been an embarrassing display for Backblaze and I hope they redouble their efforts on infosecurity. But mistakes happen. If there's a pattern, then that's a different story.
There are known mechanisms to prevent leaks/exploits of this sort by sending some additional headers[0], on most of the modern browsers. Apart from that, the cookies should be strict for such a service. I logged in to my friends account to find neither implemented.
They better act quick about "ensuring it doesn't happen" or I will distrust them completely.
Backblaze is missing the part where you conduct an investigation, request assistance from Facebook, and determine what data was leaked to third parties.
> Our Engineering, Security, and Compliance/Privacy teams—as well as other staff—are continuing to investigate the cause and working on steps to help ensure this doesn’t happen again.
Silo the sales and marketing department away from the real products like the person in that other thread was saying. Sales people can't be trusted with private data.
Man, your distrust seems toxic to me. The simplest explanation is they goofed as they admitted, if they were malicious, they'd upload those file names from their server straight to FB's, not through your god damned browser.
And of course they only addressed it after they "got caught", they didn't know about their fuck up before that, if you want to sneer at them, sneer at them for not being careful enough to let this happen, not what you wrote.
I'm not trying to defend them, more like I want to protest against ungrounded casual insulting bashes like yours that seems way too freaking common nowadays. I do think they care about customer privacy, because it affects their income. (Admiteddly they were too casual about it, they were still partnering with Facebook...).
> We take the privacy of our customers’ data and personal information very seriously
I wish companies would actually mean this and apply it proactively instead of just copy and pasting this in their apology after something like this happens.
> We take the privacy of our customers’ data and personal information very seriously
Yet the first thing I'm greeted with is a cookie consent dialog with "allow all cookies" being the only prominent button.
If I don't want to allow the privacy-invading tracking cookies I need to click a less prominent piece of text called "manage cookies" and avoid clicking on another prominent button called "accept all cookies".
Even the title suggests that it's just a heads up on some minor functionality. Not really what I would expect from a company that puts "Trusted Storage Cloud" in their <h1> and encountered an incident that completely eradicates that trust.
I've been a happy Backblaze B2 user for the past few months, but this and another recent event leaves me feeling like I'll have to abandon their service (or at least mirror everything to S3 just-in-case) soon. Recent event being when GoDaddy suspended their user-facing B2 domain a month ago because of abuse requests that were apparently not handled correctly. That and the fact that they still don't have a user-facing status dashboard to communicate the status of their systems has me worried.
Yev from Backblaze here -> thanks for being a Backblaze customer. We've finished our root cause analysis and have updated that blog post with additional information. We also have moved domain registrars to make sure that behavior doesn't happen again. Sincere apologies for the issues you've experienced.
Looking at this submission, comments and its ranking here, I’m sure that nobody from Backblaze will comment here or on HN for some more time. This blog post makes it seem like an internal gag order has been imposed. Some hand wavy updates are what we should expect I suppose.
The post title is also dishonest in saying third party tracking and not adding details — this is Backblaze that intentionally and explicitly shared sensitive data with third party tracking. It’s not as if some third party tracked Backblaze’s paying users surreptitiously.
This response is actually helpful for the competition as far as the HN audience is concerned.
Edit: I’m really curious on the extent of coverage of this issue. I searched online and found only mjtsai.com and theregister.com covering this sensitive information leak. Are there any other major tech sites that have covered it so far?
I've started using rsync.net with the borg client (replacing b2 and rclone) and loving it so far. I'm also surprisingly finding borg/rsync.net is suiting my usecase (personal backups) better than b2 did which is a nice bonus. I am paying more but can't complain as the value I'm getting is still good.
If you're in the EU and a backblaze user and consider that some PII may have been leaked to 3rd parties, you can contact your local GDPR watchdog (aka national "Data Protection Authority", or DPA).
I expect zero http requests to any Facebook servers when I visit anycorp dot com and that’s regardless of whether I’m a paying customer or logged in user.
If any actual data is transferred that’s even worse, but already having a connection to Facebook via a tracking pixel for “audience building” or similar is over the line already.
oof, this blog post is even more cringy than their repeated replies on twitter "...we’ve looked into and verified the issue and have pushed out a fix. We will continue to investigate and will provide updates as we have them."
this is a perfect example of how NOT to handle a PR nightmare.
Oh well, that's it then. I guess I'll only use Backblaze through `rclone` and utilize client-side end-to-end encryption. And I might still not use Backblaze.
You are totally right what info could leak a files named:
ClinicWhereTheyReattachYourJunkWhenItFallsOff_procedureOverview.txt, ClinicWhereTheyReattachYourJunkWhenItFallsOff_invoice.pdf and ClinicWhereTheyReattachYourJunkWhenItFallsOff_WhatWentWrongWithYourJunkBeGratefulYouHadTheChanceToSayGoodbye.rtf ..
It's the old debate on the fact that filenames might be kinda sensitive.
> We take the privacy of our customers’ data and personal information very seriously...
It's astonishing that there is no apology of any kind on this post. Is that too much to ask for such a grievous violation?
> On March 8, 2021 at 8:39pm Pacific time, a new Facebook campaign was created that started firing a Facebook advertising pixel, intended to only run on marketing web pages. However, it was inadvertently configured to run on signed-in pages.
It went unnoticed for two weeks, with Facebook gathering information about the customers' files! And yet, no apology!