> In the end, the conclusion is that there’s no real way to us fix this that would stop “attacks” against small consumer grade sites without also significantly degrading the overall functionality.
Nonsense. Every web crawler should have some form of rate limiting. That's just good etiquette. I can control the number of requests that the Google search indexer sends to my site via webmaster tools. I don't see a good reason why Facebook can't be a good net citizen and do the same.
I think in this intance we've passsed the line between "not being nice" and "being a dick", or in this case being an accessory to a dickish move.
If the issue is used to DoS a site Zuckerberg cares about, not that I would encourage such action as it would in itself be a dick move, I'm sure some form of rate limiting will be implemented in short order...
I'm kind of wondering why they need so many different servers to fetch the file from the remote host.
It would be smart to at implement rate-limiting, and also delegate to a specific server close to the host to fetch the image, and then sync the image/file across their own networks to whatever server needs it.
It is just a huge waste of bandwidth that 100+ servers need to fetch the file, instead of Facebook itself absorbing the cost.
Well I would assume that when you add a link to a note the link gets thrown into a queue and a cluster of servers pops items off that queue to fetch and store the result. Also remember FB sees each link as a different link so it can't fetch it once and share it.
This is actually the third time this has been submitted [1] (including once by the actual author of this post). There's an extra `/?` appended to the url so HN thinks it's a different link. Not sure why - just adding the slash and question mark doesn't change the target.
Actually it has a name, "The HN Effect" (inspired by "The Digg Effect", I guess) and it's a real thing as many web sites go down when they are featured in the front page.
Are you saying this is a bad thing? Personally I think it's a good thing that the duplicate detector is easy to get around. Allows for submissions to get multiple chances.
Yeah, the HN admins (including dang, very recently) have said that the detector is meant to be porous.
It isn't meant to stop you if you are intent on reposting a dup, it is just there for informational purposes and to make you stop and think about it before you really do it.
I don't get it. Why not just allow duplicate submissions, but show a warning and link to the previous submissions before confirming the submission? Or if you intend the URL trick to be semi-secret and thus limited to "power users," then just allow duplicate submissions for users meeting some criterion, like account age or a karma threshold.
Thought Facebook had already fixed it. I tried with an image on my domain and it only went till about 660KB/s.
Then I found a bigger file.
It's now maxes out my upload speed (50mbps on fiber). The notes have been deleted but Facebook continues to do http requests for the file. That, or Apache continues to write requests to the access log after they finished and Facebook does not close active connections when it knows that the answer will never be used.
Edit: Found a video (big file) hosted by Facebook. Guess who's under attack now :D
Edit2: Seems they're loading at about 1.6-1.9gbps speed, calculating from how quickly the images seem to 'load' (become blank 1x1px images) on my client and how big the actual file is.
I wonder, what is the legality of having them DDoS themselves, using something they've suggested will not impact on someone of their scale, and that they won't fix?
Are you breaching their ToC or AUP? (I'm sure I've "agreed" to it but I doubt I've ever read it in full.)
> In the end, the conclusion is that there’s no real way to us fix this that would stop “attacks” against small consumer grade sites without also significantly degrading the overall functionality.
Why not just limit the number of request per site and per user? Ie. John Smith can only make X request (let say 1000) to www.google.com.
When you're as totally awesome and uncaring about your customers as Facebook, you can toss out words like "consumer grade" to describe people affected by your fuck-ups.
Outbound traffic. That's only serving up a few large images a second. It's entirely IO bound on network output. Getting a 1Gbps connection is not a big deal.
I know, it was two separate comments. A: It's "just" IO, performing probably the most heavily optimized IO pattern in the world (serving static files) and B: buying a 1Gbps uplink is trivial.
1 Gbps dedicated uplink is not trivial everywhere in the world. You also need more quota. Facebook easily crawled more than 1 TB of data during an hour. The more the better may not be best solution here. The higher the uplink, faster Facebook may crawl. So instead of transfering 1 TB in an hour for a 1 Gbps uplink, the transfer will be 10 TB if you have 10 Gbps. This will also depend how much bandwidth they allow their crawler, there must be an upper limit though.
I'm surprised they aren't going to give him a bounty for this. I also assume they realize that most reporters will post their rejected findings soon after they get denied.
If enough people start using technique they will have no choice but to create a fix for this.
Exclusions
The following bugs are not eligible for a bounty (and we do not recommend testing for these):
...
Denial of Service Vulnerabilities
...
That said, they do appear to be quite hypocritical when it comes to enforcing this, as at least one user on the /r/netsec thread claimed that Facebook paid them for a mail bombing vulnerability.
I guess you have to use this to attack high-profile targets (or Facebook themselves) until they notice.
It's not clearly stated. I think what they mean to say is DoS on Facebook using brute force mechanism as testing this would bring down the Facebook system itself. Now manipulating Facebook's server to cause the attack, that'd be different I guess.
> If you can easily cause any website to become an on demand low orbit ion cannon
You can do this with any website that lets users post links which may or may not be automatically loaded (by other users too). There are tons of these around; forums, blogs, etc. See also: Slashdot effect. That's also a true DDoS, coming from many IPs, as compared with a few of Facebook's servers.
They were probably aware of the issue, as this kind of bug has been already published for Google [1] (by the same author, it seems) a month ago. Also, Google did not give him a bounty reward for his discovery.
I actually found this bug first in Facebook and later in Google. Facebook said they needed time to think about this and I was hoping they would fix it, so I could not disclose this back then.
Join the club. I've reported it before as well, years ago. I even discussed it on my FB wall at one point with some tests against my own site. I never re-tested, figuring they'd take care of the issue. Guess not.
While this is true, it's also not a huge problem, exactly:
> HTTP status codes are extensible. HTTP applications are not required
> to understand the meaning of all registered status codes, though such
> understanding is obviously desirable. However, applications MUST
> understand the class of any status code, as indicated by the first
> digit, and treat any unrecognized response as being equivalent to the
> x00 status code of that class, with the exception that an
> unrecognized response MUST NOT be cached. For example, if an
> unrecognized status code of 431 is received by the client, it can
> safely assume that there was something wrong with its request and
> treat the response as if it had received a 400 status code. In such
> cases, user agents SHOULD present to the user the entity returned
> with the response, since that entity is likely to include human-
> readable information which will explain the unusual status.
To the people saying that this won't really hurt facebook, I beg to differ, this is a relatively simple to deploy DDOS attack, if enough people start utilising it, I can see a fair amount of Facebooks resources being tied up in these attacks.
They will only fix it when it starts to hurt them it seems.
This can be fixed easily ( i have read it fast, but i suppose this could be a fix)
Add a boolean to a attachment in the db, queryDiff
Add a string (for md5 hash)
Calculate the hash on every file with a query parameter, if the file is requested the second time (with different query parameters), check if the file hash is the same.
If the file hash is the same, change the bool queryDiff
Next time you fetch the file, queryDiff is false, so you shouldn't fetch the url and only get the original one (which was already downloaded)
There's no way to know if difference in query parameter is a different image or not, you can't simply re-use a prior URL which is equivalent but for query param in place of a new one.
contain the same images, chances are very high that other query parameters will be the same.
If they are different, chances are very high that other query parameters are also different.
Want to up your chances, then instead of only 2 requests, raise the bar to 10 different query parameter checks and add some additional db values (like CountFetch:int,sameUntillCurrentFetch:bool)
As soon as sameUntillCurrentFetch = false, then you request all images in the future.
If CountFetch = 5 and sameUntillCurrentFetch = true, then queryDiff = false.
It's kinda weird my answer was downvoted, any better solutions to avoid the stated problem then?
So giving a 404 for unknown get parameters should fix this for your own site? A way for Facebook to detect such a thing would be to hash the images and when two images have the same hash and only differ by some get parameter it could remember that it can ignore that parameter.
Not a reliable check. If the first 2 are the same, then this would cause a false positive as the third, fourth and fifth could all be completely different.
Nonsense. Every web crawler should have some form of rate limiting. That's just good etiquette. I can control the number of requests that the Google search indexer sends to my site via webmaster tools. I don't see a good reason why Facebook can't be a good net citizen and do the same.