Hacker News new | past | comments | ask | show | jobs | submit login

>I could be wrong, but I assume that Google mostly does this because of the huge amount of malicious garbage bot traffic on the web. Once you go fully anonymous and private, there's no way to distinguish you between the hordes of malicious traffic.

What's "malicious" about simply opening a page on a website? Google bots are doing this all the time and they don't have to solve CAPTCHAs.




If your site is at all worth scraping, it will be scraped, republished and monetized by someone other than you.

Google plays the long game though. By setting themselves up for being the gatekeeper of bot traffic, any bot belonging to a competing search engine cannot possibly index the web as well as Google can.


I assume Google Recaptcha has a way to recognize its own Google Bots.


Isn’t that kind of an antitrust thing?

How does a good, non-Google not solve a Google Recaptcha?


> Isn’t that kind of an antitrust thing?

> How does a good, non-Google not solve a Google Recaptcha?

I'm not sure if it's antitrust, because I would imagine that website admins would _like_ Google to have access to their site while excluding others.

What people fail to realize, I think, is that if this practice had been common, something like Google would have never came to be in the first place! Imagine if websites blocked all but AskJeeves user agents from crawling them?


Just to be clear, I meant to say "a good, non-Google bot".




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: