Yes, site operators actually block non-Googlebot crawlers. See the example [0] f... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

loeg on Dec 25, 2020 | parent | context | favorite | on: Ask HN: Why does Pinterest dominate Google text se...

Yes, site operators actually block non-Googlebot crawlers. See the example [0] from https://news.ycombinator.com/item?id=25538842 .

Spoofing crawler identity completely defeats the point of the honor-system robots.txt.

Join us for AI Startup School this June 16-17 in San Francisco!
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact