> Googlebot has a range of IP addresses that it publicly announces so websites c... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

quitethelogic on March 26, 2021 | parent | context | favorite | on: Only Google is really allowed to crawl the web

> Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.

Google says[1] they do not do this:

"Google doesn't post a public list of IP addresses for website owners to allowlist."

[1]https://developers.google.com/search/docs/advanced/crawling/...

johncolanduoni on March 26, 2021 [–]

From that same page they recommend using a reverse DNS lookup (and then a forward DNS lookup on the returned domain) to validate that it is google bot. So the effect is the same for anyone trying to impersonate googlebot (unless they can attack the DNS resolution of the site they’re scraping I guess).

Consider applying for YC's Spring batch! Applications are open till Feb 11.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact