Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
sn_master
on March 27, 2021
|
parent
|
context
|
favorite
| on:
Only Google is really allowed to crawl the web
Their IP range can be blocked. Google has known IP ranges and if the website admin wants can allow them based on that rather than robots.txt, and that would really mess up everyone else.
Consider applying for YC's Spring batch! Applications are open till Feb 11.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
