How does this relate to the robots.txt file? Even being deferential it doesn't m... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

throwaway91111 on April 21, 2017 | parent | context | favorite | on: Robots.txt meant for search engines don’t work wel...

How does this relate to the robots.txt file? Even being deferential it doesn't make much sense to respect it.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact