Hacker News new | past | comments | ask | show | jobs | submit login

robots.txt has been a de facto standard for over 20 years. Someone might be able to claim ignorance, but the Internet Archive has shown that they know about it. It has a specific format; if it can be parsed, it's safe to assume that it isn't part of a movie script.

In most cases, copyright law requires the reader of a document not to republish it, so the robots.txt standard is actually much more permissive.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: