Hacker News new | past | comments | ask | show | jobs | submit login

Yes. They use the robots.txt file to essentially ascertain ownership or control of the domain. Archive.org doesn't want to delete the content they have (for whatever reason), so the compromise they came up with is to read the robots.txt and then hide the content they have archived if the present domain owner/controller wants it to be that way.

If you remove the robots.txt setting, the archives become available again.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: