On: Robots.txt meant for search engines don’t work well for web archives

[icebraining]

Yes, it is. Even loading a file to RAM was considered a copy (see MAI Systems Corp. v. Peak Computer, Inc) until Congress made an explicit exception.