I don't think a poorly written scraper would follow robots.txt rules according t...

eknkc on April 21, 2017 | parent | context | favorite | on: Robots.txt meant for search engines don’t work wel...

I don't think a poorly written scraper would follow robots.txt rules according to spec. So, in any case the site should have other measures (rate limiting?) anyway.