Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
baccheion
on July 10, 2018
|
parent
|
context
|
favorite
| on:
Lessons learned scraping 100B product pages
The best buffer against scrapers/spammers seems to be lag. That is, progressively slow the rate at which data is returned.
Many bypass protections by limiting request rate and using a pool of lesser known proxies/IPs.
Consider applying for YC's Spring batch! Applications are open till Feb 11.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
Many bypass protections by limiting request rate and using a pool of lesser known proxies/IPs.