I've also been playing with Scaleway's ARM servers for a scraping project and to learn more about distributed computing. (details here in other HN thread https://news.ycombinator.com/item?id=10371316)
I've been able to scale my scraping to over almost 100 requests per second in aggregate across 1.8MM domains. And these little Atom quad core machines are great.
I just got my account upgraded to "developer" which mean I can spin up 100 machine (400 cores!), so I'm going to be experimenting with that.
I've been able to scale my scraping to over almost 100 requests per second in aggregate across 1.8MM domains. And these little Atom quad core machines are great.
I just got my account upgraded to "developer" which mean I can spin up 100 machine (400 cores!), so I'm going to be experimenting with that.