On: Web Scraping 101 with Python

[rinze]

Not my project, but if I had to do it I'd try something like the following:

* Set an autoscaling group with your instance template, max instances 1, min instances 0, desired instances 0 (nothing is running).

* Set up a Lambda function that sets the autoscaling group desired instances to 1.

* Link that function to an API Gateway call, give it an auth key, etc.

* From any machine you have, set up your cron with a random sleep and a curl call to the API.

And that should do the trick, I think.

[ses1984 → rinze]

>From any machine you have, set up your cron with a random sleep and a curl call to the API.

You might as well just call the ASG API directly.

[warsheep → rinze]

Why use autoscaling and not just launch the instance directly from lambda? The run time is short so there's no danger of two instances running in parallel