Bursty workloads is key here. In your opinion, what's the main challenge with existing hourly options (like Amazon's Elastic GPUs, or similar offerings from Google Cloud) - is it the management/provisioning? So, if someone made a Lambda-like service, where one would need to worry about provisioning - you think there'd be a market for that?
Btw, I found some tasks (and some algos) don't parallelize well (e.g. would get any "faster" from employing 40 GPUs instead of 4), but it would definitely make sense to be able to run multiple models (or training) in parallel. Once your needs are well understood, it's almost always significantly cheaper to build your own GPU cluster.
Yes, it is precisely the management and provisioning that is difficult.
No one does hourly billing anymore, thankfully. GCE and AWS do per second or per minute...
What I do now is autoscale a group of GPU instances in AWS based on the size of the jobs queue. I create as many instances as will reasonably help, sometimes up to 100. Work is distributed to them. They turn off immediately once there is no more work to do.
The code runs in Docker containers but I am forced to maintain the base linux system and nvidia drivers b/c no one provides a container or FAAS for nvidia gpu computation...
I get the sense that this is a common problem nowadays. The way NVIDIA manages software releases doesn't help anything. There's quite a bit of .. churn. They don't play nice with anybody else hence Linus's famous words.
There's totally a market for containers as a service with passthrough access to NVIDIA hardware. It best not be more than 30% more expensive than a raw instance though, or it won't be very exciting.
