On: Introducing Preemptible GPUs

[boulos]

Losing some work in exchange for a large discount is still a good trade though. It's a non-goal to provide 100% efficiency :).

[argonaut → boulos]

My point is, you would have to write a nontrivial amount of code to get this working. It would not work out of the box with Keras right now, for example.

[boulos → argonaut]

I think you skipped over my first comment. This is built into Keras (and was the second result on Google for 'keras checkpoint'): https://keras.io/callbacks/#example-model-checkpoints

I don't consider that a nontrivial amount of code to write. Am I missing something?

[argonaut → boulos]

I am well aware of Keras' checkpoint code. It only supports checkpointing every epoch. So no, it does not work out of the box with Keras.