For most applications, getting training images onto the GPU isn't the bottleneck by far. Training the Inception model, for example, handles batches of 32 images (299x299x3) in 1.2 seconds. That's a pretty boring ~300KB * 32 ~= 10MB/sec of read bandwidth off the SSD for Imagenet. Even dealing with "real" images is probably only 10x that, which is trivial to get over the PCI bus.
The question would be whether we can turn the crank on the design of models to make it possible to do something really cool given access to very high-speed SSD storage.
The question would be whether we can turn the crank on the design of models to make it possible to do something really cool given access to very high-speed SSD storage.