I wonder if a kind of Seti@Home approach could work - although I'm guessing the limited VRAM in most consumer cards compared to an H100, as well as the much slower "virtual WAN interconnect" versus the mellanox goodies that nVidia clusters enjoy would be too big an obstacle?
Even if you could get that to work, how many people would be willing to run their >300W GPUs at full tilt 24/7 in order to contribute to the training cause? You would basically be asking people to deal with the logistics of running a cryptocurrency mining operation but without the prospect of getting paid for it.
Depends on the logistics. If I were confident about the security, I wouldn't mind letting my GPU participate in a distributed effort to significantly improve an open source model. This should be a few dollars a month on my power bill, not dozens or hundreds of dollars, especially if I undervolt.
Now, I don't know of any distributed training technique that will make a significant impact on improving a model, and that security component is a big "if". But if something promising comes a long, I'd bet lots of people would be willing to donate some GPU time, especially if it were easy to set up.
