Usually the inference time is small compared with download time so even if this were technically feasible you wouldn’t save much time.
For reference I have a 31mb vision transformer I run in my browser. Building the inputs, running inference, and parsing the response takes less than half a second.
LLMs like ChatGPT only generate one token at a time. To generate more you run inference repeatedly until you reach a stop token or some other predetermined limit.
I don't see streaming helping anything besides maybe Time-To-First-Inference, but regardless, you're still not getting any output until the entire weights are downloaded.
For reference I have a 31mb vision transformer I run in my browser. Building the inputs, running inference, and parsing the response takes less than half a second.