Hacker News new | past | comments | ask | show | jobs | submit login

> Nothing in Protobuf is suited for streaming.

Uhh... Why? Protobuf supports streaming replies and requests. Do you mean that you need to know the message size in advance?






No, Protobuf doesn't support streaming.

Streaming means that it's possible to process the payload in small chunks, preferably of fixed size. Here are some examples of formats that can be considered streaming:

* IP protocol. Comes in uniformly sized chunks, payload doesn't have a concept of "headers". Doesn't even have to come in any particular order (which might be both a curse and a blessing for streaming).

* MP4 format. Comes in frames, not necessarily uniformly sized, but more-or-less uniform (the payload size will vary based on compression outcome, but will generally be within certain size). However, it has a concept of "headers", so must be streamed from a certain position onward. There's no way to jump into the middle and start streaming from there. If the "header" was lost, it's not possible to resume.

* Sun RPC, specifically the part that's used in NFS. Payload is wildly variable in size and function, but when it comes to transferring large files, it still can be streamed. Reordering is possible to a degree, but the client / server need to keep count of messages received, also are able to resume with minimal re-negotiation (not all data needs to be re-synced in order to resume).

Protobuf, in principle, cannot be processed unless the entire message has been received (because, by design, the keys in messages don't have to be unique, and the last one wins). Messages are hierarchical, so, there's no way to split them into fixed or near-fixed size chunks. Metadata must be communicated separately, ahead of time, otherwise sides have no idea what's being sent. So, it's not possible to resume reading the message if the preceding data was lost.

It's almost literally the collection of all things you don't want to have in a streaming format. It's like picking a strainer with the largest holes to make soup. Hard to think about a worse tool for the job.


Ah, you're an LLM.

Protobuf supports streaming just fine. Simply create a message type representing a small chunk of data and return a stream of them from a service method.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: