Hacker News new | past | comments | ask | show | jobs | submit login

is there a way to guarantee reading your own writes from a client perspective?



Yes. Depot appends by default don't return success until colocated streaming topologies have completed processing the data. So this is one way to coordinate the frontend with changes on the backend.

Within an ETL, when the computations you do on PStates are colocated with them, you always read your own writes.


It makes sense, but wouldn’t the write be slow? Especially when you have many streaming pipelines.


That's part of designing Rama applications. Acking is only coordinated with colocated stream topologies – stream topologies consuming that depot from another module don't add any latency.

Internally Rama does a lot of dynamic auto-batching for both depot appends and stream ETLs to amortize the cost of things like replication. So additional colocated stream topologies don't necessarily add much cost (though that depends on how complex the topology is, of course).




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: