Thanks for posting this. Let me add some background for this demo we built to showcase incremental computation with Feldera (SQL compiled to do incremental evaluation). https://github.com/feldera/feldera
The gist of it is that if you update a cell, we incrementally update the spreadsheet which means we will only emit a minimal amount of changes for the cells affected by your update. The nice thing about it is that this is something that Feldera does automatically (and it would do that for any SQL that you end up writing, so it doesn't have to be a spreadsheet, but a spreadsheet is a nice example that everyone understands and knows about).
There is a more detailed explanation in this video https://www.youtube.com/watch?v=ROa4duVqoOs if you're interested what's going on under the hood -- or if you prefer reading about it we have an article series that goes over all the parts of the demo
Hi, you might be thinking of another key-value store (FoundationDB maybe?). RocksDB can be linked as am embedded library so it doesn't need to run in a separate address space.
Coincidentially, SQL is having a "revival" in the streaming world... feldera, materialize, arroyo and many others are all betting on SQL as the language to express computation on moving data.
I didn't - I wanted to build something I could understand, and the rust implementation comes with a load of (reasonable) complexity, some resulting from the execution model, some from performance requirements.
stepping has 3 Zset implementations - in memory, sqlite and postgres. Currently considering whether to write a small rust one.
We do think so! (disclaimer: I'm a co-founder at Feldera)
To give some more background: We are trialing feldera with several industry/enterprise partners from different domains. Our core team also built differential datalog (https://github.com/vmware/differential-datalog) in the past. And while ddlog is used quite successfully in products today, we believe the many lessons we learned with ddlog will help us to build an even better continuous analytics platform. FYI our code is open-source at https://github.com/feldera/feldera if you'd like to try it out.
If you don't want to use malloc in a kernel, that is fine. However, you should at least consider having something similar. This can be as simple as finding the available memory in your system and writing a frame or slab allocator. It will make your life much easier. It would be interesting to see how easy it is to integrate your memory manager directly with the rust language.
