I agree. So many disparate solutions. The streaming sql primitives are by themselves good enough (e.g. `tumble`, `hop` or `session` windows), but the infrastructural components are always rough in real life use cases.
Various streaming processing frameworks often have domain specific languages with a lot of limitations of how to express aggregations and transformations.
Yeah I have a feeling something like polars for streaming would be super popular and useful, but it just hasn't happened yet. It's much easier to just do say kafka and a long running python script and write out the transformations by hand, than it is to use anything on the market right now. None of the current streaming processors want to be embedded as far as I can tell, that's not where the money is. They all want to be paid to run it in the cloud for you and follow that vc playbook model. Which, fair! I do think there's a lot of space out that isn't being occupied though and I hope somebody tries to fill it soon.
(As an aside, feldera doesn't want to be embedded into your app, materialize either, and clickhouse might just pull a great streaming library out from nowhere, they seem to be good at just doing stuff like that).
I think slow writes are not a major concern, as most databases already use some fast log-type data structure to persist writes, and then merge/save these logs to a higher-capacity and slower medium on specific events.
You'd still need to implement any custom selection widgets, data transformations (like other statistical tests) etc. still missing, but i like the technical design to build on top off. It uses https://github.com/observablehq/plot under the hood, which aims to have just as flexible a grammar as ggplot (already quite capable) but with interactive features (built by the creator of d3 and uses it under its hood).
I don't think it was for the technical fit or performance reasons, but more a philosophy about everything starts with data, and graphics are just visualizations anchored to the data points (or a functionally derived property of the data points).
That also means the d3-* libraries compose really well, since the data is the common binding, and not some conceptual class or custom element.
I also have written a lot d3, between versions 2 and 7, and the refactoring that has happened meant a lot of examples online that were hard to comprehend were even harder to update.
I feel like its more stable now though. Something clicks for me since ive started writing it in more imperative style with svelte+d3 rather than d3 alone. The generated elements are easier for me to reason about, rather than otherwise relying on inspecting the generated elements with dev-tools after the generation.
