It’s been discussed for a long time, but the ecosystem is still spotty. Many databases have some kind of “live query” feature for instance but with limitations or not intended for production use.
A lot of work has taken place in the Kafka, Flink, DataFlow ecosystem but that still leaves a lot of work for the developer over a simple subscribe to query results.
I do think a lot of work has been done, but it all needs to move up a few levels of abstraction.
> A lot of work has taken place in the Kafka, Flink, DataFlow ecosystem but that still leaves a lot of work for the developer over a simple subscribe to query results.
Personally I find it much easier to write the code explicitly than try to understand what a query planner is doing, especially if performance is relevant. (That's not to say there's not plenty of room for improvement in the streaming world - but I'd rather have a helper library that I can use on top of the low-level API, than have to go through a parser and planner for every query even when I know exactly what I want to do) But I seem to be an outlier in this regard.
Our focus (Hasura) is on the last mile so that innovations on the data side (eg: materialize, ksql, timescale continuous aggregates) are “just obvious” to start building applications and services against.
A lot of work has taken place in the Kafka, Flink, DataFlow ecosystem but that still leaves a lot of work for the developer over a simple subscribe to query results.
I do think a lot of work has been done, but it all needs to move up a few levels of abstraction.