I would love to know more about how they do stateful stuff, since I haven't found anything that can compare to Flink there, but Flink has really poor quality of life stuff compared to some other options (e.g. Flink serialization setup pains vs Beam coders) but their docs kinda trail off here:
> Non-idempotent stateful topologies are stateful topologies that do not apply processing logic along the model of "multiply by zero" and thus cannot provide effectively-once semantics. An example of a non-idempotent
(I'm puzzled by their idempotent vs non-idempotent stateful topology description, because if something is mutating an internal state upon receiving events, it will likely be non-idempotent by design... unless they just mean "idempotent stateful" here to refer to keeping track of source/output position state and such.)
(They also do say that can only support state storage in ZK or local FS, which feels like a likely non-starter compared to Flink for some of my use cases.)
Flink serialization become much less painful when we switch every value type in our job to Avro. It requires a bit more code compare to using Java POJO or Scala case class, but you can actually add more field to existing data type without breaking anything now :)
> Non-idempotent stateful topologies are stateful topologies that do not apply processing logic along the model of "multiply by zero" and thus cannot provide effectively-once semantics. An example of a non-idempotent
https://heron.incubator.apache.org/docs/heron-delivery-seman...
(I'm puzzled by their idempotent vs non-idempotent stateful topology description, because if something is mutating an internal state upon receiving events, it will likely be non-idempotent by design... unless they just mean "idempotent stateful" here to refer to keeping track of source/output position state and such.)
(They also do say that can only support state storage in ZK or local FS, which feels like a likely non-starter compared to Flink for some of my use cases.)