Show HN: Goka – distributed stream processing library for Kafka written in Go

[flosch]

https://github.com/lovoo/goka

[malandrew]

Anyone interested in this should check out Uber's Cherami, which has support for go, java and python:

http://eng.uber.com/cherami/

https://github.com/uber?utf8=%E2%9C%93&q=cherami&type=&langu...

It supports consumption of kafka topics:

https://github.com/uber/cherami-server/commit/226fb2c55b7c51...

[u320]

Why would anyone want to use this over Kafka Streams? Seems a lot more verbose.

[fdsfsafasfdas → u320]

Perhaps because they would like to call into code written in Go.

[ah- → u320]

No Kafka Streams for Go yet.

[jaytaylor → ah-]

re: yet

Is the work in progress? If so, please tell us where.

[alexott → jaytaylor]

As I remember, they said that tgey want to stabilize API before adding support for new languages

[0xdb7 → u320]

Goka is not a replacement for Kafka Streams, they can be used together. For example, you can use streams and tables created with Kafka Streams inside services built with Goka.

[agnivade]

How does this compare with https://github.com/movio/kasper ?

[0xdb7 → agnivade]

There are several differences. Here are some of them.

- Kasper is micro-batched, Goka consumes one message at a time.

- Kasper requires a centralized state storage (such as Redis or ElasticSearch), Goka only depends on Kafka: processors persist their state (key-value tables) in a compacted topic in Kafka. That is really nice because processors can then join streams with tables of other processors.

- Kasper statically assigns partitions to processors. Goka automatically splits the input and state partitions among processors.

- By default Goka employs sarama and sarama-cluster libraries to access Kafka, but that can be easily replaced with Confluent's go library. Similarly, the local storage is by default LevelDB, but can be replaced even with some remote storage if necessary.

[wubal]

Happy to see a Go alternative.. Good job OP