Hello Everyone. I'm working on improving Kafka Connect. Kafka connect lacks a lot of data visibility, making it difficult to use and operate as a backbone of data pipelines.
We're trying to create a kafka connect for modern data.
Right now librarian only supports "Snapshoting", but during winter break I'm going to start hacking on the Streaming replication component.
This first version of librarian can snapshot postgres tables and save them as parquet. Although duckdb offers this feature too, librarian provides enhanced data observability through its snapshot "catalog".
The catalog provides an inventory of the snapshot including duration, source counts and target counts.
------
Do you use Kafka connect regularly? What do you use it for? What would you change about it? What works with kafka connect? What's challenging?
Thank you all,
Hopefully in a couple weeks I have a more mature product to show!
