Show HN: Open-Source Webhooks Gateway for Platform Engineers

[rtukpe]

https://github.com/frain-dev/convoy

Hey Friends,

Convoy is an open-source webhooks gateway. Webhooks continue to be hard at scale, and large teams require consistent tooling for inbound & outbound webhooks. Convoy enables developers to securely send, receive and manage millions of webhooks reliably with features like retries, rate limiting, circuit breaking, customer-facing webhook logs, zero downtime secrets rotation and more.

Since our initial launch [0], we've learned a lot about our users and made several important improvements we are excited to share:

- We are now a webhooks gateway! Akin to API Gateways that sit at the edge of your network to receive all API traffic and route them to the respective microservice, webhooks gateways sit at the edge of your network to receive webhooks from any backend service and route to client endpoints as well as ingest events from multiple providers and route them to the required backend services.

- We now have first-class integration with Pub/Sub Systems. Our users wanted increased deliverability guarantees. Your backend services write events to a queue/topic etc. Convoy consumes the queues, creates webhook events and dispatches those reliably to client endpoints. We currently support Amazon SQS and Google PubSub. On our roadmap, we have the following planned - Kafka, RabbitMQ & Nats (In that order)

- We switched our backend store to PostgreSQL. This improves the self-hosted experience tremendously. MongoDB was great for storing schemaless data, but was severely lacking in some important features for our users, e.g. transactions don't work on a single node; you need to bootstrap a replica set; also, we wanted to ship updates frequently, but the lack of migrations for schema & data changes slowed us down.

- We decided to go the open-core route of OSS monetisation because it offered us a good balance to serve our community and make enough money to run the company. Like GitLab, we hope to be good stewards of our community edition. Since our enterprise edition is simply a wrapper around the community edition with enterprise features like RBAC, Audit Logs etc., we are properly incentivised to continue making it excellent.

- Our Cloud platform is in the private alpha stage. Please contact us at founders@getconvoy.io to gain access!

Our mission is to serve hobbyist developers all the way to the most ambitious teams on the planet with a consistent and easy-to-use webhooks gateway for asynchronous communication on the internet.

We welcome you to try it out using our getting started at https://github.com/frain-dev/convoy#installation-getting-sta.... Share with us your webhook horror stories and give us feedback.

[0]: https://news.ycombinator.com/item?id=30469078

[toomuchtodo]

Hello! Love it (webhooks are traditionally an unloved but crucial system component)! How would you say you compare to and differentiate from https://svix.com?

Edit: Thank you for the reply! Wishing you much success.

[subomi → toomuchtodo]

Thanks!

While we are pretty similar, several differences still exist today.

- Convoy is fully open-source, including our UI, adding multiple users, creating multiple projects, debugging & managing events. Svix has only its dispatcher open-source without a UI.

- Convoy directly integrates with PubSub systems like Amazon SQS & Google PubSub to ingest webhook events. Svix can only ingest events via a REST API.

- Convoy has features to send and receive webhooks. Svix has features to send webhooks only.

- Convoy is written in Golang, Svix is written in Rust. :)

[TeeWEE]

What’s wrong with a autoscaling container behind an load balancer that has a http handler with optionally a pub sub in between? Security, retry etc are not hard. You could use a service mesh. Why not use k8s

[subomi → TeeWEE]

It's the usual how difficult can this be to build and operate. But what we've found is it's a pain at scale.

- How do you ensure that one or a set of bad endpoints do not clog the queueing system and delay delivery of other working endpoints?

- How do you provide easy tools for developers across projects to debug and fix webhook delivery problems?

- How do you ensure this container is multi-tenant to serve multiple teams and no two teams can view each other data?

- How do you ensure flexibility if a new team decides to use RabbitMQ instead of Kafka?

Once you start to factor in all this requirements & more, you eventually build your own Convoy. :)

TLDR: Convoy is that container that is flexible and powerful enough to manage all your webhooks needs.

[djbusby]

Excellent! I was working on taking an internal project to FOSS that does exactly this. Thank you!

[avinassh]

This looks great! I will give it a spin

> e.g. transactions don't work on a single node

any Mongo has this restriction

[MaBeuLux88 → avinassh]

If you are in a development environment, you can set up a single node Replica Set. Some features like transactions and change streams rely on the oplog collection which only exists in Replica Set. A Replica Set can work with a single primary node. You won't have fail over of course, but it's not the point in a dev environment.

[subomi → avinassh]

Yup, it sucked! Users had to spin up a replica set all the time. Wasn't fun.

[avinassh → avinassh]

wanted to type: any reason* Mongo has this restriction

[rtukpe → avinassh]

I don't know tbh. According a post on the MongoDB community forum Transactions are built on concepts of logical sessions they require features which are only available in replica set environment. You can see more here [0]

[0] https://www.mongodb.com/community/forums/t/why-replica-set-i...

[pojzon]

Can this work as a replacement for zapier?

[toomuchtodo → pojzon]

Consider Activepieces, Trigger.dev, or Automatisch for Zapier alternatives (if Zapier doesn’t fit your use case for whatever reason). All have recent HN threads if you use search at the bottom.

Convoy might still be a useful tool to front webhooks to the above mentioned no|low code alternatives, depending on your needs.

[ashrafsam → toomuchtodo]

Thank you for the mention. Founder of Activepieces here ready to help/support with whatever needed - we hang out on Discord all the time.

[subomi → pojzon]

No, it can't. But it's a better replacement to receive the webhook before pushing it to zapier. Zapier doesn't give you good logs of webhooks to retry so you can re-trigger a workflow. Convoy does. :)

[ddorian43]

How does it compare with svix.com?

[subomi → ddorian43]

Hey,

I responded to this in the comments below, see here [0]

[0]: https://news.ycombinator.com/item?id=35380957