Hey HN, I’m an engineer at Supabase [0] and one of the creators of this demo. My team and I have been working hard to bring developers the next version of Supabase Realtime.
The current version of Realtime [1] is a Change Data Capture (CDC) server for a PostgreSQL database that broadcasts changes via WebSockets to authorized subscribers. It’s written in Elixir/Phoenix.
The server utilizes PostgreSQL’s logical replication functionality, which writes database changes to Write-Ahead Logging (WAL) segment files, and a replication slot, responsible for managing and retaining WAL files.
Database changes are polled from WAL by the server using PostgreSQL’s replication function pg_logical_slot_get_changes and changes converted to JSON objects using the wal2json [2] extension by setting it as the output plugin.
Security is enforced through two checks - each check ensures only authorized client subscribers are sent database changes. The first check validates a JWT that is sent by clients subscribing to database changes. This JWT must contain an existing database role and optional claims, both of which can be referenced in Row Level Security (RLS) policies. Every valid client subscription is then inserted into the realtime.subscription table with an assigned UUID, database role, and claims. The second check calls the realtime.apply_rls SQL function from Write Ahead Log Realtime Unified Security (WALRUS) utility lib [3]. This function takes the database changes, executes a prepared statement to verify if the database role and claims have SELECT permissions on the changes, and outputs an array of authorized UUIDs. Then, the server finds all the subscribers whose UUIDs are in that array and broadcasts the changes to them.
The next version of Supabase Realtime will offer three features: Broadcast, Presence, and Extensions.
Broadcast, our Pub/Sub offering, can be used to pass ephemeral data from client to client such as cursor movements. This runs on a distributed cluster of nodes built on top of Phoenix PubSub + Channels.
Presence, can be used for tracking online/offline users and their state. This is built into Phoenix, and uses replicated state across a cluster using an Observe-Remove-Set-Without-Tombstones (ORSWOT) CRDT [4] which prefers adds over removes when resolving conflicts.
Extensions, are a way for the community to add additional functionality to take advantage of the WebSocket infrastructure. We have converted the existing Change Data Capture system to an extension that supports connecting to multiple customer databases (multi-tenancy). Other possible extensions include listening to other databases like MySQL and getting stock market events server-side [5], then broadcasting them to connected clients.
This demo is built using a Supabase project, Supabase Realtime, and Next.js and deployed on 20 Fly [6] nodes located around the world. You can find an introduction and walkthrough of the demo here [5].
Supabase Realtime is entirely open source and you can find the demo code here [7]. Once we have stabilized the release we will add it to the self-hosted offering [8]. This demo is a way to highlight the upcoming features and gather feedback/ideas.
Feel free to ask me anything and let me know what you think!
[0] https://supabase.com
[1] https://github.com/supabase/realtime
[2] https://github.com/eulerto/wal2json
[3] https://github.com/supabase/walrus
[4] https://gsd.di.uminho.pt/members/cbm/ps/delta-crdt-draft16ma...
[5] https://supabase.com/blog/2022/04/01/supabase-realtime-with-...
[6] https://fly.io
[7] https://github.com/supabase/realtime/tree/multiplayer
[8] https://github.com/supabase/supabase/tree/master/docker
