The architecture I eventually ended up with for my product (https://reflame.app) involves:
1. A strongly consistent globally replicated DB for most data that needs fast reads (<100ms) but not necessarily fast writes (>200ms). I've been using Fauna, but there are other options too such as CockroachDB and Spanner, and more in the works.
2. An eventually consistent globally replicated DB for the subset of data that does also need fast writes. I eventually settled on Dynamo for this, but there are even more options here.
I think for all but the most latency-sensitive products, 1. will be all they need. IMHO the strongly consistently replicated database is a strictly superior product compared to databases that are single-region by default and only support replication through read-replicas.
In a read-replica system, we have to account for stale reads due to replication delays, and redirect writes to the primary, resulting in inconsistent latencies across regions. This is an extremely expensive complexity tax that will significantly increase the cognitive load on every engineer, lead to a ton of bugs around stale reads, and cause edge case handling code to seep into every corner of our codebase.
Strongly consistently replicated databases on the other hand, offer the exact same mental model as a database that lives in a single region with a single source of truth, while offering consistent, fast, up-to-date reads everywhere, at the cost of consistently slower writes everywhere. I actually consider the consistently slower writes also a benefit since it doesn't allow us to fool ourselves into thinking our app is fast for everybody, when it's only fast for us because we placed the primary db right next to us, and forces us to actually solve for the higher write latency using other technologies if our use case truly requires it (see 2.).
In the super long term, I don't think the future is on what's currently referred to as "the edge", as this "edge" doesn't extend nearly far enough. The true edge is client devices: reading from and writing to client devices is the only way to truly eliminate speed-of-light induced latency.
For a long time, most truly client-first apps have been relegated to single-user experiences due to how most popular client-first architectures have not had an answer for collaboration and authorization, but with this new wave of client-first architectures solving for collaboration and authorization with client-side reads and optimistic client-side writes with server-side validation (see Replicache), I've never been more optimistic about the future (an open source alternative to Replicache would do wonders to accelerate us to this future. clientdb looks promising).
Good post. Do you have any resources describing the 'new wave of client-first architectures' you mention? I'm struggling to understand how you can do client-side authorization securely.
Replicache (https://replicache.dev/) and clientdb (https://clientdb.dev/) are the only productized versions of this architecture I'm aware of (please do let me know if anyone is aware of others!).
But the architecture itself has been used successfully in a bunch of apps, most notable of which is probably Linear (https://linear.app/docs/offline-mode, I remember watching an early video of their founder explaining the architecture in more detail but I can't seem to find it anymore (edit: found it! https://youtu.be/WxK11RsLqp4?t=2175)).
Basically the way authorization works is you define specific mutations that are supported (no arbitrary writes to client state, so write semantics are constrained for ease of authorization and conflict handling), with a client-side and server-side implementation for each mutation. The client side gets applied optimistically and then sync'ed and ran on the server eventually, which applies authorization rules and detects and handles conflicts, which can result in client state getting rolled back if authorization rules are violated or if unresolvable conflicts are present. Replicache has a good writeup here: https://doc.replicache.dev/how-it-works#the-big-picture
Replicache has a commercial license but is source-available. Clientdb is open-source, but doesn't seem as mature yet. I'd love to see more open source solutions in this space too.
There with you. We chose cockroach and fly to achieve our global needs in the simplest manner. But to use their cloud multiregion offer, it costs ~3400usd/mo at their recommended specs and 3 regions. Pricey depending on how you see it.
We hope their serverless tier meets feature parity soon.
Yep, Cockroach's dedicated offering was pretty cost prohibitive when I last looked too, and I really didn't want to have to operate my own globally replicated database, so Fauna seemed like the best option at the time.
Really looking forward to Cockroach's serverless options too. More competition in this space is very welcome.
Anyone used Yugabyte before for a globally replicated DB? I use Hasura a lot, which depends on Postgres, and Yugabyte seems like a possible drop in candidate for Postgres, but wondering if others are using it in prod.
How are you finding working with distributed databases? I've worked with Fauna and Dynamo and they are a nightmare from the DX and iteration speed point of view compared to Postgres.
Dynamo has definitely been a bit of a nightmare to work with, but I actually find Fauna reasonably pleasant. You're not going to get the vast ecosystem of SQL based tooling, but the query language itself is designed to be composable, which reduces the need for ORMs (though I do still miss the type generation from Prisma sometimes), and there's a pretty nice declarative schema migration tool that handles that aspect pretty well: https://github.com/fauna-labs/fauna-schema-migrate
Yeah, I mean, apart from missing intellisense, Fauna forces you to set in stone everything, which is not good for fast iteration and prototyping. I understand it's for performance, but things, especially in startups, continuously change.
1. A strongly consistent globally replicated DB for most data that needs fast reads (<100ms) but not necessarily fast writes (>200ms). I've been using Fauna, but there are other options too such as CockroachDB and Spanner, and more in the works.
2. An eventually consistent globally replicated DB for the subset of data that does also need fast writes. I eventually settled on Dynamo for this, but there are even more options here.
I think for all but the most latency-sensitive products, 1. will be all they need. IMHO the strongly consistently replicated database is a strictly superior product compared to databases that are single-region by default and only support replication through read-replicas.
In a read-replica system, we have to account for stale reads due to replication delays, and redirect writes to the primary, resulting in inconsistent latencies across regions. This is an extremely expensive complexity tax that will significantly increase the cognitive load on every engineer, lead to a ton of bugs around stale reads, and cause edge case handling code to seep into every corner of our codebase.
Strongly consistently replicated databases on the other hand, offer the exact same mental model as a database that lives in a single region with a single source of truth, while offering consistent, fast, up-to-date reads everywhere, at the cost of consistently slower writes everywhere. I actually consider the consistently slower writes also a benefit since it doesn't allow us to fool ourselves into thinking our app is fast for everybody, when it's only fast for us because we placed the primary db right next to us, and forces us to actually solve for the higher write latency using other technologies if our use case truly requires it (see 2.).
In the super long term, I don't think the future is on what's currently referred to as "the edge", as this "edge" doesn't extend nearly far enough. The true edge is client devices: reading from and writing to client devices is the only way to truly eliminate speed-of-light induced latency.
For a long time, most truly client-first apps have been relegated to single-user experiences due to how most popular client-first architectures have not had an answer for collaboration and authorization, but with this new wave of client-first architectures solving for collaboration and authorization with client-side reads and optimistic client-side writes with server-side validation (see Replicache), I've never been more optimistic about the future (an open source alternative to Replicache would do wonders to accelerate us to this future. clientdb looks promising).