I'd love to know how Apple distributes its CloudKit data across all of (what I assume are) its dozens of globally distributed FoundationDB clusters.
As of version 6.0, a single FoundationDB cluster currently only supports one active, writable 'master' region + one hot failover region. Transactions are all essentially LAN so performance characteristics are great. It doesn't support the slower TiDB/CRDB style active/active configuration where all nodes are writable, data is globally distributed and transactions can span across all nodes in the cluster.
Do they have some kind of scheme whereby ACID is preserved for local transactions only, while far away regions are asynchronously replicated to with less guarantees on data consistency?
Maximum fdb cluster size is actually not that large, in host count terms, and the maximum practical size per host is similarly not large for the Apple scale, so they are almost certainly sharding objects across hundreds or thousands of clusters.
I suspect it’s because they’re not just running a single cluster, but many many clusters with customers distributed globally. They had a talk about how they do it with other systems some time ago that I can’t find at the moment.
As of version 6.0, a single FoundationDB cluster currently only supports one active, writable 'master' region + one hot failover region. Transactions are all essentially LAN so performance characteristics are great. It doesn't support the slower TiDB/CRDB style active/active configuration where all nodes are writable, data is globally distributed and transactions can span across all nodes in the cluster.
Do they have some kind of scheme whereby ACID is preserved for local transactions only, while far away regions are asynchronously replicated to with less guarantees on data consistency?