This is getting confusing. The tweets sound like you are concerned about write scalability, and here it sounds like you are concerned about read scalability?
> So we can get, say, 1000 updates, bundle them all, get it synced in say ~100ms, and then answer all 1000 requests at once, and still only take ~100ms.
I assume the same trick is applicable to RDBMS as well? So you batch the 1000 updates, and do one commit with a single fsync.
> Virtually every other database I've used is quite naive about how they flush blocks to disk, which dramatically reduces their effective transactions/sec. It's rare to see one that made all the right choices here.
Can you elaborate on this? Anyway RDBMS worth its salt should be able to saturate the disk IOPS, i.e. the act of flushing itself wouldn't be the bottleneck.
> Our database has very small amounts of data but a very, very large number of parallel readers.
So the control plane is the sole writer of this database, and there are maybe 100s/1000s of other readers, who each has a watcher on etcd? Who are these readers? If they are different processes on different machines, how did it work when the database was in the json file?
Sorry for the barrage of questions, but I have to ask out of curiosity.
> So we can get, say, 1000 updates, bundle them all, get it synced in say ~100ms, and then answer all 1000 requests at once, and still only take ~100ms.
I assume the same trick is applicable to RDBMS as well? So you batch the 1000 updates, and do one commit with a single fsync.
> Virtually every other database I've used is quite naive about how they flush blocks to disk, which dramatically reduces their effective transactions/sec. It's rare to see one that made all the right choices here.
Can you elaborate on this? Anyway RDBMS worth its salt should be able to saturate the disk IOPS, i.e. the act of flushing itself wouldn't be the bottleneck.
> Our database has very small amounts of data but a very, very large number of parallel readers.
So the control plane is the sole writer of this database, and there are maybe 100s/1000s of other readers, who each has a watcher on etcd? Who are these readers? If they are different processes on different machines, how did it work when the database was in the json file?
Sorry for the barrage of questions, but I have to ask out of curiosity.