Most distributed newSQL databases are an SQL layer on top of a distributed KV store. What they try to do is hide the distributed reality of their database so it acts like a regular database from the client side. Of course there are always caveats that might not be completely obvious but can cause terrible performance.
We take the opposite approach. We make the user aware of the distributed nature and force the user to use the distributed database like a distributed database should be used. You must split your data into chunks (actors) and you have a full raft replicated SQL engine (SQLite) within that chunk.
ActorDB has a very different design. I've never used it (I've been meaning to, but I don't have a use case yet), so my information might be a little bit off, but:
Basially, ActorDB doesn't hide the fact that it's partitioned. Rather, it forces the client to deal with partitions (or "actors") at the application level.
In particular, this means that while you can have transactions spanning multiple actors, queries can't. So you can't do joins across actors, and if you want to select from multiple actors in one network roundtrip, you have to do a special kind of looping statement that first finds the actors to operate on, then executes a statement on each.
You can work with multiple actors at once, but the database doesn't pretend that it's a single database; rather, each actor is sort of like a separate database, and it's up to you to design the data model and the SQL statements to distribute the data in an optimal way.
In those cases where you do need joins or aggregations that span multiple actors, you'll have to jump through some hoops. Joins, in particular, are probably not going to be very efficient. You might precompute some data that other databases would figure out on the fly. Again, since I've not used it, I don't know all the ways you would work around such limitations.
Unlike ActorDB, TiDB and CockroachDB present the illusion of a single database, on top of a distributed key/value store. There's a magical execution engine that takes selects, even with joins, and automatically splits the query plan into multiple parallel requests to the shards that hold the data, and then merges the results back together. You can do "select * from sometable" and it will return your table in one piece, no matter how it's distributed physically.
There are certainly benefits and drawbacks to both approaches.
Many distributed databases supporting linearizability fail to provide consistent backups.
MongoDB's docs: "To capture a point-in-time backup from a sharded cluster you must stop all writes to the cluster"
Cassandra's docs: "To take a global snapshot, run the nodetool snapshot command using a parallel ssh utility ... This provides an eventually consistent backup. Although no one node is guaranteed to be consistent with its replica nodes at the time a snapshot is taken"
Riak's docs: "backups can become slightly inconsistent from node to node"
CockroachDB's docs: "The table data is dumped as it appears at the time that the command is started ... there is no guarantee that NOW() is monotonic in transaction order"
Does TiDB support consistent backups? Are there any docs covering how it is implemented?
Yes, it does. Thanks to MVCC, TiDB supports the repeatable read isolation level which guarantees that any data read cannot change, if the transaction reads the same data again, it will find the previously read data in place, unchanged, and available to read.
So we can use any MySQL tools such as mysqldump or mydumper to backup the database consistently.
There are lots of other differences. The default distributed storage engine of TiDB is TiKV, and TiKV is written is Rust. The transaction mode is different, and so on. The are something similar are they are both NewSQL, both use Raft to replicate data.
Any idea why the same people appear to have used Rust for their KV store but Go for their SQL frontend to it? They are both great languages, I'm just surprised to see one team with major projects spread across the two.
Go is good at concurrent concurrency and has great development efficiency. We could easily develop complex SQL logical and parallelize many operators. But its GC and cgo overhead is not suitable for developing storage engine. So we choose Rust to develop TiKV.
Isn't the overhead of calling Rust from Go quite high?
Also, I'd assume that Rust would be better at expressing and pattern-matching against the sort of complex execution plan trees and query expressions you need for this sort of thing.
I was looking at Apache Spark the other day, which is written in Scala, and which has a planner that goes SQL -> AST -> logical plan -> physical plan. The planner/optimizer relies extensively on pattern matching to apply rules to the logical plan (pushdown and so on), and it manages a lot of this with "match" statements and not a lot of recursion.
But that stuff is murder in Go, which doesn't have pattern matching, or generics for that matter. I know it's bad because I'm in the middle of something similar right now, in Go. The Spark code is exactly how I wanted to organize it (minus the awful class inheritance they do), but that's not possible in Go.
Go code communicate with Rust code through network. So there is no overhead.
Go has interface, which could be used for expressing the AST, plan tree. We define some interface for AST node and plan tree node. It is convenient to apply some rules (predict-pushdown/column-pruning/cost-computing) on the tree.
You could refer to the code in https://github.com/pingcap/tidb/blob/master/plan/plan.go
We use RocksDB as the single-machine storage engine and build TiKV above RocksDB as a distributed storage engine. At first we consider use Go to build TiKV, but the cgo overhead between TiKV and Rocksdb is considerable. So we turn to Rust.
The network communication overhead between TiDB and TiKV has nothing to do with cgo.
Actually we are using RPC to call Rust from go, and it has nothing to do with Go or Rust, because either way we have to do RPC call(by encoding and decoding data), currently we depend on protocol buffer and customized RPC , and we are trying to migrate to gPRC. We know Go doesn't have pattern matching and generics, and it's kind of pain sometimes. But still we like Go very much because of simplicity and concurrency.
Yes, Go has higher development efficiency than Rust, especially for new comers. For a storage engine, the most important thing is stable. We focus on write right and stable codes. Rust's compiler is quite strict. Which is good for write right codes. We have many good programmers who is very good at writing Rust. So we do not suffer from the low development efficiency.
About the libraries, we could get most of we need. But some were missing when we begin to develop TiKV (GRPC/HTTP2 for example). But thanks to Rust Team/Community, they give us great supports. GRPC for Rust is available now. We also contribute back to the community. We develop prometheus client for Rust. The Rust community is active and helpful. So we do not worry about the lack of libraries.
TiDB's transaction model is different from Cockroach, and it's highly layered, you can use tikv(the underlying storage layer of tidb) as a key-value database without sql for better performance.
Likely because of it's nature as a network first language.
Go is largely speaking a language designed to push bytes over sockets to lots of concurrent clients.
There are two more things: 1. Go has high development efficiency. So we could build complex SQL logic easily. 2. Go is easy to write concurrent code. So we could enjoy the benefit of multi-core cpu. For example, we could use multiple goroutine to scan data, do join, do aggregation.
It is missing NewSQL, which is happening right now, but it shows how we are going in circles. There are benefits of NoSQL, particularly for type of data that it is ok once in a while to have individual values wrong or missing (tracking users, shopping carts etc). CRDTs are also useful. I'm wondering what NewSQL would bring, but I'm thinking that we will go back to traditional relational databases once again. We probably end up with some kind of hybrid approach, and decide for given piece of data whether it should be distributed (at the cost of consistency) or vice versa.
