Maybe I’m missing something but I would imagine query planning and caching is the hardest thing in terms of database memory usage. This post doesn’t seem to address that.
Even if the messages are fixed sized surely the memory cost of executing the queries depends on the complexity of the query, and the expected numbers of results returned.
TigerBeetle's storage engine is designed also for range queries, and we have some interesting ideas for our query engine in the works.
To be sure, there are some tricky things that crop up, such as pipeline blockers for queries. However, we do have limits on literally everything, so that makes it easier—static allocation is best when it's done viral.
We wanted to go into so much more detail on all this but there was only space for so much. Stay tuned.
Even if the messages are fixed sized surely the memory cost of executing the queries depends on the complexity of the query, and the expected numbers of results returned.
Are they just doing get/set style queries only?