The description of this new storage engine does not explain how it manages the durability of the data.
When you compare with the extreme efforts traditional databases take to ensure that unplugging a server will never ever result in data loss[0], silencing this problem makes me wonder.
Is it that at this ingest rate even trying to ensure durability is a vain effort?
Consider that a regular scrape has happened and that data has been accepted by the DB but not yet flushed to disk.
Whether the database dies just before or just after the scrape produces the same result: The data for that scrape isn't present when the server restarts.
There plenty of other ways a scrape might not succeed that we have no control over (e.g. other end is overloaded, network blip), so there's not much point obsessing over this particular failure mode.
> Is it that at this ingest rate even trying to ensure durability is a vain effort?
It's not in vain, but it'd be a bad engineering tradeoff in terms of throughput.
When you compare with the extreme efforts traditional databases take to ensure that unplugging a server will never ever result in data loss[0], silencing this problem makes me wonder.
Is it that at this ingest rate even trying to ensure durability is a vain effort?
[0] https://www.sqlite.org/atomiccommit.html