Even if NoSQL solutions are new, there are no data durability concerns in my opinion.
I can talk for Redis but my guess is that's the same for the other datastores. In Redis to have full durability enable append-only-file: it's a text-only file, you can check by hand what's written inside your database.
In addition even while running in append only file mode, you can still ask Redis to "BGSAVE" (crete a copy of the dataset into an .rdb file in background). It's a single file and you can save it everywhere.
With the append only file, it's trivial to write a script to export ALL YOUR DATA into MySQL every 24 hours for instance. So you have your data in:
a) compact .rdb file you can backup easily.
b) the append only file
c) MySQL, where you can perform query to check if the data is sane.
In addition in the next months Redis will get a JSON export feature built-in, so you can also d) save data into a JSON file.
This new products are still younger than MySQL, and there are more bugs probably, but it's just a matter of being a bit more careful to use this DBs in production without concerns, experiencing a huge performance gain.
p.s. since the release of Redis 1.0 no critical bugs were found.
Actually, we do run redis in production, and it has been awesome for us. But, for this particular aspect of our application, we see about 20GB of data growth per week. So, an all-in-RAM db isn't really practical.
Either way, durability wasn't the issue for us. Mongodb has no write concurrency. So, when it got so overloaded with data that queries were taking forever, we had no way to purge older data without blocking all read & write operations for hours.
Ok sorry for assuming that the problem was durability / reliability.
If MongoDB has no write concurrency MySQL is simply better from this point of view for your use case as far as I understand, but I'm not very good at MongoDB.
Btw Redis will get Virtual Memory in a few months, I hope you can reconsider it later ;)
This reminds me of Elephant, a CL OODBMS that can use Postgres as a key/value store backend. OODBs, document stores, et al. are useful but using a complete RDBMS as a backend is both overkill and inefficient in my opinion.
Looks great! I remember seeing the article about FriendFeed's datbase setup and thinking how neat it was. Very nice to see a simple implementation of that idea become easily available.
Now I just have to find a project where I'll need to use it!
how is the cpu trade-off like in this type of setting? one more point is that i dont know ruby and i'm curious how can he add an index from that column?
Actually, friendly definitely "beat me to market", but I did start building this on top of sqlite3 a few weeks ago. My purpose was just for play--I was going to naively compare its performance to something like Tokyo Cabinet to see if matched at all. I can throw the code up somewhere, but it's neither complete, nor well thought out, merely a hack.
I can talk for Redis but my guess is that's the same for the other datastores. In Redis to have full durability enable append-only-file: it's a text-only file, you can check by hand what's written inside your database.
In addition even while running in append only file mode, you can still ask Redis to "BGSAVE" (crete a copy of the dataset into an .rdb file in background). It's a single file and you can save it everywhere.
With the append only file, it's trivial to write a script to export ALL YOUR DATA into MySQL every 24 hours for instance. So you have your data in:
a) compact .rdb file you can backup easily. b) the append only file c) MySQL, where you can perform query to check if the data is sane.
In addition in the next months Redis will get a JSON export feature built-in, so you can also d) save data into a JSON file.
This new products are still younger than MySQL, and there are more bugs probably, but it's just a matter of being a bit more careful to use this DBs in production without concerns, experiencing a huge performance gain.
p.s. since the release of Redis 1.0 no critical bugs were found.