A few key problems in Redis persistence

[wglb]

http://antirez.com/post/a-few-key-problems-in-redis-persistence.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+antirez+%28antirez+weblog%29

[fizx]

I guess you're in the right spot when everyone else is writing the love articles and you have to write your own hate. :)

[jules]

Could you memory map your memory to disk and let the kernel take care of it?

[reitzensteinm → jules]

The reasons for not doing that are discussed here:

http://antirez.com/post/redis-virtual-memory-story.html

[jules → reitzensteinm]

So basically:

1. Redis in memory data structures have poor locality, which hurts a lot when a "cache miss" means going to disk.

2. Redis in memory data structures take up to 10x as much space as the on disk structures, so writing them to disk verbatim wastes a lot of bandwidth.

So apparently using memory mapped files wouldn't work for Redis. But what if you used data structures that do have good locality and are optimized for space usage? Has this been tried? I think it is an interesting approach and has the potential to outperform Redis' approach, especially for bigger databases. If your data structures use say 2x as much space as Redis' on disk structures, then you would still be able to fit much more data in memory than Redis because you don't lose half your memory as described in the OP, plus the data structures are more compact.