> What's your procedure for adding new nodes to increase capacity? Would you have to take your redis cluster offline to redistribute data from all nodes over the new keyspace?
It's not easy, actually. The short answer is that we don't add capacity (because we've only needed to once, and we have tons of room to grow now). The long answer is that I have a switch I can flip that starts incrementing/adding data to a whole new cluster of Redis nodes while it still updates the old ones. We can then backfill all data to the new nodes and when they're setup, flip a switch to read/write only from/to the new nodes. It may sound a bit weird, mostly because it is. Moving sets of random keys from one node to the other while you're expecting live reads/writes is a huge pain, so I just punted on the problem.
> wonder if consistent hashing might be a bigger win in the long run.
I'm not sure that it's applicable. Consistent caching is really handy for caches when you don't want everything to miss as soon as 1/N servers drop out of the ring. You have to (imo) think of each Redis shard as a "real" DB. If your master PostgreSQL instance dies, you don't just start reading from another random instance and returning "None" for all of your queries. If a shard goes down, you either depend on an up-to-date read slave or nothing at all. I'm not sure how consistent hashing helps when adding nodes to a "real" DB, either. Say Node1 holds all of the data for CNN, you add a new node to the ring and now some % of CNN keys go to that new node. Now all of your writes are updating new/empty keys and all your reads and reading those new/empty keys. How does consistent hashing help with the migration?
(I'm really asking, because if I'm missing something I'd love to know.)
> I'm not sure how consistent hashing helps when adding nodes to a "real" DB, either ... How does consistent hashing help with the migration?
Instead of backfilling all data to an entirely new cluster, you'd only backfill the small amount of data from the keyspace "stolen" by the new node, and expire the keys at the original locations. If you use M replicas of each node around the ring (typically M << N) you only involve M+1 nodes in the migration process.
I'm still experimenting with this idea myself, and would also love to know if anyone's tried something similar with data store sharding (not just with caching).
> you'd only backfill the small amount of data from the keyspace "stolen" by the new node
I think this is the part I'm not so sure about.
Say I have 100 stats, and of course each stat is per forum, per day (going back from 1 day to ... 5 years?). How do I know what keys were just "stolen"? Do I have my new-node code hash every possible key (all stats for all forums for all hours for all time) to see which might go to that node? And then it reverses that key to know what it "means" to backfill it? (I need to do that followup post as the way our data 'flows' in is applicable here)
> How do I know what keys were just "stolen"? Do I have my new-node code hash every possible key (all stats for all forums for all hours for all time) to see which might go to that node? And then it reverses that key to know what it "means" to backfill it?
Right, you'd have to iterate through all zset elements on the existing node, applying the consistent hash function to decide whether or not the element will be stolen by the new node.
If the element itself doesn't contain user id (or whatever you shard on) all bets are off.
In a new post from antirez directly addressing your scenario, he suggests starting with a fixed but large number of shard instances on the same machine, then migrating the instances off to other machines using master / slave replication as needed.
It's not easy, actually. The short answer is that we don't add capacity (because we've only needed to once, and we have tons of room to grow now). The long answer is that I have a switch I can flip that starts incrementing/adding data to a whole new cluster of Redis nodes while it still updates the old ones. We can then backfill all data to the new nodes and when they're setup, flip a switch to read/write only from/to the new nodes. It may sound a bit weird, mostly because it is. Moving sets of random keys from one node to the other while you're expecting live reads/writes is a huge pain, so I just punted on the problem.
(I elaborated a little more in a comment on my post: http://bretthoerner.com/2011/2/21/redis-at-disqus/#comment-1...)
> wonder if consistent hashing might be a bigger win in the long run.
I'm not sure that it's applicable. Consistent caching is really handy for caches when you don't want everything to miss as soon as 1/N servers drop out of the ring. You have to (imo) think of each Redis shard as a "real" DB. If your master PostgreSQL instance dies, you don't just start reading from another random instance and returning "None" for all of your queries. If a shard goes down, you either depend on an up-to-date read slave or nothing at all. I'm not sure how consistent hashing helps when adding nodes to a "real" DB, either. Say Node1 holds all of the data for CNN, you add a new node to the ring and now some % of CNN keys go to that new node. Now all of your writes are updating new/empty keys and all your reads and reading those new/empty keys. How does consistent hashing help with the migration?
(I'm really asking, because if I'm missing something I'd love to know.)