Full disclosure, I'm heading up the backend development for Zipline Games Moai Cloud (http://getmoai.com/). We're targeting game developers so that lead us to some different use cases and language choices (like Lua).
Thanks for sharing this, at first read through it sounds a lot like the architecture mongrel2 provides (which we use). If you were to swap out the node.js dispatcher with mongrel2 and the redis queue with ZeroMQ.
Have you run into any issues using redis as a queue? If it were replicated across machines I wonder if you could have multiple workers dequeuing the request. If it's on a single server wouldn't the blocking operation on the list become a bottleneck?
Again, thanks for sharing, the offering looks great.
We actually tested a few designs using m2 and 0mq during our R&D phase. 0mq's push/pull sockets provide the same "take" behavior as the Redis RPUSH/BLPOP, so we definitely could have used m2 as the HTTP end of a similar architecture to what we use now.
One of the considerations that led to the choice of Redis was the transparency of the queueing and dequeueing. The messages in the queue are easily inspected, and the Redis MONITOR command helps greatly in debugging.
More important from a design perspective was our desire to hide the HTTP specifics from the workers. m2 pushes a JSON or tnetstring representation of the HTTP request to its workers, but we want the task we send to the workers to be generalized, stripped of information that is only meaningful for HTTP. We also want to classify the task by resource type and requested action, which allows us to use multiple queues. Multiple queues allows us to implement workers on an ad hoc basis.
M2 could work here if our request classification only depended on the URL. But that is a limitation we are not willing to accept. Request headers can be very useful in dispatching, especially those related to content negotiation (Accept, Content-Type, etc.)
There is an interesting hybrid approach using mongrel2: write one or more m2 handlers that perform the same function as our node.js dispatchers. I.e. m2 sends the JSON-formatted request to an m2 handler that deserializes it, removes the HTTP dressing, classifies the request according to type and action, then queues a task in the appropriate queue. A worker takes the task, does its own little thing, and sends the result to an m2 handler that knows how to re-clothe the result as an HTTP response and deliver it to the m2 cluster.
Regarding the question about queue behavior across replicated Redises:
I do not know for certain, but I do certainly hope it is not possible for an item in a Redis list to be popped by more than one client, no matter how the replication is configured.
With our architecture, we could relieve at least some of the strain on the task/result messaging system by using a cluster of Redis servers for the task queues. Each queue server might have its own cadre of workers listening for tasks. The return trip (getting a result from a worker back to the HTTP front end) is a little trickier, because it matters which HTTP server is holding open the HTTP connection. You could use PUB/SUB (which I believe is how m2 currently does it), or each HTTP server could be popping results from its own result queue.
When using a single Redis server, the only hard limitation we have seen with using the BLPOP operation is the number of client connections Redis can keep open. In case it's not clear, the BLPOP is blocking for the client, not the server.
The "queue" is merely a Redis list. Durable by default.
Redis PUB/SUB is not suited for the task queues we use, because any number of subscribers will receive the messages. We want to guarantee that only one worker will act upon each message.
Err, I guess I meant something else. For instance, what happens if one of the workers goes down after popping? The message is lost, right?
I think I invented your architecture in my head that isn't near reality. I imagined you were using the workers to take incoming published messages and pushing them into the queues that subscribed connections are popping off of. Effectively building your own fanout. So in this context, I was wondering why not just use pubsub which will handle fanout and get rid of the entire worker model.
Thanks for sharing this, at first read through it sounds a lot like the architecture mongrel2 provides (which we use). If you were to swap out the node.js dispatcher with mongrel2 and the redis queue with ZeroMQ.
Have you run into any issues using redis as a queue? If it were replicated across machines I wonder if you could have multiple workers dequeuing the request. If it's on a single server wouldn't the blocking operation on the list become a bottleneck?
Again, thanks for sharing, the offering looks great.