The outbound dispatcher here seems like a prime candidate to write in a different language (Go for instance). Presumably it just makes HTTP requests so can be very simple -- building the request payload for instance could still happen in Python. And as you use Celery you already have a message queue over which it could communicate with the rest of your application.
There is not need to overdo it. Asyncio + threads will do nicely. 500 req/seq is not the end of the world. I have similar load on purely threaded systems and they work fine.
Crossbar.io would help though, but is not even necessary.
We have considered this actually, and I think it will be something we look into more as our scale increases. As you say, we already have the communication infrastructure in place via Celery/RabbitMQ.
I think that is what they are doing with celery. Each server has 92 celery workers. Each worker runs in its own process. Each worker takes tasks off of a queue and completes them.
As a complete tangeant, I find that Zapier's UI is what holds back my usage of it.
They added folders which is fantastic.
But editing a Zap is slow in the UI. I can't open a Zap into a new tab by Ctrl+Clicking the Zap title. Lots of Zap management is checking one zap's setup, and adding that configuration in turn to another zap.
When editing an existing zap (in Firefox anyway), and if there is a filter selected, and if it has a long name, then I can't see the full filter's name. (Sorry, I'm not logged in right now, so I can't see the UI's exact wording).
For example, if the source trigger is Sendowl. Add a filter to the zap, like product name. The name of the filter is longer than the drop-down list. When coming back to edit the zap, I can't see what trigger is selected.
After editing a zap in Firefox, and I click on the save button at the bottom of the page, something nothing happens. It's like it only gets focus with my first click. So I have to click a second time for the save button to go and save the zap.
Sorry for this type of report that's probably difficult to reproduce.
If I create a zap in a folder, I'd normally expect to be brought back to that folder, not to see the home screen again where folders are listed. I want to be able to visually see my new zap listed.
We haven't evaluated RQ specifically, since RabbitMQ is a core part of our infrastructure and we're really happy with it. We have experienced the pain of managing Celery, and considered alternatives a few times though! A post about our experiences there is on the list to write! :)
Ahh... For being on Rabbitmq, I suppose there are few other options. Without going into a blog post, why not Redis? Other than fairly sophisticated message routing... Any other reason?
I think he was referring to global variables inside the 3rd party libraries rather than their codebase. If a library is not thread-safe they cannot use it.
