Rather than handling the multiprocessing and message passing yourself it would be much easier to use celery + gevent and let celery do the work of spinning up processes for executing the tasks.
It may feel limiting but my advice would be to keep all the celery job queue stuff isolated from your server, especially if you are using an async web framework. Have your web server just put all the jobs in the celery queue and let it handle executing them, regardless of whether they’re cpu or io-bound. If you try to optimize too much by doing something like leaning on celery for cpu-bound tasks but letting your web server handle the io-bound ones you’re going to be in for a world of hurt when it comes to both debugging and enforcing the order of execution. Celery has its warts but you’ll at least know where in the system your problem is and have reasonably good control over the pipeline.
Can Celery help handle state internal to the workers?
What about using Pkykka alongside Pyro such that each Pyro remote object is actually a Pykka actor? Such that Pyro allows splitting workers across separate Python OS processes and the "messaging" while Pykka handles the internal state.
Similar to how one can pass around state inside an OTP GenServer in Erlang.
In Python, it could be a class managing a session type of object, like a TCP socket connection, or managing some piece of hardware that is doing something independently of the other parts of the system, or a database writer, etc.
Oh sure, if I'm understanding you correctly. It's been a few years since I've wrenched on such code, but IIRC, you can pass any seralizable object you want to the function being executed. We used to pass around an in-memory class object, no problem. The tricky part was control flow, but that wasn't celery's problem.
It definitely isn't, but this is a new role with a rushed timeline in a place dominated by Python.
> and it works but I find the code extremely hard to reason about
This is also a major concern of mine.