Practical question here. How are Elixir/Erlang apps being deployed nowadays?
My understanding is that BEAM is designed for long running setups where hot reload of the code makes sense, and reboots are not happening that often thanks to the supervision trees.
But practically speaking how does it work under k8s env? When pod is being destroyed, what happens to all the messages sitting there?
Pushing state out of actors is a possibility, but isn’t it ruin the whole idea of stateful actors ?
Just to make it clear: you don't have to write stateful apps, just because you can. It's still possible to have the more traditional stateless model like Rails.
If you do write a stateful app, and run it in k8s, you can have state handover to a newly created pod. It works pretty well.
Well, let’s assume we are talking about a webshop. Every user is represented as an actor, a cart might be a state.
So cart as a state should be stored somewhere else, let’s say redis.
And every single read/write operation would be a network hop. What are the benefits of using actor model then ?
As for state handover between pods - what are the keywords to look for ?
What you describe sounds more like Orleans grains. Erlang's lightweight processes serve a different purpose than actors or objects, they're really about fault tolerance.
A typical Phoenix app spawns one process for every HTTP request, websocket channel, or DB connection, for example. All these things have a lifetime shorter or equal to the pod's.
The VM gives guarantees and tools to ensure one of these process crashing will not affect the others, unless you want it to. For instance, if a request crashes, its memory will be reclaimed, open files will be closed, DB connections will be re-opened, the exit reason will be logged, error metrics incremented, etc. This is possible with no defensive programming like try/catch and so on, because the processes are isolated. For example they don't share memory, so it's always safe to deallocate something if its owner process died.
This gives a two-layer safety net that allows for very reliable apps, where the VM protects you against bugs in your own code, and Kubernetes protects you against bugs in Erlang, or really catastrophic failures. There's a good blog post on this: http://blog.plataformatec.com.br/2019/10/kubernetes-and-the-...
What i had in mind is some kind of gen_fsm or gen_statemachine.
For stateless applications it is pretty obvious how supervision trees can improve the reliability. Essentially the only ‘state’ there is request itself. Worst case scenario client would just retry.
But with some state involved it becomes not as simple.
Essentially the answer to my question probably would be like : “you should store the state in the external system, and design your system in such a way that stored state is always consistent. In case of failure supervisor will respawn the process and it will recreate what it needs from the saved state”
> you should store the state in the external system
Disk works too, but if you're multi-node this means you now have a distributed database embedded in your system, which may or may not be your goal :)
RabbitMQ does this, they developed a library for "persistent, fault-tolerant and replicated state machines" based on Raft: https://github.com/rabbitmq/ra.
Ahh great. “On Erlang, State and Crashes” is exactly answering my question.
As for disks - in good old times when servers were pets, not cattle that was a good idea. But now when the servers are as ephemeral as actors, we need to approach it differently, hence my original question.
Sidenote - i have a strange relationship with Erlang. I first learned it in 2006, liked the idea and was hoping it will eat the world as scale increases. I even contacted Joe Armstrong in hope to translate his thesis. Zero Erlang books in the world at the time. Then i did some load tests using Tsung in 2012.
Then i used akka.net in 2018. But till this day i never had a chance to properly use in production.
> you should store the state in the external system
It's an old talk, but this one has a strategy for robustly persisting state across ephemeral containers. The example here is running a multiplayer game server while doing code updates etc. using Horde and CRDTs
Thank you, i was not aware of the Horde package. Definitely worth looking.
I was also thinking what should happen in case of ‘electricity disappears’ scenario, but came to a conclusion that it is good old ‘save the state you care about’
I think many modern applications aren't attempting the "long running, hot reload" model - they use a more conventional approach.
One thing that Elixir does well is handling faults - the relevant chunk of code 'crashes', and then restarts. The restart process does the appropriate recovery of any state. This turns all fault handling into a single code path (the startup path).
This approach also fits with the stateless style - the application should already be designed to start up properly and set up its state from scratch. Yes, in practice you design your application differently for a stateless deploy model, but Elixir helps you a lot in this model just as it does in the other one.
My understanding is that BEAM is designed for long running setups where hot reload of the code makes sense, and reboots are not happening that often thanks to the supervision trees.
But practically speaking how does it work under k8s env? When pod is being destroyed, what happens to all the messages sitting there? Pushing state out of actors is a possibility, but isn’t it ruin the whole idea of stateful actors ?