All right, let me tell you the history of recur to explain this choice. :-)
I wrote the initial version in Python in 2023 and used simpleeval [1] for the condition expressions.
The readme for simpleeval states:
> I've done the best I can with this library - but there's no warranty, no guarantee, nada. A lot of very clever people think the whole idea of trying to sandbox CPython is impossible. Read the code yourself, and use it at your own risk.
In early 2024, simpleeval had a vulnerability report that drove the point home [2].
I wanted to switch to a safer expression library in the rare event someone passed untrusted arguments to `--condition` and as a matter of craft.
(I like simpleeval, but I think it should be used for expressions in trusted or semi-trusted environments.)
First, I evaluated cel-python [3].
I didn't adopt it over the binary dependency on PyYAML and concerns about maturity.
I also wished to avoid drastically changing the condition language.
Next in line was python-starlark-go [4], which I had only used in a project that ultimately didn't need complex config.
I had been interested in Starlark for a while.
It was an attractive alternative to conventional software configuration.
I saw an opportunity to really try it.
A switch to python-starlark-go would have made platform-independent zipapps I built with shiv [5] no longer an option.
This was when I realized I might as well port recur to Go, use starlark-go natively, and get static binaries out of it.
I could have gone with cel-go, but like I said, I was interested in Starlark and wanted to keep expressions similar to how they were with simpleeval.
Wow that had been quite a journey - thanks for the detailed response. We’ve been using cel internally in a golang codebase and been pretty happy with it. I’ve only know about starlark in the Bazel context - I’ve learned a couple of things from your post. Thanks :)
No, there is no pure-Python Starlark.
So far there are only Python bindings for the Go and the Rust implementation:
https://github.com/laurentlb/awesome-starlark#getting-starte....
I thought about porting the Go implementation to Python.
Doing it as a subgoal for porting recur seemed a little like scope creep.
(Tokei says there are 16792 SLOC in the latest d4d7611 commit of starlark-go and 899 in recur 67b38c1.)
I believe Starlark was renamed Skylark, even internally. Bazel is a build system that uses Starlark as a configuration language, not a fork of Starlark.
Feedback: everybody’s question is going to be on why this over temporal? I’ve noticed you answered a little bit of that below. My advice would be to write a detailed blog post maybe on how both the systems compare from installation to use cases and administration, etc - I’ve been following your blog and while I think y’all are doing interesting stuff I still haven’t wrapped my head around how exactly is restate different from temporal which is a lot more funded, has almost every unicorn using them and are fully permissively licensed.
That blog post should exist, agree. Here is an attempt at a short answer (with the caveat that I am not an expert in Temporal).
(1) Restate has latencies that to the best of my knowledge are not achievable with Temporal. Restate's latencies are low because of (a) its event-log architecture and (b) the fact that Restate doesn't need to spawn tasks for activities, but calls RPC handlers.
(2) Restate works really well with FaaS. FaaS needs essentially a "push event" model, which is exactly what Restate does (push event, call handler). IIRC, Temporal has a worker model that pulls tasks, and a pull model is not great for FaaS.
Restate + AWS Lambda is actually an amazing task queue that you can submit to super fast and that scales out its workers virtually infinitely automatically (Lambda).
(3) Restate is a self-contained single binary that you download and start and you are done. I think that is a vastly different experience from most systems out there, not just Temporal. Why do app developers love Redis so much, despite its debatable durability? I think it is the insanely lightweight manner they love, and this is what we want to replicate (with proper durability, though).
(4) Maybe most importantly, Restate does much more than workflows. You can use it for just workflows, but you can also implement services that communicate durably (exactly-one RPC), maintain state in an actor-style manner (via virtual objects), or ingest events from Kafka.
All execution and communication is async, durable, reliable. I think that kind of app would be very hard to build with Temporal, and if you build it, you'd probably be using some really weird quirks around signals, for example when building the state maintenance of the digital twin that don't make this something any other app developer would find really intuitive.
Thanks for the detailed answer - please turn it into a blog post! Excited to see competition and different architectural approaches to tackle durable execution. Wishing you all the very best!
100% agree! Bartosz Ciechanowski along with Amit Patel are national treasures. Their expository writing coupled with ChatGPT do dig into some of the more salient points has been a complete game charger for my self learning.
reply