The Elixir ecosystem is growing incredibly well. I am impressed both from an engineering and a product perspective.
It shows that they are playing the long game.
I think they are achieving what I wished happen to the Clojure ecosystem: productive, well designed, respected and popular. Clojure missed the last step, unfortunately.
It seems making a Lisp popular is an impossible task.
That said, I'm not sure if the data agrees with you. I think Clojure is more popular and widely used at this time. Not sure, but I think from what I remember of the few rankings, and just the fact I don't know an equivalent success story to NuBank for Elixir, I think maybe Clojure is at the moment more popular in practice.
But with the amazing learning material Elixir is putting out, maybe it won't last.
In terms of major companies using Elixir, Discord and Whatsapp are built with it. Whatsapp is slowly chipping away at its Elixir for infra homogeneity reasons with the rest of facebook but discord is still all aboard the Elixir train.
Neat, I didn't know about Discord. Whatsapp I thought it was all Erlang, are you sure about Elixir being prevalent there?
I wonder if it's only a few teams handling major events at Discord, or if that's truly their default backend service language all their backend teams uses. If so, maybe I should look at their job listings :p
I would think that NuBank would want to market and promote Clojure to ensure that there would be ample numbers of Clojure developers (since NuBank depends on a lot of Clojure code). Perhaps they do market Clojure in Brazil, but I don't see it elsewhere...
With 213 million people, they do have a ton of devs and a ton of ESL speakers. There’s lots of places like that, like Turkey or SE Asia, where word doesn’t reach NA/wEU.
I've wondered whether it's easier to add data analyst stuff to Elixir that Python seems to have, or add features to Python that Erlang (and by extension Elixir) provides out of the box.
By what I can see, if you want multiprocessing on Python in an easier way (let's say running async), you have to use something like ray core[0], then if you want multiple machines you need redis(?). Elixir/Erlang supports this out of the box.
Explorer[1] is an interesting approach, where it uses Rust via Rustler (Elixir library to call Rust code) and uses Polars as its dataframe library. I think Rustler needs to be reworked for this usecase, as it can be slow to return data. I made initial improvements which drastically improves encoding (https://github.com/elixir-nx/explorer/pull/282 and https://github.com/elixir-nx/explorer/pull/286, tldr 20+ seconds down to 3).
Hm, I think I've read somewhere that both Go and Elixir and kinda cooperative. A process in Elixir can yield control after certain number of reductions (function calls I think) and in Go a goroutine can only yield control on function calls, so if you have an infinite loop just adding numbers it will run uninterrupted. Both of them are "less cooperative" than Python with explicit yield statement. Do I get this right? I started digging into concurrency not that long ago.
Yes technically they are not fully preemptive in the sense that an os thread is (the os sends an interrupt which halts the processing at the CPU level), but in both go and elixir the programmer has no control over when the context switching happens, and "function calls" which are the yield boundaries happen all over the place, so it's "effectively preemptive".
Elixir is in practice more preemptive than go (last I checked with go) because you cannot infinitely loop and lock cpu in elixir -- a loop requires you to tail-call in elixir, so that's a yield boundary, and I'm certain that in earlier go that wasn't the case if you `while true {}`
Yes, unless you're in a BIF or NIF every call is a reduction: https://blog.stenmans.org/theBeamBook/#_reductions so if you call out to native code or are using a VM builtin you can lock it up indefinitely but other than that it's not possible.
[I have no experience with Elixir.] Would that be possible to have elixir intelligently manage multiprocessing of python scripts? I would be especially interested in being able to have the scripts talk to each other, but have no idea how it could work, besides perhaps having them communicate by writing to / reading from the same files, which seems risky.
I think you can have the scripts talk to each other through Elixir via the BEAM/OTP approach: for each script you will have a GenServer module that manages it in a separate process. Call this module MyApp.ScriptServer and put "use GenServer" at the top of the file. This GenServer process will have a client API, which you can write, which will be how you interact with your script. In the BEAM programming model, processes communicate by sending messages to each other's mailbox, so you set up some logic for this and maybe have another module called MyApp.ScriptCoordinationServer that manages some global state.
This is just my amateur guess -- I'd be happy to hear an Elixir/BEAM/OTP expert chime in.
Oddly, it seems to me that Clojure (the established people in its community) don't really care if anyone is attracted to it.
It seems like the people who come to Clojure do so because they caught a glimpse of it somewhere, did some digging and searching, and decided to give it a try. Not exactly the same as Sun spending $500 million to market Java.
That's not to suggest the Clojure community is unwelcoming or silent, but they generally fly under the radar.
Several years ago I went to a few of the Amsterdam Clojure meetup group meetings, and at that time there were 10 to 20 people there.
Meanwhile, the Elixir meetup would have 20-40! And Elixir was still comparatively young. Of course the Ruby meetup group was even larger, but that's no surprise given the prevalence of Rails in business.
Elixir also opens you up to the entire erlang and Beam ecosystems. Not as common as Java, I know, but still decades of knowledge and tooling around it cutting across many industries.
Seems like Beam is a more specialized tool than JVM but if you need that sort of functionality it's phenomenal. With things like built-in performant persistent key value stores and cluster management I'm kind of surprised more people haven't pushed through the "this is useful but lacks broad adoption" barrier.
Well… yes and no. A lot of things built in are pretty spartan and not always appropriate (Mnesia …cough…).
This said it’s the only environment I’m aware of that is built around the idea of cluster, not just a local process that may do some rpc to other processes.
You can find libraries for many things but they are not the same quality and breadth as the big Java ones.
Seemingly reliable for a specific type of problem, though. Most reliable software doesn't require a distributed network to be reliable, and I don't think binary execution on BEAM is any more inherently reliable than the JVM, right?
The BEAM is designed for a software architecture approach called OTP in the Erlang community -- it is designed around trees of process supervision. So there is an entire robust framework for reliability really built in as a primary concern for the BEAM.
If you need high concurrency, heavy traffic, real-time system with high resilience, there’s probably nothing out there that comes even close to BEAM. It is definitely possible to write such a system in Java or Rust, but most likely at huge cost for something that you can get out of the box elsewhere. BEAM has narrower scope, but in that scope it’s probably far ahead of anything else.
I tend think the opposite. Developing for Android I found the Java build process and maintaining a solid development environment a headache. If that follows on to Clojure I can see why it wouldn’t be as popular even if the language itself is overall better.
My experience would make me agree with you. Developer experience around tooling in Clojure makes a lot of assumption around prior knowledge of Java ecosystem.
Elixir, on the other hand, welcomes the newcomer better.
Except using Java ecosystem as-is would not be idiomatic in Clojure, which means you need an ecosystem of wrapper libraries. I recall someone making similar argument on some other post, on how it adds more space for bugs, performance issues, and still making it hard to work with. The context was in comparison to Kotlin, where using Java ecosystem is much nicer experience.
They discussed it, but no one has taken it on yet. I think Wojtek Mach(the desktop initial author) is a mac user, and they actively wanted to provide a way for windows users to easily try it.
We don't plan to add Linux support ourselves at the moment but we'd love if someone contributed that. As was mentioned elsewhere it might be challenging to provide a bundle that works across different distributions so investigating the possible options would be an important first step.
An AppImage or Flatpak package would make a great starting point that works across distributions. Making Livebook more accessible to Linux developers would improve its adoption, as well as the adoption of Elixir, Phoenix, and Fly.io.
Props to the Elixir community, it seems for some reason they've attracted a lot of people who have the will, the free time and the talent to provide amazing educational content. I've never seen a language of this size have so much quality learning material and tooling.
Really appreciate the kind words and happy to hear that you are enjoying the book! Hugo Baraúna and I have really enjoyed putting this book together and feel that Livebook has become pivotal in explaining certain topics and really understanding how things work under the hood. I wrote a blog post about some of my thoughts around Livebook as a learning tool on the DashBit blog: https://dashbit.co/blog/livebook-as-an-educational-tool
I love the concept of local Web servers as GUI substitutes, and hope we get many more of those. Tachidesk is my favorite example. Thanks to Phoenix Liveview, Elixir is becoming a great option in this space. This method gives us every benefit of Electron and almost none of the downsides.
Or just bookmark them. They usually have the program name in the page title, on firefox just begin with * on the url bar, or press B if you have Vimium.
I often need to inspect/transform some JSON/CSV/text/whatever data, move around/organize a bunch files, and similar things. I find that notebooks work great for that as you don't need to set up an environment, you can easily collect and rerun your scripts if you do need them again, and the output options are great for debugging and looking at data.
I know this is a Livebook love-fest, but you may also consider setting up org-mode sometime. There's a lot to love about org-mode but to stay on subject, you can write your documents in a plain text syntax akin to markdown and any of the code blocks that you add in as examples are _executable_. And you can take the output from one and pipe to another.
At work, for example, we have to hit a remote API and the secrets are stored in AWS secrets manager. I have some documentation that has a block describing how to get the secrets and some other documentation elsewhere describing how to issue a post request and I can pipe the output from one block into another and receive the output either in the same file or put it somewhere else.
The one downside with this extreme flexibility is I sometimes catch myself squirreling away useful functionality in my own notes instead of adding it in a utility CLI or similar where it can be shared with the team.
If having access to a scripting language is duct tape, piping output from one script to another in this way in a single file is like having inter-dimensional duct tape.
This idea is incredible, and extensible beyond these notebooks. I’d like to see something like this as a general code authoring/collaboration tool, would be nice to see when I’m working in the same code as a co-worker, or supercharged pair programming. Obviously scaled too large this would be chaos, but on a 6 person team I think this would be killer.
I, too, am interested but I'm not terribly literate in elixir yet. I am aware that you can compile executables in elixir with a release command but I'd be curious to hear more about (or at least be pointed to the specific spots in the code) how the bundling was done to give you a discrete application per platform, with app icon and everything.
Having a script that opens a web browser pages is simple enough, but the extra sugar on top to wrap in an app bundle that shows up in your task bar or dock is something I am very curious about.
EDIT: Someone further down the thread linked to a more specific place and that led me to the macos.ex file[1] (which is what I was most interested in, being on macOS myself.
It looks like at least in the case of macOS it will build the bundle by hand, template in enough swift code to build the application, sign and notarize a bundle, and add the compiled elixir program in there. Pretty interesting. I guess the idea is that you write just enough swift to manage the life cycle of the compiled elixir application. It's an interesting approach but it seems like it would be frustrating and fragile.
Right, so our goal was to allow configuring the bundle with app icons etc and importantly custom URL schemes and file types [1]. And then macOS- [2] and Windows- [3] specific launchers capture the "open file" and "open url" events and trigger the Elixir side of things. Another reason for native launchers is if for any reason we couldn't start the Elixir side, we want to let the user know that something went wrong and point to logs. (We have that on Mac and will have on Windows very soon.)
I know linux is a future supported platform but as far as I'm aware, there's not a clear unified way to build bundles in this way across linux distributions. What kind of things are you folks considering if you don't mind my curiosity.
It would be crazy awesome if someone could figure out how to layer python underneath this. There is a very nice feature set in livebook that doesn't really exist in jupyter.
As others pointed out, the app uses wxWidgets for the UI and Elixir for backend, along with Phoenix. I think similar apps can be created using wxWidgets with Python by replacing Phoenix with another framework, maybe Flask or Django.
Yeah you could but it hasn't been done. I've struggled with jupyter/ipython for a decade. CRDTs like automerge and OTs exist, but they're not in jupyter.
It's simply put, not that easy to manage distributed live state in a sane fashion with python (see openstack), the programming model of the underlying language puts obstacles in your way. Meanwhile Erlang was created by a physicist who understood that the speed of light is a thing and consistency of data across distances is a real issue.
Any tips why would you want to learn Elixir or what would you use it for?
I tried to learn it a couple of years ago. Bought a book and couldn't shake the impression that it was created to be complex and hard to reason about or maybe I am wrong and it is "easy", but has a steep learning curve?
That being said, I looked at Go around the same time and I clicked instantly. Easy to read, simple and yet powerful.
It shines at creating highly concurrent applications with a great deal of built in features to provide reliability and resilience. Sounds a lot like a server, and that's what much of the modern use for Elixir is. In particular, one area where it shines is stateful, long-lived requests (e.g, websockets) where many other concurrency models will either rely much more on databases to track state or potentially get choked up.
If nothing else it's neat to learn as one of the few somewhat well used languages in modern times that actually adheres to Alan Kay's original conception of OOP.
As someone who also doesn't know Jupyter, what sort of fun and useful things would I do with this? It looks like I can add content and some code blocks that I can execute. I guess taking notes and tinkering?
I'm messing with Elixir just for fun, so this caught my eye.
> not sure what the livebook is--the landing page is not sufficiently clear or coherent.
The difference between features and benefits. It's good at telling you the features (what it does) but not so much the benefits (why you would use it, how it can help you).
A common issue, but more understandable in this case than most as your average visitor to the Livebook site will likely know they have a need for it already, rather than end up there based on a 'random' link (like this thread).
I think this is a fair analogy, but maybe I'd add that LiveBooks uses MarkDown markupfor the non-code part of the notebook, make it far more readable, and with more visualizations coming in.
For example, recently an addition to add automatically generated sequence diagrams to Livebook Kino (https://github.com/livebook-dev/kino/pull/165). So generate code which shows a supervision tree, and automatically create a visual diagram of the code in one place. Pretty great for learning and explanatory material.
Related question: would people pay for this kind of localhost desktop app? I'd love to sell an elixir app without bothering Electron or Rust based engine.
Thanks for linking this. I was interested in how it worked but I didn't know enough elixir to get around the codebase by myself. This entry point is very informative for how they managed to do it.
I just looked at the macOS part, but the tl;dr for that is they seem to have compiled the elixir program into a binary and wrote enough swift to manage the lifecycle of the elixir program and signed it and notarized it and prepared it for shipment using CLI via shell! invocations. I didn't look at the windows way of doing things but I imagine it's similar.
I just installed it and it actually doesn't use Electron. It's basically just a local web server that runs on your computer, and opens a browser tab in your default browser pointing to http://localhost:60904/
Not quite, this is just your standard server side rendering for providing a convenient GUI alongside your erlang vm. There's no layering here, just bundling of things that are usually distributed separately (editor and vm).
It shows that they are playing the long game.
I think they are achieving what I wished happen to the Clojure ecosystem: productive, well designed, respected and popular. Clojure missed the last step, unfortunately.