My company replaced the integration server I had originally written in Python/Django/Celery/RabbitMQ with one built in Elixir/OTP. The results were impressive.
Our task load was getting out of hand, and during peak load throwing memory and CPU at it was not helping. Task monitoring was also a pain. It needed a complete rewrite, because the original design, while robust for what it targeted at the time, could not keep up with our growth.
The integration developer I hired to take over the project looked into using Go at first, but then discovered Elixir. I started learning it with him (he quickly surpassed me), and after a few trial task implementations, I gave him the go-ahead. We couldn't be happier with the result. Thanks to OTP and the existing tools built around it, we have great monitoring, and the ability to easily and quickly scale up workers at peak times.
Correct. Not saying anything more than that. The BeamVM/OTP is a better fit for long-running high-concurrency fault-tollerant systems. That's what it was designed for.
I have not switched to Elixir/Phoenix for my other web development. I still use Django for that.
I had a simple rabbitmq -> elasticsearch logger I wrote in scala, which had to handle peaks of about 8000 messages/s on a single t2.large instance (with elasticsearch on the same box).
It took about a week and a half to write, and would have taken a bunch more time to fix to apply backpressure properly, so it suffered from getting OOMed. Even it's base memory usage was around 300MB. I rewrote it in elixir + genstage in about 8 hours, now it takes 60MB of ram for basically as many messages as I can throw at it, because 1) beam uses a lot less base memory 2) backpressure concerns are sorted out.
In scala, I would have to use akka streams or similar, and it would take me weeks and still I don't think I could get it to fit in such a small footprint.
I plan to add websockets to view a live stream, but I have a little trouble finding the right genstage patterns (as it's still new).
If I were doing high frequency numerical calculations, maybe scala would be a better fit, but for most high speed messaging elixir is pretty amazing.
It sounds like you hired a great engineer to be honest. Elixir, and the OTP environment is well designed. But it shouldn't downplay the achievement of the guy who re-implemented it (or yours, for letting him do so and provide him with the adequate mentorship). Props to you both, it's an impressive feat.
As someone who writes both, it's even difficult to compare these as they are apples and oranges.
Elixir (due to Erlang base) is great for long-running or distributed services and includes tooling to, for example, easily introspect the state/memory usage of your services down to a single mutable data structure, to heal from failures, and to hot-swap upgrade/code while the server is still running, and to perform node-to-node RPC.
Go would be a replacement for something that was previously written in C/C++, that may need concurrency, and that should compile to native code. You get very fine-grained control of everything from network to file operations and excellent concurrency constructs. A web crawler, a device driver, or writing an Nginx/HAProxy-type daemon, all may be great example use cases.
One could write their next social network/app backend using Go (just as one could do it in C), but it would probably take more work than just using a higher-level language.
Go is much more suited for web apps than any driver or real time performance application; although, I will agree that using Go for command-line tools is great. I haven't used Elixir but as somebody who uses Go, I highly recommend not replacing C/C++ with it unless it isn't performance critical.
PS: Go's memory usage is pretty up there some times, another reason you should probably stick with C/C++ in the context mentioned above.
I'm the wrong person to answer that, perhaps, but with OTP you get a task broker / monitor for free. With Go we would be back to piecing together a system using other tools such as AMPQ or similar. It also isn't as easy to write it for high-concurrency monitored tasks. But I have little experience with Go, so correct me if I am wrong.
Happy to see more and more non-trivial platform-based implementations in Elixir. We're using the Elixir/OTP+Phoenix stack for our own projects, handling networks of radio-connected sensors. It has been a breeze developing with Elixir, as not only the language is maturing, but the ecosystem is steadily growing as well - and there are a number of great libraries, from Ecto to Phoenix itself.
The OTP foundation of Erlang and hence Elixir just screams "platform" or at least "network", as it was designed to handle parallel flows of data. Great to see yet another project utilizing this aspect of the language.
Not sure what you mean by "parallel flows of data", but Erlang/Elixir isn't particularly good at handling multiple streams of data between distributed nodes if you're relying on the built-in distribution subsystem (net_kernel).
Only a single TCP connection is opened between any two nodes and there's a head-of-line blocking issue with message delivery (intentionally for backward-compatibility reasons).
Certainly you can build your own multi-connection subsystem if you want to, but that's true of almost anything. I've had to do this for Erlang several times.
Instead of rolling your own connection subsystem, there is also this https://github.com/lasp-lang/partisan Although they call it still experimental, it seems to be pretty actively developed.
I write a lot of server processes that communicate with remote devices.
Elixir has become my go-to language for whipping up new communication utilities. The syntax is comfortable, although I had to get used to a Functional Programming style at first.
What I like most is the language and runtime's features that make parallelization so easy. Spawning threads that are monitored, have their own protected state, that are managed if they crash, and are trivially spread across different machines on the network made it straightforward to write fairly highly available code right from the beginning without a lot of planning.
I came to Elixir from Scala, and for me the dynamic typing is a big step forward, because now I don't have to deal with the half of the Scala community that tries to make Scala into Haskell.
Between Dialyzer (a static analyzer that finds type errors) and runtime pattern-matching (extremely powerful destructuring, sort of borrowed from Prolog initially), I don't find myself wanting to go back to strong typing on the JVM.
I'm currently working with Scala and recently dabbling with Elixir and Phoenix. It's a good language and great platform, however, I still prefer static typing.
I'm still getting used to dynamic typing in Elixir. Most of the time I feel like just matching against Map data structure or records.
While Dialyzer is great, writing the typespecs is a bit of a maintenance overhead. I suppose it's a tradeoff...
> I came to Elixir from Scala, [..] now I don't have to deal with the half of the Scala community that tries to make Scala into Haskell.
IMO Scala will continue to be a multi-paradigm lang. On pure-FP and libs, it's a choice and really depends on the problem domain. It might not be a good solution to everyone, but I'm glad it exists.
I think of Elixir typespecs as documentation more than as overhead. Given that I'm writing comments that tell the shape of the inputs and outputs of every function anyhow, I might as well do it in a way that lends itself to static analysis. :)
Oh, screw side-effects -- Elixir is much more hardline about immutability than Scala. I'm talking about jamming half your program logic into the type system, then braying to anyone who stays still long enough about how that's the right way to do things.
It has a type system, just a differently structured one to support its goals. In a naturally distributed language there are a number of headaches that are created by overly strict typing (think REST vs WSDL as an example). It's an entirely different way of thinking about the problem that enables you to accomplish things that you can't effectively do outside of the BEAM.
Because literally five seconds of googling would have gotten you the answer. Your comment didn't add anything for anyone but yourself. That's why you got downvoted.
Elixir is not a framework around Erlang. It's a language running on the Erlang VM (BEAM) with painless erlang interoperability. It's the most popular alternative language to erlang on BEAM.
As an avid Elixir user, your comment got me thinking about how and where I look for help. I realized that it wouldn't even really occur to me to use Stack Overflow for answers regarding Elixir, as the other platforms you mentioned tend to be so active and well suited for that purpose.
This isn't to say that I wouldn't use SO for other purposes, as I actively visit the site for JS/PHP content. Anecdotal evidence to support your insight by all means, of course.
Agree with this. I've had a couple of questions on Elixir & while I use SO for looking up answers to other languages, for some reason it never even occurred to me to ask Elixir questions there. Possibly because the folks on Elixir Forum & the Slack channel are incredibly welcoming. They also have areas specifically for those questions you know/feel are very beginner noobish levels.
Discourse is garbage but that's an ancillary point I think. Developers are on StackOverflow. If you have a C#, C++, JavaScript, Java, Python, Ruby, or Haskell question and you search for it online chances are it's already been answered on StackOverflow. If it hasn't there is still probably enough information to guide your future search.
"Go to the Slack channel" is not really an acceptable substitute for that level of independence in research.
SO shouldn't be used as a gauge of popularity of a language or framework. Well established languages/frameworks will not see a lot of new questions being posted there since they have an established set of answers. Newer frameworks/languages will see a lot more activity there building up a base set of questions/answers.
The other angle that SO can misrepresent is that inexperienced developers are using the site more than experienced devs. If you know your language/framework/libraries pretty well, then you are mostly going other places for help like API docs or maybe directly to the Github repo for a library and asking committers since you are much deeper into the language.
Another reason could be that SO itself is trending downward. I know that there are people, like myself, who don't often use it or who just use it as part of a set of google results for the topic they are researching.
I prefer LFE (Lisp Flavored Erlang) because I prefer Lisp syntax to the Ruby-like Elixir syntax. LFE compiles down to core Erlang, while Elixir compiles down to vanilla Erlang. True macros can be done in LFE, but as I understand it, macros are done after parsing in Elixir. LFE was created by one of Erlang's co-creators.
It's fine to be a fan of LFE, but not so fine to spread misinformation about Elixir macros.
Macros in Elixir are "true" compile-time macros... They take the AST as an argument, and are expected to return an AST. Possibly the first, or one of the first, implementations of "true" macros in a non-homoiconic language, which was previously a strong selling point of homoiconic (Lisplike, syntax-less) languages.
I'm curious to hear what you've built with LFE, and what the experience was like? What would you say were the biggest stumbling blocks you ran into?
I'm also in your camp (not liking Elixir or Erlang's syntax, but really liking BEAM). I like Clojure a whole lot, but would prefer it on BEAM for a whole variety of reasons. LFE seemed like a step back from Clojure in many ways when I looked at it.
Our task load was getting out of hand, and during peak load throwing memory and CPU at it was not helping. Task monitoring was also a pain. It needed a complete rewrite, because the original design, while robust for what it targeted at the time, could not keep up with our growth.
The integration developer I hired to take over the project looked into using Go at first, but then discovered Elixir. I started learning it with him (he quickly surpassed me), and after a few trial task implementations, I gave him the go-ahead. We couldn't be happier with the result. Thanks to OTP and the existing tools built around it, we have great monitoring, and the ability to easily and quickly scale up workers at peak times.