I appreciate the time obviously spent on this, but i see a couple errors and ommissions. C# and F# are VM languages that run on the CLR which is extremely similar to the JVM.
Also, Go and Erlang use fiber blocking threads (userspace threads). A fair comparison would use fiber blocking in Scala/Java as well using Comsat/Quasar.
I have a feeling JVM based languages would perform far better than Erlang, and possibly even as fast as Go if fiber blocking threads were used.
Edit: they're using Akka which (I believe) IS fiber blocking so part of my critique may not apply. However, Scala is known for generating somewhat slower code than Java so I would expect Java with fibers to perform 2-10x better
Akka is built on top of the JVM which doesn't support fibers (yet: see http://openjdk.java.net/projects/loom/). Akka implements its own scheduling atop Java's fork-join thread pool. Fibers at the JVM layer would be faster.
Similar to Quasar then I assume. I'm sure native fibers would be faster, but bytecode is fairly close to ASM and i don't think the performance difference is large.
Benchmarks I've seen for Quasar show no performance difference to regular threads at low paralellism but it can scale much higher
Loom is quite a bit different than Quasar, but pron (aka Ron Pressler), is leading the way on both. Loom is essentially Quasar would have loved to have baked into the JVM. It's the solution for the bytecode rewriting hackery and resulting limitations it faced.
I don't think erlang procs are fibers, they are allotted a certain timeslice and if the instructions for the function exceed the timeslice the function is preempted and the VM comes back to it later, it's also preempted if it blocks on io or waiting for a message, of course.
Erlang does not preempt, a reduction is always completed before the scheduler decides on whether or not to move on to another thread. Reductions are small enough that this isn't a problem.
well normally, there are some cases where you can compile C# and f# native now:
mono can do it, that is how it targets IOS
microsoft has tooling that does it for universal apps
and coming soon! AOT native compilation for anything you want.
But yes, the normal way of things with C# and F# is just like Java, and Java has special cases where you can do AOT compilation too.
Actually you can compile VB.NET, C#, Managed C++ (latter C++/CLI) and eventually F# to native since .NET exists, but many seem to keep forgetting about NGEN, as it requires strong assemblies.
Likewise Java always had AOT compilers to native code, but since Sun was religious against it, only commercial third party JDKs offered it as option. It is quite common in embedded deployments.
But as usual among recent trends regarding dev tools, very few were willing to pay for them.
Well, you don't have to use those abstractions. But yes, it's a sad fact that when programming in Scala, the closer to Java-style you program, the faster your program will be.
I haven't read this paper closely, but it looks like it's more measuring the performance of Akka, rather than the performance of Scala. Akka does reasonably well here compared to Erlang, so I don't think there's anything we can complain about.
Could you get all of Akka's features with better performance if it was made in Java? I doubt it. Optimized Scala and Java code should be identical.
> Many of Scalas abstractions, unlike Kotlin, have runtime cost
Whoa there, many of Kotlin's do too. Because one uses an option class and another inserts precondition checks in a lot of places, doesn't mean the latter is zero cost. While Kotlin overall probably has less overhead with better primitive use and what not, there are cases where Scala is less because of macro support.
This depends on the VM you are using.
Currently GraalVM gives Scala a huge speedup (sadly the speedup comes with startup cost, etc, which would not benefit the bechnmarks game)
I recently switched the core pages of my website (a small search engine) to Go from PHP. Have not implemented any goroutines yet, just focused on a line-by-line port for now, and it was overall a fun experience. I would highly recommend Go. Only downside is the smaller (but growing) community of developers, so you might spend a little more time finding answers. Also I wish it didn't force you to comment out unused libraries or unused variables when you're trying to troubleshoot something. It should be allowed when running, but keep it restricted when building maybe?
> In the experiments we use the following versions of the languages: Erlang/OTP 19.0, Go 1.7.1, Scala 2.12.0 with Akka 2.4.12.
These are pretty outdated, it would be interesting to see numbers on the latest versions. I'd be especially curious to compare with Erlang/OTP 21.0 (coming out today) as they did a lot of optimization work on the attributes tested in this benchmark.
If you are comparing to Go then just regular Haskell has all the same capabilities and more (lightweight threads and communication primitives, plus extra cool stuff like software transactional memory). Cloud Haskell is trying to put it more in the Erlang camp where you can distribute beyond a single process transparently, but I don't think it is well maintained or really production ready.
(1) Cloud Haskell is primarily designed for execution on distributed memory architectures where the messages pass over a network. So expect that the it would show high communication latency in the first PingPing benchmark.
(2) GHC Haskell processes are lightweight, i.e. maintained in the GHC Runtime Environment, and elsewhere in this discussion called fibres. So expect that it would have fast process creation and be able to sustain a large maximum number of processes (second benchmark).
(3) It's hard to speculate on throughput (third benchmark). The high communication latency may be hidden in a steady state by having multiple requests in the network. Haskell execution time is slower than many languages, but is probably comparable with Erlang and Scala.
Those benchmarks do not compare like-for-like. Often the Haskell versions try to solve the problems elegantly with high-level functional code at the expense of performance, e.g. see Mandelbrot.
GHC Haskell is an advanced optimising compiler that will produce considerably faster functional code than both Erlang and Java. For maximum performance, it is perfectly possible to write imperative array-oriented code in Haskell, in which case performance should be comparable to Java.
My post was more for the benefit of other readers, who might have otherwise concluded that modern GHC is "slow". There are many ways to compare the 'execution time' of e.g. Java and Haskell. Idiomatic Haskell code may well be slower than (traditional) idiomatic Java code. But these days more and more functional code is written in Java/Scala/Clojure and the JVM does not optimise such code nearly as well as GHC.
I can certainly see merit in that idea. The problem is that it's often not clear what the idiomatic style for a particular language applied to a particular problem is. Many of the Haskell solutions also look quite old, predating for example the now standard Haskell "Vector" library.
It's a useful resource, but I think we should we wary of making very general remarks based on the results.
We should we wary of making very general remarks, period.
The benchmarks game is not restricted to "the idiomatic style" so it isn't a problem that "the idiomatic style" for a particular problem is often not clear.
> Many of the Haskell solutions also look quite old…
This paper covers all the mainstream languages but I feel an "alternatives considered" section is missing. I would love to know what's the author's input on Swift server side, Elixir and Crystal.
For the benchmarks done in the paper, elixir should basically benchmark identically to erlang, since if there's any performance regression, it's a very thin wrapper.
I doubt crystal is mature enough in its multithreading model (which is still nascent, and IIRC not fully supported in the current rev, and inherited from ruby's in concept) to perform well at all in this. Erlang, Go and Akka were all conceived of specifically to handle multithreaded connections.
There were many languages that we would like to have considered. We do expect the performance of Elixir to be pretty similar to Erlang as it runs on the same VM (BEAM).
With regards the "alternatives considered", Section 2 outlines key characteristics of server languages, and Table 1 analyses a selection of languages against these characteristics.
We welcome others to report results for other languages using the benchmarks.
One thing you might find interesting - a small, crude experiment suggests that changing maxBlocking.erl to use process hibernation for those dormant processes could almost double the max number of processes Erlang can support.
I changed maxBlocking.erl's final few lines to read
Mainstream? It does not cover Java and C# and makes false assumptions on them. I have to assume that the detail selection group was not determined out of the 12 others but pre-defined. This paper is academic.
Only C++ and Java(C#) can share memory between threads. If you are going to compare things you probably should understand that and design the test so that it shows the difference. If the application is so simple it can scale on different machines, the performance doesn't really matter.
Go prefers message passing of copies using channels, but absolutely does permit shared state if you want it. I've written multithreaded code with shared state in Go myself. You just need to pass a pointer to the state across a channel, and can then control access using synchronisation primitives such as mutexes in exactly the same way you would in Java or C.
This comment is terribly misinformed. You absolutely can share memory between threads in Rust and can do it without race conditions (eg Arc pointers and Mutex).
Go has a built in message passing system in the language but you can share memory between goroutines without friction, and you can lock goroutines to real os threads if you want, too. I did both in my gameswithgo stream. It was a fun time!
Also, Go and Erlang use fiber blocking threads (userspace threads). A fair comparison would use fiber blocking in Scala/Java as well using Comsat/Quasar.
I have a feeling JVM based languages would perform far better than Erlang, and possibly even as fast as Go if fiber blocking threads were used.
Edit: they're using Akka which (I believe) IS fiber blocking so part of my critique may not apply. However, Scala is known for generating somewhat slower code than Java so I would expect Java with fibers to perform 2-10x better