Every time I see JS performance data like this, I'm always mystified to see lots of error ranges of a couple percent. Working on a JS game has made me dive pretty deeply into JS performance (on Chrome at least), and I constantly find that the hardest part of performance tuning is getting any kind of consistent results to tune.
For a random example, I tried reproducing the "Swap rows" test for Vanilla.js. The author found this took 7.83 +/- 0.18
ms, with five warmup iterations. On my machine, running the same command six times consistently gives results like these:
swapRows took 10.944999999999709
swapRows took 1.3250000000007276
swapRows took 1.3200000000006185
swapRows took 1.269999999999527
swapRows took 11.875
swapRows took 1.2750000000005457
Also, and orthogonal to the above, I've always found that even when performance is consistent across repeated trials, it varies widely between page reloads. (This isn't at all surprising, since v8's optimizing compiler isn't necessarily going to make the same choices every time - but it still makes things very hard to tune).
As an example of this, I tried repeating the "append to large table" test. The author's results were 225.3 +/- 1.89ms. On my (similar) machine, trying this five times (reloading the page between) varied between 225 and 329ms.
Am I missing something here? I've looked pretty hard into v8 perf tuning and various benchmarking libraries, and I've just never seen the kind of consistent results one always finds in comparisons like these.
You're basically right. Thus we've decided to perform 10 runs for each benchmark and drop the worst 4. This eliminates cases where the gc or a background system process causes a slow run. In consequence this also strongly reduces variance, but is blind on the gc eye. It would be interesting to include gc, but I have no idea how to do it in a real good way.
The results are comparable (but nowhere equal) between runs and the difference between the frameworks is in most cases large enough that the ranking stays consistent (but you wouldn't prefer preact to cycle.js v7 for having a 0.01 better result in the average slow down, would you? For ember, mithril and cycle.js v6 performance might be something to consider depending on your use case.).
If you look at the console output this is just an approximation, the real measurement is performed on data from chrome's timeline using selenium. The console measurement uses a timer to guess when rendering was performed. Of course this is more inaccurate than using the timeline.
I think an equally interesting result would be to keep only the worst 1-4 benchmark results and compare those. In my experience some frameworks put much more long-term memory pressure on the GC, and by discarding those, your benchmark turns a blind eye on that.
In another words, I'd be interested more on the worst possible performance of a given framework, than their best (assuming the app is written in a reasonably good way otherwise).
Also, +1 to the sibling post that asks for longer runs, that would solve the GC question by including it altogether!
It's hard to get consistent performance when you have any form of memory management, caching, or a JIT behind the scenes. I feel like this is just something you have to accept when writing code in a high level language.
Sure, and in tuning my game I've tried to work out ways of accounting for this. My point is basically that JS performance data with really small error ranges makes me suspicious about whether the data is meaningful or not.
I've worked on a few apps that feature large tables like this benchmark does. With Angular for example, it's a really bad idea for performance to create DOM elements for each table row when most aren't even visible which I think is done in the benchmark:
https://github.com/krausest/js-framework-benchmark/blob/mast...
There's lots of grid/table libraries available that use what's called virtualisation to stop lots of rows slowing the app down; rows that aren't on the screen aren't computed or rendered.
My feeling is if you avoid rendering data excessively in this way the performance differences between most frameworks aren't that important. If you're required to display 1000s of dynamic values at a time your UI probably needs work.
I usually come away from things like this with the distinct impression that they're pretty much all "fast enough" for my needs. Admittedly, I'm not asking a lot of JavaScript, and my skill level with JavaScript tends to lead me to take the path of least resistance, or at least the path of best documentation and most examples.
That said, this kind of third-party benchmarking is probably useful, because so many frameworks have benchmark charts on their own site which indicate they're all faster than everything else. (Which may not even be inaccurate...for the things they benchmark, they may be the fastest. Which again maybe indicates they're all "fast enough" and unless your use case exactly matches that of the specific benchmarks in question, the benchmarks might be misleading you about which one will be fastest for your app.)
Also, I had no idea there were so many JavaScript frameworks that I hadn't heard of (and there's at least a few that I have heard of that aren't on the chart). Every time I turn around, there's two or three more JavaScript frameworks. Surely, this will calm down at some point, and a couple of leaders will emerge and folks will kinda settle into moving those leaders forward rather than reinventing everything?
It is definitely easy to get overwhelmed by so many choices. In the end Pareto still applies, 20% of fewer of technologies are being used in 80% or more of the projects. Right now, React, Angular and Ember are probably leading the pack.
It's already starting to calm down. On that list, you have a few broad categories, with some overlap.
Angular and Angular-ish (I use the term loosely) frameworks like Aurelia.
Observable based frameworks (Cyclejs, Angular 2?)
React and React near-dropin replacements (Preact, React, Inferno, etc)
A bunch of MVC/MVVM frameworks that are not Angular-like (Mithril, Vue, Ember...). This is where there's the most variety, namely because many of these are a bit older (even though many kept up with time).
So there's not really nearly as many as you'd think, and in some cases migrating from one to the other as need be isn't crazy.
DOM manipulation should be added to the title and before calling one faster than other one should understand that a JS framework is (should be) a lot more than painting views super fast.
Seconded, I adore Riot and it bugs me that it's left out of most benchmarks.
FYI Riot's alpha 3.0.0 release is much faster than 2.5.0, and is almost production ready (in fact, I'm using it on a project and it's rather reliable).
I am not trying to be snarky with this question, but does it really matter with JS?
Its always going to be run in a browser by one user, not having multiple people connecting to it like the server side does. As long as it seems snappy to the user it shouldn't make a great deal of difference.
Counter opinions welcome.
I feel like these comparisons are interesting but a bit one dimensional. A broad generalisation you could make is that frameworks with more features tend to be slower, but mean developers have to write less code.
A LOC or character count for each benchmark would help add a bit of context.
I agree with some of that, but I'd be more interested not in how much code is written for the test, but how far into the private API the code must go before the actual action is done. Kind of like using Test Coverage techniques. I imagine theirs only like 2 or 3 ways developers are, say, appending lots of rows, and the difference maker is how efficiently those techniques are being implemented.
This is benchmark porn. You can create a fast-performing web app with any of these frameworks, as long as you focus on the right metrics, like “time to interact”.
I'm really surprised that d3.js isn't on the list. It should be. I _suspect_ it would completely own most of the tests, but be bad on memory allocation.
I'm not a fan of Ember, but it's focus on tooling, developer experience, and sticking to it's core principles in the name of consistency is something to be admired. It's not as shiny as Cycle or React, but if you started working with Ember years ago, you've had reasonably robust tooling all along (as opposed to the bleeding edge constantly breaking crap of the other frameworks), and have been able to reuse your knowledge for years, so you're potentially quite productive.
Why is that ? I've never used Ember, but it seems like the framework with the best tooling and backwards compatibility track record. Maybe I'm wrong, I don't know.
The templating is very inflexible and the routing becomes extremely complex as your application grows in size. There are also a lot of gotchas with the runloop that require time and experience to understand.
But isn't React also a library rather than a framework by this measure? I think knockout is no longer in development so like Backbone it doesn't really merit inclusion.
These frameworks are all so different it renders these results pretty moot. Vanillaja is fastest but it doesn't offer half of what React or Angular do.
I suspect there's a perf bug in React 15 that causes the 'clear rows' test to be an order of magnitude slower than react 0.14. Otherwise, React is getting faster.
For a random example, I tried reproducing the "Swap rows" test for Vanilla.js. The author found this took 7.83 +/- 0.18 ms, with five warmup iterations. On my machine, running the same command six times consistently gives results like these:
Also, and orthogonal to the above, I've always found that even when performance is consistent across repeated trials, it varies widely between page reloads. (This isn't at all surprising, since v8's optimizing compiler isn't necessarily going to make the same choices every time - but it still makes things very hard to tune).As an example of this, I tried repeating the "append to large table" test. The author's results were 225.3 +/- 1.89ms. On my (similar) machine, trying this five times (reloading the page between) varied between 225 and 329ms.
Am I missing something here? I've looked pretty hard into v8 perf tuning and various benchmarking libraries, and I've just never seen the kind of consistent results one always finds in comparisons like these.