> The benchmark premptively runs the fans and pauses between runs in order to cool off. These measures give JRuby at least a 25% speed gain. However, JRuby still might be running slower than it would on a box with appropriate cooling.
Right. Your computer can't be trusted to not throttle its performance due to crap cooling, and you're trying to benchmark using it?
Without knowing what runtimes and parameters are used it's really hard to tell what's going on in this benchmark; JRuby can see 100% and sometimes more improvement on many workloads by tweaking JVM options.
It looks like JRuby is just run under the system JRE (which is what?) with no additional options; that's going to be slow for most workloads.
It'd be nice to see a lot more documentation around the runtime configurations and what the Rails app itself is doing.
Down the bottom he states that it is running on Osx. The fan is premptively spun up, which gives Jruby a speed boost. It's probably going to run even faster with proper cooling.
Not sure how comparing different versions of rails shows if its 'fast' yet. I'd think setting up some request target and seeing how far RoR is from that target with each version would be better. Or comparing RoR to other platforms is also a good bet, eg http://www.techempower.com/benchmarks/
I just don't really see how this answers the question other than 'ehh I think it looks a bit faster than it was awhile ago?'
EDIT: Also interestingly, JRuby seems to be slightly better on the small instance EC2 test at techempower than regular Ruby (small instance should be similar to a laptop?). Yet in these graphs it's much slower. Anybody know the reason?
>EDIT: Also interestingly, JRuby seems to be slightly better on the small instance EC2 test at techempower than regular Ruby (small instance should be similar to a laptop?). Yet in these graphs it's much slower. Anybody know the reason?
Thanks to some community experts, we're now using TorqBox (TorqueBox 3, built on Undertow [1]) for our JRuby tests. The JRuby community seems satisfied [2]. :)
Ah, that explains it. Pretty massive difference really. I guess that makes Torquebox+JRuby the best way to run ruby code? I wonder why the RoR community generally ignores it? Maybe it's just a timing thing and as more companies learn about Torquebox they will swap?
I'd wager that it's the TB/Undertow code running most of the time on that benchmark. On real-world benchmarks the difference is going to be a lot smaller. Not saying that you shouldn't run your JRuby apps on Torquebox - it's a great choice.
Indeed. In our tests, the plain Rack implementation is extremely impressive, but that is probably because there is so little Ruby code being run [1]. By contrast, the Rails implementation on JRuby is just a bit better than plain Ruby, but that is an improvement versus our previous rounds that were not using TorqBox. Previously, we had been using JRuby deployed on Resin, and the results put that JRuby+Rails combination slightly slower than plain Ruby+Rails. For what it's worth, Resin is fast as far as Java Servlet containers go.
Since TorqBox/TorqueBox 3 is very new, I don't suspect it's being used much yet.
Amazon says their "m1.small" instance has 1.7 GiB of memory. That's a lot less than most current laptops.
Maybe JRuby uses more memory in total, but can handle more concurrent requests per MB of memory used? That could explain the discrepancy if it's true, though it's just speculation on my part.
Another theory is that the benchmarks are too application-specific. Rails is a complex framework with which you can build many different kinds of applications, each with its own performance characteristics. Conceivably, application A could perform better on JRuby, while application B could perform better on MRI. Depending on which application you use for your benchmark, either interpreter could appear faster.
This calls into question the utility of benchmarking an entire framework. It might be better to do micro-benchmarks, with each one testing a specific scenario. Scenarios could include, for example: A request that loads 1,000 database records into memory; a request for a static image; a request that queries a many-to-many relationship; a request that saves an uploaded file to disk.
The thing is, those are all I/O bound operations and in the case of the databases most of the code you're benchmarking is native extensions, so I would expect the difference to be negligible.
Testing whole Rails apps really does seem like the best way to get good data. The interesting speedups are likely going to be in places like string matching / string copying and other places that don't involve network access.
Startup time is slow because it needs to warm up the JIT. Same with Rubinius, which is mysteriously missing from most of these graphs.
Memory consumption partially depends on how much you tell the JVM to use - it'll run the garbage collector more or less aggressively to keep within that bound - default appears to be min(system memory/4, 1GB).
That's not to say it doesn't need more memory than MRI, but you should factor in the concurrent native threads allowing one process to serve many clients, and the precise compacting GC limiting memory fragmentation and associated heap bloat with long-running processes.
> Startup time is slow because it needs to warm up the JIT.
Question (I'll end up simultaneously googling...): if the code doesn't change between restarts, is there a way one can preserve JIT statistics, so the JVM doesn't have to rediscover hotspots? Like an RDBMS able to load custom statistics for its plan engine.
I've benchmarked MRI vs jruby a number of times over the years on real apps and MRI usually cones out faster on average with worse numbers at 95th and 98th percentiles due to GC performance.
I never took the time to figure out why it is faster, but I assume that it is due to C libraries being faster for json serialization and my apps doing primarily json serialization after fetching from the database.
Edit: I should test the performance of the json libs, which would be easy to do
I'd wager that a bigger goal is being able to take part of the Java ecosystem.
That said, we really need to know which flags he's running Jruby with - and which
The internet tells me that you achieve better performance with -server at the expense of a bigger start-up time penalty. Anecdotally, it takes a while for the JIT compiler to find the hotspots in your code and optimize them away.
All of this will depend on how Jruby-rails keeps code in memory, yadda yadda. Someone who actually has used Java/Jruby recently can chime in with more details.
Looks like it's running with the threadsafe mode off. That would probably give JRuby an edge, as it's able to take advantage of greater thread parallelism than MRI. No idea if he's letting the JVM warm up enough either.
As far as I know, Passenger Enterprise and Puma are the only multi-platform ruby rack app servers that support multi-threaded concurrent request dispatch. There may be other jruby/java-only deployment scenarios that support concurrent request dispatch.
Without concurrent request dispatch, jruby's hypothetical greater thread parallelism is unlikely to effect requests-per-second benchmarks. Also, the benchmarking test would have to be willing to issue requests concurrently!
> Without concurrent request dispatch, jruby's hypothetical greater thread parallelism is unlikely to effect requests-per-second benchmarks. Also, the benchmarking test would have to be willing to issue requests concurrently!
That's a good point.
That said, JRuby would most likely benefit from a concurrent test, as the thread-per-request model of JRuby can scale much higher than the process-per-request model which is used a lot in the MRI world.
Yes, that's what I said, _if_ you have an app server that supports concurrent request handling. If you are deploying a JRuby app with thin or unicorn or the free version of passenger -- you don't get multi-threaded concurrent request handling.
Wouldn't it make more sense to test various features of Ruby for comparison rather than comparing an entire web framework? Given that, the URL should be israilsfastyet.com, or more appropriately howdoesrailsbehaveonvariousrubyversions.com
Well, it's a comparison of different ruby versions, using Rails. That's pretty useful to people who use Rails, which is a lot of people.
Of course, it's not a real Rails app presumably, but just a sample/dummy one. So you may or may not see comparable speed-up with a real app. But it's probably suggestive of the upper limit of improvement you can get by switching ruby versions.
"Will I get a performance improvement from my Rails app by upgrading ruby" is a real question a lot of people will have. Micro-benchmarks of ruby wouldn't answer it as well as the OP does.
If you test small features, you get knocked for concentrating on microbenchmarks which don't have any real-world meaning. If you test whole applications or frameworks, you get criticised for benchmarking that rather than the language implementation.
I'm surprised at how negative the comments are here. Graphing how different Ruby versions/implementations perform running Rails is very helpful! Glad to see that MRI continues to get faster with every version. Merry Christmas!
The negativity comes from the fact that this benchmark is useless and, at worst, misleading. True benchmarks are hard to get right. Running a bunch of stuff on a laptop isn't sufficient evidence of anything.
Kudos to making the benchmark opensource at least, but even from a quick peak, I wouldn't trust this at all. As others have mentioned, JRuby shouldn't be this slow. Odds are the benchmark is throwing away the JIT gains or the configuration is poor. Add that to the fact that there is no discussion of what a 'rails request' is or what version of Java is being used and there seems to be little to learn from this research.
I'm not sure I have ever seen a benchmark on HN that wasn't called misleading and useless in the comments.
But I agree it is hard to get benchmarks right -- I'm not even sure there is a 'right', as a benchmark will usually be based on certain assumptions that only sometimes apply. (Unless you are benchmarking your own actual production code, in which case you know it's assumptions are valid -- for your own code).
So I don't think that's a reason not to share benchmarks. But I _do_ think it's a reason to be really transparent about exactly how the benchmarks were done. Ideally all code should be available on github.
Then people can examine your assumptions (including ones you knew you were making and ones you didn't) and modelling, as well as fork or pull request to do benchmarks in some other way to better model the scenarios they care about.
Although also in my experience, you get LOTS of people telling you your benchmark is wrong because of X, Y or Z -- sometimes they are correct that it doesn't accurately capture something they are interested in, sometimes they are just confused -- but virtually nobody actually interested in writing code via fork or pull request to capture what they are interested in. Once you put out a benchmark, everyone in the peanut gallery thinks they are entitled to keep demanding you change it in various ways for them.)
No matter how much time you spend on it and thought you put into it, publishing a benchmark on HN is probably the easiest way to have lots of people telling you you're an idiot who did things wrong. It almost seems like HN collectively wants to discourage people from running benchmarks -- or from sharing what they found publicly.
But it's because doing good benchmarking is so hard -- well, that's exactly we need more people working on it, and sharing how they did it so others can improve upon it, etc, right?
While I'm sure it's not the aim of this site, there is some small value in knowing what sort of performance you get with a naive, untweaked deployment.
Wouldn't it make more sense to do this on generic server hardware with a LTS Linux OS? Once that vintage and customized MacBook bites the dust the website pretty much dies as well.
There's nothing magical about a Linux server. If you're arguing an LTS edition will run on newer hardware, well sure but the benchmarks will be just as incomparable.
You're right that theres nothing magical, but there is a clear advantage to faster disks, a more modern & faster CPU, newer & faster ram (this is running DDR2 in the benchmark), etc. I'd be curious why this particular machine was used for benchmarking, and if the same benchmarking method yielded different results on EC2 or something like that.
What are the odds that the Linux kernel is faster than OS X for server workloads? I don't have any data to back this up but I've heard awful things about their filesystem in particular. (Supposedly they have a global filesystem lock.)
If you're going to call it "Is Ruby Fast Yet", you should really be comparing Ruby implementations to implementations of other programming languages. Asked this way, I believe the answer to the question is a definite "no".
As a small example, I benchmarked a real Scala web app running on worse hardware (a 1GB Linux VM) at over 650 requests/s. There was no degradation in load at this rate of requests, but it was 10x the load we saw in production so I stopped there. The benchmark of Rails here goes to ~120 requests/s.
Well there is a JRuby and Rubinius comparison. Those are different implementations of the the interpreter. Ruby will never be faster than C, but this site is aiming to show the improvements over time.
Comparing across languages was originally part of the long-term goal. However, I haven't yet come across any good, multi-language, real-world benchmark.
I've been hacking on a little scripting language of my own lately[1]. To get a handle on its perf, I have a few small benchmarks[2] I run that compare it against Lua (5.1.5), Python (2.7.5), and Ruby (2.0.0p247). (All of those and my language are bytecode-compiled, dynamically typed languages, hence that comparison.)
I happen to have Ruby 2.0 installed, and I was surprised to see it handily beat the other languages. I had expected Ruby to be the slowest of the pack, but they've apparently made big strides in 2.0.
On the off chance that anyone is curious, here's the results of running them on my machine:
Those numbers are score, time, and percentage. Score is basically 1/time, just to yield a number where bigger is better. Time is best of ten runs. Percentage is how my language's score compares to that one. So, for example, the 264.20% lua method_call score means my language is about 2.6x faster than Lua on a benchmark designed to stress dynamic method call performance.
Oh, man, this reminds me of the pain of writing even the most trivial microbenchmarks in seven or eight languages. How about throwing in some of the fast scripting languages for healthy competition? E.g. JavaScript, LuaJIT and Julia. Feel free to borrow (MIT licensed, but I'm willing to relicense whatever I can):
I'm certainly impressed by those implementations, but I don't think I'm competing with them. :)
I had JS (using node) in there for a while, but it was regularly 10x compared to the other languages. Using a JIT makes it effectively a different class of implementation. It's like entering a car in a bike race.
Thanks for the benchmarks! I might steal some of those. They may be more arithmetic focused than I need (which makes sense for Julia, but less sense for an embedding scripting language where most math is probably done in the host app).
It was a turn of phrase, but really I just meant that in my experience it's healthy to have something to aspire to – that's why we always compare to fully optimized C and Fortran. It's easy to fall into an "interpreter bubble" and forget that there's a whole other realm of performance out there. And of course, interpreters can be quite fast – CPython is no slouch and Wren is giving it a run for its money, so kudos. Femtolisp [1] (which we use for Julia's parser) is a great example of a small and simple, yet fast and featureful interpreter. I really want Jeff to write a little book about it, walking the reader through the implementation.
> I just meant that in my experience it's healthy to have something to aspire to
I had actually thought that's what I was doing when I compared it to Lua! I was quite surprised at how well Wren did in comparison. The trick will be to see if I can keep it that fast as the implementation gets more robust. It takes some shortcuts right now that don't scale well, but happen to be faster.
> I really want Jeff to write a little book about it, walking the reader through the implementation.
I would love that. I've skimmed femtolisp a few times and it's clean but suffers from the almost total lack of documentation that other programming language implementations also seem to have.
I don't think that the standard Lua implementation is especially fast – it's the LuaJIT implementation that's so impressively fast.
Programming language implementors do seem to be even less fond of writing comments than most programmers, don't they? I think one of the issues is that internals can change a lot fairly quickly and dealing with comments just makes that worse.
I know you were just going with what was already on your system most likely, but wouldn't it be a good idea to compare against a more up to date version of Python (version 3.3.x) and Lua (5.2.x)?
There is one sentence missing from each graph that would've made this a lot easier to grok:
"x is better" where x is more requests, faster startup time, or less memory usage.
I prefer graphs that lead me to what I should expect and that sentence would go a long way to eliminate the "mental math" I had to do to understand each one.
On another note, the 2 year graphs are particularly interesting. They're more consistent than not but those inconsistencies seem a little awkward.
The variance is sometimes due to RVM troubles. The benchmark is partly a test of the entire Ruby ecosystem. If RVM is bugging, then people are installing sub-optimal implementations. For example: https://github.com/wayneeseguin/rvm/issues/1418
It is when your development process involves rapidly iterating the code by running the process many times (i.e. TDD). Anecdotally, when the feedback loop is tighter, I'm much happier. Developing (especially refactoring) in Rails feels like watching paint dry, since it's soooooo slooooow. Every time the process needs to restart, I'm left hanging........waiting......for..............the..............
..........process.......................to....................
...................................................s..........
................t.............................a...............
..............r...............................................
..tttt...........
Also the benchmark is the best case. In real world projects, I've seen Rails take over 60 seconds to start (it took a few days of development effort to bring it down to ~20 seconds). Every time you rerun tests, run a rake task, start or restart your server, etc. It's like the heartbeat of the development process grinds to a halt when you add more features to your Rails server.
And no, tools like Spork are not the answer, since you still have to balance the amount that is resident in memory to fork with coverage of what you can change. It also opens up a fractal of complexity.
No can do. It got downvoted and HN does not allow me to edit negative karma posts. So please upvote my previous post so I can fix the layout of these comments :-)
I'm assuming the negative votes are due to the formatting and not the message. I hope you, dear reader, don't miss the irony of this situation...
If it's downvotes because of the message, well I feel bad for the downvoters and wish them much luck with using their framework.
You missed the edit window. If you don't come back and edit your post within one hour, it's locked. Negative karma has nothing to do with it. That's the risk of fire-and-forget posting.
Or it's the risk of failing to account for human behavior...
It's not exactly a moral imperative to obsess over your HN comments, IMO.
Most likely, it's a simple bug, that seems to be fixed now :-) No ethics involved. Nothing to see here.
Btw, in the OP, I was trying to express the effect that the increased startup time affects flow and iteration. It's a similar effect to a comment breaking the layout of the page.
I'm building http://techendo.co/ and it's running rails on heroku. I was wondering if there was some easy ways to get some better performance out of it immediately now that we're picking up traffic. If anyone could give me some pointers or link me to a page I'd appreciate it.
That's an incredibly broad question, so it's pretty hard to answer. Here's some general ones:
* More instances will give you more requests/second, but only so long as secondary system (database, search server, whatever) can keep up with the load
* N+1 queries will kill your performance. They can add a lot of time to processing requests.
* Everything in the most used parts of your site should hit an index on the database. If it's not I can almost guarantee it will be slow.
* Look for work that you can move "offline". As long as they go out within a minute nobody will notice that you aren't sending emails at the exact time they are being generated. Move that work to a background queue so your web server can just add a queue entry and get on to the next request.
* Cache anything you reasonably can. It's a lot cheaper (i.e. faster) to read from a cache than to do a database lookup. It's cheaper to read from a cache than to render an ERB template. Caching can introduce its own difficulties, but the performance gains can be worth it.
Without knowing some specifics of your implementation I really can't give you better advice. Depending on the specific libraries you use it might not even be possible to move to JRuby due to support for C extensions. You might get some performance benefits out of it, but I'd expect it to only buy you some time to execute on some of the above improvements.
Going off the techempower benchmarks, looks like you should be able to get some speedup by switching to TorqueBox. No guarantees, but if you've got the time and need the performance it could be worth testing. Here seems to be a decent guide you could follow: https://gist.github.com/bbrowning/4296297
Make sure you're running New Relic or another monitoring service that can give you some insight into various parts of your application. If you can't see what is slow, then you can't fix it.
Fixing bad queries (n+1 is a common thing that will get you in trouble), and caching API calls (Rail.cache.fetch is your friend) will help a ton.
Background any tasks that must be slow. I like using Resque/Redis myself.
At some point it helps to cache HTTP requests when possible too. I've been working trying to get a special Buildpack working with Nginx for better caching on Heroku, but I'm not done yet. Also make sure that you aren't serving static assets from Heroku, and you're using something like CloudFront for it.
Email me if you need more help. I can normally make these things fly.
There's great data here that's obscured by choosing terrible colours for the graph and showing things that are pretty much irrelevant. A summary chart giving relevant percentage differences would be a lot better than this line chart.
I don't like those charts at all. I want to see the progressive Ruby versions on the X axis, I think. The dates are pretty meaningless. If it's the same code, it should run in the same number of cycles; anything else is just noise.
That said, +1 for having a go at it!
Also, yes, Ruby is pretty slow. But it's fast for getting something up and running quickly. If you get wildly successful, you can always swap it out for something else if you need to.
If I understand this correctly, this is running periodically on someone's Macbook; people are complaining that's a terrible platform for a benchmark.
It should be possible (if you care enough) to re-run this from the source control history on platforms of your choosing, across whatever date ranges you wish. Anyone really, really care enough?
Right. Your computer can't be trusted to not throttle its performance due to crap cooling, and you're trying to benchmark using it?