Ok, understood. Then the other benchmark that matters in the typical Rails workflow is how fast rails c starts, and how fast tests run. If MRI runs tests twice as fasts it's hard not to use it. Think 30 seconds vs 1 minute. It really makes a difference. Or running a single rspec example.
Tests seem a really cold start scenario, even if only a bit of code changes among test runs. Is it possible to save (I don't know what, the internal state of the VM?) for the unchanged parts among runs? Would that even work?
Agreed. That's why I presented numbers for running the language specs. It was something I could easily and fairly compare MRI vs TruffleRuby with. And it's representative of what a typical RSpec suite will look like.
With the SVM, we're now at 1s (MRI) vs 9s (TruffleRuby/SVM). Previously, we were at 1s (MRI) vs 68s (TruffleRuby/JVM). There's a lot of ways to look at that data. You could say TruffleRuby/SVM took 9 times as long as MRI and that rules it out. Or you could say it took 8s longer and that's acceptable for my entire test suite, especially if I don't have to juggle multiple versions of Ruby. I really can't say -- that's for each team to decide on their own. But, we're looking to make the decision easier by further reducing that run time.
It could be possible to save the state of the parsed AST and embed that in the binary. It's actually what we do for the core library to help with start-up time. I'm not sure you'd really want to do that for individual codebases though. You would need to rebuild the static binary every time your code changes, eliminating a lot of the benefit of using a dynamic language is the first place. It would be far better if we just make TruffleRuby and the SVM faster, which we're working on.
Tests seem a really cold start scenario, even if only a bit of code changes among test runs. Is it possible to save (I don't know what, the internal state of the VM?) for the unchanged parts among runs? Would that even work?