To be clear: we track it in the sense that we still have it running as part of old dashboards. It's cheap, and every so often it catches an unintended regression.
The SpiderMonkey team is very much on board with ignoring microbenchmarks. We care about Speedometer, because it correlates reasonably well with real-world performance. We pick and choose which parts of Jetstream2 we bother with. Anything older than that is strictly a measuring stick, rather than a target: it's nice when the numbers go up, but it will only happen as a side-effect of work we're doing for other reasons.
Notably, all the SunSpider wins on this graph are side-effects of Speedometer work we've been doing this year.
I understand the logic, but I never understood why we (and I mean, collectively, the JS VM engineers across the browser vendors) didn't just rewrite the benchmark (or fix it) and name it SunSpider 2. I suggested this many times. It's not that much code.
Yeah, I've been making this argument for years too.
Speedometer 3 is in the works, and it's moving in this direction. On multiple occasions we've done performance analysis of the latest draft version, noticed a particular chunk of code is hot, considered what it would take to optimize it, and then changed it because we don't think it reflects real-world usage.
EDIT: And I guess the answer for "why not fix SunSpider specifically?" is that once you've done the work to turn it into something worth optimizing, it no longer bears any resemblance to the original. It probably looks a lot more like Speedometer, honestly.
> EDIT: And I guess the answer for "why not fix SunSpider specifically?" is that once you've done the work to turn it into something worth optimizing, it no longer bears any resemblance to the original. It probably looks a lot more like Speedometer, honestly.
I get that, yeah, but superseding the old name is probably worth it though; the internet is full of people who never get the memo except via version numbers :)
I tongue-in-cheek proposed that we match the Sunspider ASTs in V8 and either make it infinitely fast or infinitely slow. Now that will kill a benchmark once and for all!
The SpiderMonkey team is very much on board with ignoring microbenchmarks. We care about Speedometer, because it correlates reasonably well with real-world performance. We pick and choose which parts of Jetstream2 we bother with. Anything older than that is strictly a measuring stick, rather than a target: it's nice when the numbers go up, but it will only happen as a side-effect of work we're doing for other reasons.
Notably, all the SunSpider wins on this graph are side-effects of Speedometer work we've been doing this year.