Great post. Thanks for the praise of New Relic as well.
I'd love to get your feedback on what use cases you had where we were not sufficient. Given that we've been around for 3 years and this is the first release of your profiler, it's not fair to compare features, but I'd still love to know what metrics tipped the scale toward building this.
Three huge features we offer that it seems like you're currently missing out on are automatic profiling of slow requests (vs. manual profiling), automatic real user monitoring to provide insight into the end-user experience, and automatic instrumentation (vs. stitching profiling code into your app).
We totally agree that it would be nice to see web frameworks embrace the concept of performance metrics and provide hooks that tools can use. Rails is way ahead of the curve here. Starting with version 3, Rails ships with plenty of hooks across various aspects of the models, views and controllers.
The big advantage we have is "production ubiquity". As a "trusted user" navigates through the site - in production - she can see the time it took to render the page ... on the page ... this constantly updates with all the ajax calls, which is an innovation. At any point the user can click on the time and break it down into little bits, without navigating away. You can see it in-action at http://data.stackexchange.com just log on.
Monitoring end user experience is a VERY cool feature. It ties in to what I like to call "birds-eye" profiling. This is not at all the problem we are trying to solve with our profiler. In our context we get that same information from the chrome dev tools and firebug. When it comes to "birds-eye" profiling, you guys are innovating and leading. I can not think of any competing products that have a more complete feature set.
Automatic profiling in almost any platform is real tricky business in production. It gets even trickier in statically compiled apps. In an ideal world I would write code like the following:
def bla;
fast_method;
fast_method;
slow_method;
fast_method;
end
My "ideal" production profiler would automatically "know" to instrument "slow_method" and forget about the fast ones, this would all work automatically in production. Technically this can be achieved in a .Net app, but is very tricky. You would have to spin up a new appdomain instrument say 3% random calls using ICorProfiler and route a portion of the traffic to it, then you would have to cycle and do another 3% and so on. At the end of the process you could use something like mono cecil to dynamically recompile the app with all the "interesting" spots profiled and re-route all the traffic to it. The trouble is, the set up is way too elaborate. In a Ruby app this would be way simpler to do and in fact somebody could build something that does something similar ... today.
The trouble with ICorProfiler on a site the scale of StackOverflow, is that it introduces stability issues which are unacceptable, particularly when CPU is way high. We can not afford to use is for the constant "production" profiling.
Now, there is a bunch of profiling you can stitch up more simply like, time in controller, time in view and time in db. But, the .Net framework is way behind on providing the hooks there, sadly.
All of this said, I must admit I have not tried NewRelic in the last 6 months, and would be happy to give it another shot and send you through some of my ideas, you are doing real important work that is making the internet a better place.
I'd love to get your feedback on what use cases you had where we were not sufficient. Given that we've been around for 3 years and this is the first release of your profiler, it's not fair to compare features, but I'd still love to know what metrics tipped the scale toward building this.
Three huge features we offer that it seems like you're currently missing out on are automatic profiling of slow requests (vs. manual profiling), automatic real user monitoring to provide insight into the end-user experience, and automatic instrumentation (vs. stitching profiling code into your app).
We totally agree that it would be nice to see web frameworks embrace the concept of performance metrics and provide hooks that tools can use. Rails is way ahead of the curve here. Starting with version 3, Rails ships with plenty of hooks across various aspects of the models, views and controllers.