The methodology is questionable on several fronts:
- you can't use the load event to compare advert-heavy vs. advert-free load times. The load event does not fire when the page is usable by the viewer, it fires when every last request is done, in particular even requests fired from various advert iframes. However, those requests may not impact page usability. The load event may even never fire - any offscreen 1px spacer gif that times out will cause that! I don't think there is a built-in event for "done-enough", but the load event is certainly misleading.
- opening the developer tools causes various side-effects; those can cause the page to load more slowly than usual. You shouldn't benchmark with developer tools open (unless you're explicitly targeting usage with devtools open...)
- comparing load times across browsers as reported by the browsers themselves may be valid, but it's not obvious. You definitely want to check that carefully.
- Measuring "peak" CPU usage is almost meaningless without considering how long the CPU is used.
- Measuring the chrome extension process CPU usage and memory usage isn't very helpful, because running this kind of extension causes CPU usage and memory usage in every content tab. Both of these statistics for chrome in this use-case are meaningless. You'd need to measure the memory usage and CPU-time of the entire chrome process tree to get meaningful results. Even in Firefox without e10s it's not valid to measure just the main process CPU and memory usage because plugins are in separate processes (and things like flash or h264 decoding can definitely use CPU and memory).
The only thing this page really makes a decent case for is that Chrome loads pages faster than Firefox - but even there, it's not clear we're dealing with an apples-to-apples comparison.
> you can't use the load event to compare advert-heavy vs. advert-free load times
I don't see why not. It's exactly what they want to show, how much quicker pages can load if you don't have to load the ads. And I personally am not satisfied with a page that has loaded to the extend of usability. Either a site loads fast or it doesn't.
> opening the developer tools causes various side-effects
As long as it's done for all the tests it should not matter. Whatever "side-effects" are introduced will be applied for all tests. Of course this means that the values themselves can't be used to compare the result against other tests, but the goal here was anyways to compare between the different ad blocks and not to generate values for general statistics.
Some pages will be becoming usable before the DOMContentLoaded event is fired. Some will become usable shortly after DOMContentLoaded is fired. Some will become usable when load is fired. Some will become usable several seconds after that. Some will never become usable. (Cynical aside: the ones with excessive ads might fit in that classification.) There is no event that measures readiness of a page, nor can there be; there can only be a collection of reasonable heuristics.
However, we don't need a perfect measurement, just a reasonable one.
For example, you might try to wait for all stylesheets to have loaded, and then run requestAnimationFrame until the latency is below some threshold (say, 100ms) to represent a somewhat-loaded-and-not-too-laggy page.
The point is, you want some measure that plausibly excludes ad-related resources since clearly you don't care how long it takes to load those. It'd also be polite to be transparent about the limitations of your measure.
This is where a human with a stopwatch may come in handy. The clock will start when the page request is sent. Stop the clock when, in your opinion, you think the page is ready to read. Alternately, if you feel the page is loading too slowly, and have reached the limit of your patience, press this button instead.
The event does not have to come from within the vanilla browser, without human intervention. It could be from a plugin you create to collect human feedback on load times.
Of course, the human element will significantly decrease the number of trials you can do within a fixed time period.
> Stop the clock when, in your opinion, you think the page is ready to read.
Even that is not enough. A number of sites (especially ad-heavy sites on mobile devices) look ready to read, but as soon as you try to scroll, are jerky and semi-unresponsive. It's particularly a problem on ad-laden sites of small news publications. Blocking ads sometimes mitigates this, though not always, as some sites insist on using Javascript to implement "fancy" scrolling which inevitably breaks.
(Semi-related sidenote: please, for the love of God, don't use Javascript to hijack browser scrolling. It's almost certain you will not get it perfectly right and some users will suffer. It's almost never worth it. This is the #1 reason I end up having to switch to Reader Mode in Firefox).
The maybe you stop the clock when the user navigates away from the page. If it takes 30s to read the content without ad-loading and 40s to read it with ad-loading, then by subjective user time, the page is 10s slower. That may be because the ads take 10s to load, or because the distraction causes the user to read at 75% normal speed, but it's still a measurable result from a single variable change.
I'd still question the usefulness of this rather arbitrarily number.
The only reason I can think of where I'd be interested in how quickly people can navigate a site without having loaded all the content yet, is when optimizing a site for people with really slow/bad internet connections. But that seems like a whole other topic that has barely any influence on the goal of the comparison.
You don’t need perfect measurements to make decisions.
Comparing times between browser-defined events in the same browser opening the same site on the same day is about the most apples-to-apples as you can get.
I'm not sure any of that matters. As long as the same thing was measured in the same way each time, I think the results are relevant.
For your first point, the measurement wasn't "usability", it was how long it took for the load event to fire.
Second, having dev tools open should affect all results the same way. The absolute number isn't important, it's the relative ordering, which should be the same whether dev tools are open or not.
The rest of your points are just nitpicking. Maybe he didn't use the absolute best way to measure CPU or memory usage, but I'm willing to accept they're good enough proxy measures that the results are relevant.
It's not a good enough measurement - it's a ridiculously poor measurement. You're trying to measure how fast an adblocker is by measuring how long it takes... to load the ads.
How is that in any way helpful? If anything, the proper, comparable load time for the adblocked version is infinite - they never complete loading the ads.
If you're going to run this comparison, you need to at least compare the same set of resources being loaded - after all, if you're willing to omit ads entirely, you clearly don't care how long it takes for them to load.
We don't need a perfect measurement, but this measurement is extremely biased. DOMContentLoaded would be better, but still far from good - you want to measure the load time of the resources the adblocker would not have blocked.
Knowing the time saved by removing the ads is a useful number to know, IMO. I'm willing to give the benchmarker the benefit of the doubt that he used the same block lists for each blocker, although spelling it out would have been a good idea.
This is exactly what ETW was designed to help you measure.
Running WPRUI with first level triage will measure CPU along with a bunch of other stuff. Enabling ReferenceSet under memory Resource Analysis will measure memory usage.
Chrome has an ETW provider, I poked around a bit and couldn't find any for FF (it looks like they register one for the JS engine?)
I'm not familiar with how to solve the "page loaded enough but not done" problem, but IMO if you measure end-to-end times it's probably a close-enough proxy. Maybe measure several times and report on those statistics.
(Edit: Taking a trace with/without the adblocker enabled should allow you to exactly count # of sampled cycles spent in the adblocker - easy to compare in WPA, as long as there are events emitted indicating page load start/stop).
(Edit2: Using UIforETW along with Chrome 46+ will let you see ETW events:
Also keep in mind they tested all extensions with their default filter lists configuration while it reads like EasyList and EasyPrivacy would be efficient enough. As applying filter rules is basically the main work of those extensions and most extensions can use the same lists, they should have set all to the same lists where possible.
They didn't seem to know about e10s in Firefox Nightly which would have solved their "no separate process" thing.
Measuring peak CPU is very, very stupid. The best Ad Blocker should have near 100% CPU but for an extremely short amount of time. Using 100% CPU is making the most efficient use of the hardware, not being inefficient. Now, I think due to the way they're measuring, it's actually closer to being average CPU because task manager samples over a period.
Also: "Firefox didn’t like this page at all and we couldn’t get consistent readings to run tests, on every refresh the browser would simply not respond or crash."
Huh? I think somethings wrong there. Firefox obviously renders tmz.com. A bug in the thing they used to monitor page load times? Did they check the performance impact of it?
> Measuring peak CPU is very, very stupid. The best Ad Blocker should have near 100% CPU but for an extremely short amount of time.
You'd be surprised about how many people perceive this as a bad thing. I've more than once had tickets from customers complaining about high CPU usage under load scenarios. Can't wait to see what journalists make of Servo (which, if up to scratch, will hopefully peg CPU near 100% for a brief period).
However, higher CPU usage associated with longer response times is a different story entirely.
> it's actually closer to being average CPU because task manager samples over a period.
Forget about that: they're even using the wrong tool. Perfmon should have been used. Eyeballing it is completely useless.
I've more than once had tickets from customers complaining about high CPU usage under load scenarios
What they're really saying is not "you should make the CPU usage lower by spreading it over a longer time", but that the CPU is at 100% for an excessively long period of time. In other words, that 100% period should ideally be so short that it's below the sampling interval and does not appear as 100%.
> Using 100% CPU is making the most efficient use of the hardware, not being inefficient
Not necessarily with modern processors. A power-managedCPU at 50% clock-speed actually uses less than half of the power as at 100%.
The relationship of speed to power is not linear, so for a laptop user the most efficient use of hardware might actually be running one job for twice as long at half-speed.
Also if it's really using 100% of the CPU (not just one core), then it will affect the overall PC experience since it uses up all available resources and doesn't leave anything for other processes.
I personally dislike it when an application makes my "PC" lag every now and then. CPU time and RAM are shared resources, thus applications should make sure to not act selfish unless asked for (e.g. encoding a video, etc).
The argument is in favor of near 100% CPU for a short period; gets the job done faster, and uses the available resources (100% CPU) the OS handed down. The time the process spends at 100% is the difference.
> A power-managedCPU at 50% clock-speed actually uses less than half of the power as at 100%.
That's ignoring the overhead of not being able to sleep the whole CPU, including uncore stuff for twice as long if you're only computing at half the speed.
I think it's pretty obvious that the expression "peak CPU usage" implies "peak average CPU usage over a period of time", for the simple fact that literal "peak CPU usage" makes absolutely no sense -- CPU usage is always over a period of time.
So it looks like about 1/4 of the time spent loading a page these days is spent on the page and the remaining 3/4 are waiting for ad networks. There's got to be an opportunity there.
I previously worked at a digital agency and one of my final projects was building the website for a large TV channel. At one point I had to have a meeting with the ad network to go through the integration.
The technical people involved didn't know anything about how their products worked or the implications of integrating them. No idea if they were blocking or async.
At one point I was trying to understand what, if any, changes I might need to make to handle their ads that did full background take overs. "No no no, it's an expanding mpu" - sure, fine, what does that even mean!?
(Don't get me started on having to swap using js for switching images in the gallery for using individual pages because the onmiture page tracking numbers were the metric that everything was measured by for selling ad space)
/rant
You'd think with all the smart people that have put their minds to ads on the internet, there would be lightning fast, targeted ads that actually worked by now.
> You'd think with all the smart people that have put their minds to ads on the internet, there would be lightning fast, targeted ads that actually worked by now.
It's called Facebook Ads, and Google AdWords. Another name for what you seek is "product placement in movies". Yet another is PR.
For instance, looking at the trailer for the new Transporter, one has to wonder how much Audi paid to have that movie made.
And if you look closely at most journalism on the modern web, it is nothing but a thinly veiled ad for a thing.
I've got Google ads on my blog and they add 2.4s to a 4.9s load time. Google's own Pagespeed shows me that they [Google] are not doing basic things like caching immutable resources or minifying javascripts - which to me is really baffling.
Meanwhile on a commercial page I manage the second hint on my 94/100 score on Pagespeed is to "leverage browser caching" - all the resources listed, like api.js, are part of Google's analytics. Under minify javascript there are Facebook and Google resources (for Like and +1 buttons).
This is of course not a scientific analysis; instead I say it to illustrate that the smart people at Google and Facebook don't appear to be focussed on this end of things - I'm guessing the minify thing saves them resources, that they minify based on a balance of delivery costs vs. processing costs and that full minification is not worthwhile to them ... I'm just annoyed I can't get a 100/100 pagespeed score without removing Google's ads/analytics from pages!
> Google's own Pagespeed shows me that they [Google] are not doing basic things like caching immutable resources or minifying javascripts - which to me is really baffling.
If you give me a link I can file bugs and follow up with teams. This shouldn't happen and PageSpeed is right to call us out on it.
(The initial loader js you reference on your page does need to have a short cache lifetime so it can be updated, but everything else should be longcached. And everything should be minified.)
What would make things /really/ fast is optimizing an ad based around what brought a person to the page, rather than anything in particular about that person.
In other words, use referers to figure out if they came in based on keywords from a search engine (ads targeted more towards that, but still biased to the page viewed) or otherwise just based on the content of the page.
That's it; the first can be quickly memoised across a set of common terms for a page (and while doing so the faster content of page based results can be served instead).
No more silly tracking, just showing what someone is likely to be interested in at that moment.
I've worked on lots of ad networks and am now building my own. The sad fact is that most ad companies are not built by top engineers. It's an industry that has lots of shady international players with lots of fraud and politics. Even when there is good code, it's all usually focused on the backend with no thought about the end user experience. Add to that both buyers and salespeople who are clueless about technical details and this is what happens.
> The sad fact is that most ad companies are not built by top engineers.
This is an interesting thought. Working in a environment close to advertisement (travel recommendation engine), I have a few possible explanations on that.
- First, working in with a product that have no (or close to) added value for end users can be very bothering, and top engineers care about doing useful things.
- Second, to design a fast, simple and honest ad network you must fight your own business/marketing team (because they always want more data, higher click through rate, etc). Since they have most of the power here, they end up hiring more diligent engineers who won't argue about privacy, curated content and other stuff.
- Third, top engineers are working in ad companies, and secretly conspire to make the whole system collapse from the inside.
Ad companies are not typically considered "sexy". This is just an image problem because few industries drop you into such massive scale and engineering challenges from day 1. There is a lot of cool stuff to build.
It's not necessary to get computer science geniuses for this, and frankly I need people who get stuff done rather than trying to save the world through code. But the talent has been subpar because most of the companies outside of the top few (google, facebook, twitter) are smaller and not tech driven. They just pick up whoever can code up some simple server and get an ad on the page. Add in all the shady companies and it gets worse. Optimizing for the user just hasn't been a requirement.
We have a honest and fast ad network, probably the best you'll find. Quick, simple, static ads that are non-intrusive. And we actually perform quite well. It's not somehow at odds with the model, it's just harder... the question is, who's willing to work harder at this?
It's not failing... lots of doom and gloom articles but the industry is bigger and better than ever. It gets a lot of hate (for some well deserved reasons) but digital advertising is not going anywhere anytime soon.
Sure it's a nice test, and gives a keen idea of how much performance you can grab by using uBlock Origin.
However, what about pages _without_ ads? Am I going to experience a significant performance hit when running an adblocker on genuine websites?
This is important. Because, assuming the advertising industry is going to self-destruct tomorrow morning, am I going to experience a slower web if I let my add-on activated, and thus letting an opportunity for the ads to reappear later?
Would be interesting to have tested using a hosts file (like from http://someonewhocares.org/hosts/ ) instead of an ad blocker. Seems to work very well for me, and it doesn't cause the browser to use loads of RAM.
As long as you don't have a firewall blocking traffic to/from localhost, the TCP connection will be reset almost immediately (assuming you don't have an httpd running on port 80) so you won't have to wait for a timeout.
In reality, people want a page that becomes usable as quickly as possible. Specifically, people use hovers and scrolling to test that a given page is 'ready.'
That is very different from a page that fires DOMContentLoaded as quickly as possible. That metric is pretty much irrelevant to real people.
Which is part of why ads aren't quite as nasty as a lot of tech industry people make out.
When I start scrolling and the page seems ready to use, and then ads start popping in late and breaking the document flow in the first few seconds - you better believe I'm blocking them on your site.
It's both. It's a shared execution and the publishers haven't really put the work in either.
Check how well http://www.theguardian.com/ loads and you can see that it's perfect possibly to have plenty of ads while still having a great reading experience.
It's not really the backend that matters, these days its the front-end setup that makes the biggest difference. Less javascript, less huge unnecessary images, less "interactive" crap and streamlining the loading of content first then ads async can make a site load pretty quick. They also style the ad slot dimensions (since all banners are standard sizes) which keeps the content from moving around when an ad does load.
Its just simple effective optimizations, something that should be done but for some combination of bad talent, bad management and poor priorities, just isnt.
I suppose that's right. But it's also kind of a short term worry. I mean, don't you think mobile networks are going to keep getting relatively faster and relatively cheaper and will therefore be more relatively OK with a few ad-related keepalives?
Around this time in 1997, the PBS Newshour website[1] was an outrageous 113.7KB, which at the very top end of the GSM GPRS 2G networks of the time, 115 kbit/s, would take 8 long seconds to download. Why? Because of all of those obnoxious Sun Netra banner ads![2]
What do you get in 8 seconds on LTE today? Lord of the Rings in 720p. Or a whole bunch of Buzzfeed pages with some ad trackers, sure.
On the other hand, i just tried opening this page with adblock plus and it froze the browser tab for quite a while. Switched to ublock origin and at least the tab stays responsive.
At the end of last year, I spent a while working with an adtech firm, helping them figure out their product strategy. They all use Ghostery and I started using it too. Subjectively, my web-browsing experience improved massively, particularly page load times. It's nice to see that experience borne out by objective testing.
I often think that companies like Ghostery are missing a trick by not advertising themselves as a method of speeding up your internet connection because that's really what it does.
Well yeah, because Ghostery is a product of Evidon, which helps adtech businesses to get visitors to opt-in to tracking. It's a clever strategy, because they give valuable feedback about which ads and trackers are blocked for which reasons. If you can't beat them, outcompete them!
(I worked as CTO of an adserving platform and had to reject sales calls from Evidon on an almost monthly basis)
Next we need to find the best blocker cocktail. I use uBlock Origin, Ghostery and Flashcontrol on Chrome. Bit over the top possibly - can take a bit of trial an error to see what's blocking the thing if something you want doesn't load.
Why is uBlock somewhat frowned upon here on HN? From what I've seen published it's still a good solution - works well for me. I thought the original developer gave up the project and was being rather difficult about the whole thing.
Forgetting the accuracy of the OP, I cannot be the only one who doesn't care in the least that adblockers slow load times. Between my ISP's shoddy service, the popups and distracting animations adblockers stop, the net effect is a faster overall experience. A few seconds on page load is a fair price to pay for the increased privacy and security.
Nowhere near 95% of the web; I can attest to this because I disable JS on my default phone browser, and it speeds up everything massively.
Something may be broken on 95% of web pages (e.g. commenting or menus), but the main bit of the web - rendering hypertext and following links - mostly works.
Web applications less so, of course. But I don't think they're 95% of the web.
You're spreading FUD. Disabling Javascript by default is one of the easiest ways to reduce browser attack surface, improve page load times and to remove the most annoying ads. The vast majority of pages work (to some degree) without scripts.
If you really need scripting on a page, it's very uncomplicated to add an exception in Chrome: just click the "Javascript blocked" icon in the address bar, and choose "Always allow Javascript on example.com".
> improve page load times and to remove the most annoying ads. The vast majority of pages work (to some degree) without scripts.
In my experience using NoScript for the past year on all my desktops and laptops, this is at best only partly true.
Load times do definitely decrease, but this is dwarfed by the effect of having to reload the page for most new sites after re-enabling Javascript (since so many sites are unusable without Javascript[0]). In the end, it's a much slower experience browsing the web, not faster, and even with aggressive whitelisting, it's still rather frustrating.
I actually have separate Firefox profiles that I use for various websites that tend to be particularly problematic with extensions (like online banking), so I'm not even including those in this summary.
I wish using NoScript provided a faster browsing experience overall while providing better security, but the simple fact remains that right now, it does not. It provides better security at the expense of slowing down your browsing experience.
[0] So many static blogs now literally fail to display static text without Javascript!
Sadly, that's true. I'm wondering how many feel that this is not a good direction where things are going -or if it's just me being "old".
Why would someone want to use javascript on the client-side for things where it's not necessary?
Year after year, the browser and the web are turning into "something else", and while the possibilities are exciting, it seems the use cases where javascript is used for things that actually make a positive difference, are vastly outnumbered by the cases where it is used to re-implement in a more flashy, complex and resource-intensive way things that have been usable, performant and functional, 10 years ago.
IMO displaying content should be done just by HTML, using javascript if it's not to provide useful functionality, is wrong.
Whether it is browser HTML/CSS incompatibilities, flash atrocities or complex script bloat, it seems the popular web always manages to fix one problem and create another.
I used to run noscript but gave up some time ago. On your suggestion I tried just disabling JS in Chrome, and this does really seem to work pretty well. It's nice to have the blocking so well integrated into the browser.
It partially breaks 100% of sites that require the user to blindly execute scripts from unknown second or third parties before presenting a usable response, and 0% of sites that do not. At the risk of not finding a true Scotsman, I would argue that sites that do not give a meaningful response to an HTTP request as is, without client-side processing, are not making themselves part of the web.
When I click a link and see a blank page, I often click the back button or close the tab rather than allow it to run scripts. You do not need a script to display text. That functionality has been present since Sir Tim wrote version 0.1 .
Would it be so hard to provide alternate text for no-script browsers that says "Some content on this site is only viewable to users that allow scripts hosted by www.example.com and js.example.com to run on their browser clients."? No. It's not. There is no excuse for serving a blank page full of invisible scripts.
what justifies the usage of "sketchy" for these particular addons? Many of them are open source, you can audit them yourself. Or you could hope that the reviewers did a decent job.
The article's solid and substantial, it's the first I've seen to quantify the effect of ABP's acceptable ads policy. Also they have a view all button that works.
- you can't use the load event to compare advert-heavy vs. advert-free load times. The load event does not fire when the page is usable by the viewer, it fires when every last request is done, in particular even requests fired from various advert iframes. However, those requests may not impact page usability. The load event may even never fire - any offscreen 1px spacer gif that times out will cause that! I don't think there is a built-in event for "done-enough", but the load event is certainly misleading.
- opening the developer tools causes various side-effects; those can cause the page to load more slowly than usual. You shouldn't benchmark with developer tools open (unless you're explicitly targeting usage with devtools open...)
- comparing load times across browsers as reported by the browsers themselves may be valid, but it's not obvious. You definitely want to check that carefully.
- Measuring "peak" CPU usage is almost meaningless without considering how long the CPU is used.
- Measuring the chrome extension process CPU usage and memory usage isn't very helpful, because running this kind of extension causes CPU usage and memory usage in every content tab. Both of these statistics for chrome in this use-case are meaningless. You'd need to measure the memory usage and CPU-time of the entire chrome process tree to get meaningful results. Even in Firefox without e10s it's not valid to measure just the main process CPU and memory usage because plugins are in separate processes (and things like flash or h264 decoding can definitely use CPU and memory).
The only thing this page really makes a decent case for is that Chrome loads pages faster than Firefox - but even there, it's not clear we're dealing with an apples-to-apples comparison.