I found DNS ad-blocking solutions to be pretty lackluster and lots of ads were still getting through. With uBlock Origin only sites ahead of the curve were getting their ads through (porn sites, facebook, etc.). Couple this with Bypass Paywalls[1] browser extension and the web is pretty usable.
I also tried to go one step further and setup mitm-proxy to man in the middle all of my traffic to see if I could do more invasive but thorough ad filtering. Certificate pinning from the likes of instagram, facebook, apple, and google really stymied this approach. So all in all, I don't see much benefit from DNS adblocking instead of ublock origin.
I count my site to be ahead of the curve in a different direction in that we sell and host our own advertising (static jpgs) and do not use any 3rd parties for ads. This helps us control the ads content as well as get to keep 100% of ad revenue.
The only 3rd party thing we use is Google Analytics and a Google font, but the site still works fine when users block them.
I really wish more sites did this... I know that for sites that don't get, say 1M views a month, it would be difficult to match the ad networks... but for even a million views a month, it's probably better to get dedicated/approved ad support and building relationships with vendors than to try to poison your own site with ad networks, often blocked.
This is the best thing for the web, I think. It's doesn't seem as good for advertisers in terms of utility for targeting, but on the flip side, I for one am more willing to support sponsors when they have a direct relationship with the content provider.
You don't really need advanced targeting analytics when your content is narrow enough and draws a particular type of reader.
In our case, readers are likely interested in office design/office furniture products because all we do is publish office design project images and information.
I have an idea I'm hoping to work on in the coming years which aims to move the web back in this direction now that I've had some years of experience doing it.
I found the big blocklist collection to really help augment the lists of blocked URLs. https://firebog.net/ If you login to the web UI of your Pi-hole instance, you can easily add more lists in the settings -> blocklist tab. Specifically the "ticked" lists: https://v.firebog.net/hosts/lists.php?type=tick
It's not just ads, if you're concerned about privacy then DNS-level blocking won't cut it. For example, you can't block cookies or third-party scripts via DNS blocking.
> So all in all, I don't see much benefit from DNS adblocking
The benefit is as a tack-on for a home network for devices and traffic that doesn't go through a web browser. E.g. for mobile apps connected to the network. But once again, as far as privacy is concerned, that won't block e.g. Facebook SDKs embedded in apps unless you block the relevant domain entirely.
I run those same plugins on all my browsers and I'm still blocking between 30-60% of total requests for tracking beacons and other privacy-invading calls that get past the plugins.
The rationale stated for this work is preventing ISPs from being able to monitor and potentially sell information about Internet usage, which is reasonable and worthwhile. But by hosting a VPN with a third party, haven't we simply reassigned the same responsibility to someone else rather than absolve it? Is Digital Ocean more trustworthy than, say Cox Communications? How is this risk to be calculated, especially by a layperson?
I believe low-latency anonymizing networks like Tor might be a better more suited for accomplishing the task of obscuring one's own network traffic. In fact, I'm typing this comment from Firefox with uBlock Origin configured to use a Tor SOCKS proxy which is always running locally - eliminating ads and making little attributable netflow in my wake.
> Is Digital Ocean more trustworthy than, say Cox Communications?
Dunno about Cox, but I promise you I trust Digital Ocean far more than I will ever trust Comcast or AT&T. Even if they didn't have a history of being bad actors (and they do), a lot of people have exactly one choice of ISP but dozens of choices for hosting in the cloud, so the incentives are much more favorable.
Most of these residential ISPs that people hate are also major (T1/T2) upstream providers for companies like DO. If not already, at some point, these providers are going to just sniff VPN traffic straight off of their backbones.
Say your ISP is Comcast... If Comcast knows your are connected to some VPS via VPN, it's likely that anything coming out of that VPN is yours. And if Comcast (or some subsidiary or partner) is also the upstream provider for that VPS, they could pretty easily make some correlations.
This isn't really true. The wholesale networks are operating at speeds that make this kind of sniffing impractical. I'm not saying they don't ever siphon traffic for LEO or other reasons, but not for privacy-violating/ad-targeting reasons. DO probably is running 40gbps alone with each transit provider, plus sending traffic over peering circuits, so it's just way less practical.
So what you are suggesting isn't actually true, and it's hardly hypothetically even possible.
Totally agree and I considered the shear throughput making it impractical. And while it's impractical, my point is that it's still possible, and maybe even practical in the future.
Maybe I misunderstand by why is this considered impossible? Network processors read data from a packet, make decisions based on that, and then rewrite arbitrary parts of the packet. All on the fast path. This sounds doable if you had the custom firmware that did that. It'd just be a huge waste of money considering your very, very expensive network box.
Ahh so maybe it's because the packet, once out of the VPN and heading to it's destination, is (normally - it is 2019 after all!) TLS encrypted so you can't just modify the payload like I said. Fair enough.
I'm counting on greed. We know their business model depends on money from people who at least think they're not selling their traffic info. One bad rumor and evidence that they're selling the info to google and their business is dead.
Based on my own experience, having dealt with both companies a fair amount, I trust Digital Ocean to be vastly more competent than Cox. That doesn't stop DO from doing something nefarious, but negligence and incompetence probably account for more data breeches than any other cause.
I connect my mobile clients (smartphone, MBP, ...) via WireGuard through my router. You never know what WLAN or mobile network you're on. The router serves Pi-Hole and DNS over TLS. My ISP/ASP (LibertyGlobal) I can trust for now.
Tangentially related: The best feature of FireFox Focus for iOS is that it also works as a free, local-only (no VPN routing) ad-blocker for Mobile Safari. So, you can install it, never actually run it, and it makes Safari so much more usable.
Seemingly every other week for months now a Pihole post makes the front page on HN. Every time I wonder why. IMO, it's just a DNS black hole with a slick interface.
Before adblockers came along I had a script that updated my hosts file. I then moved to a DNS black hole but it’s been more than a decade since I’ve used either solution.
Do you people have that many hostile IoT / Smart thingies connected to your networks? Are you just unwilling to pay for the ad-free versions of apps. Are you using apps/services on these devices that don’t offer an ad-free option, if so why? I’m genuinely curious.
> IMO, it's just a DNS black hole with a slick interface
This is why it gets to the front page.
It's a DNS black hole with a slick interface.
You run it and it does great by itself, manages the updates, and when it does do something you don't want (or vice versa) there's this really slick interface for figuring it out and correcting it.
We underestimate how much slick interfaces are worth, especially when they make a chore that was almost entirely CLI driven and making it a non-chore for a bigger audience.
This is a thing a lot of engineers don’t seem to get. Slick interface is the most important thing for public, no one wants to use terminal or advanced settings to actually do anything related to your product.
My comment also states that since adblockers have come along a DNS solution seems a bit archaic. Adblockers are even more slick and user friendly so your reply doesn’t at all answer, why? Why move back to a less than solution?
Lack of device control, unknown alternatives, unwillingness to just say no (for whatever reason… not criticizing here) seems to be the answer.
Unfortunately, as ignoramous states there are techniques that will render DNS blacklisting useless if they want to.
IMO, it's just a DNS black hole with a slick interface.
And something, something some rsync and ftp and you've got Dropbox. Yes, Pi-Hole is just dnsmasq with a pretty face, which is precisely why I use it. $50 for a Pi starter kit, and as soon as it hits your mailbox, you are about 20 minutes away from living the #adfreelife (and most of that 20 minutes will be redirecting your network after install. Where the hell are the docs for this router?).
Sometimes I'm content to manually tweak JSON files all evening. And sometimes I just want to plug it in and pretty much works out of the box. Ad filtering on my network falls into the unsexy latter bucket of "just give me something that requires a minimum of yak shaving".
Are you using apps/services on these devices that don’t offer an ad-free option
Yes, the NYT as one example. The app still has ads. I continue to pay for the NYT to support good journalism. I don't get to pick both, so I choose to continue to pay.
A device in my house went nuts and decided it needed to ping an NTP server 1K a night. Not anymore.
In the end, I kind of get the impression you're spending more mental energy on arguments against, rather than ask yourself why someone might find it useful. I could come up with quite a list of reasons with just casual thought.
> A device in my house went nuts and decided it needed to ping an NTP server 1K a night. Not anymore.
Wait, Pi-hole was your solution here?
> In the end, I kind of get the impression you're spending more mental energy on arguments against, rather than ask yourself why someone might find it useful.
I’m asking because I believe there are better ways. I could be called out for baiting or pushing a “the only way to win is not to play” for IoT and creepy apps/services agenda. Yet, NYT, FB and many others can still be viewed and signed into with a mobile browser. The experience may not be as nice but it still works and sends the right message to these corps. When creepy app/device/service is the only option I recoil and reassess.
Besides, DNS blacklisting isn’t perfect and requires a fair amount of tweaking depending on how many thingies you’re using it with. Any compromise you make for one affects all others. I think we have the same goal of “having your cake and eating it too” just different methods. Either way, we’re both expending constant energy and compromising.
In my experience, the OpenWRT adblock package is not as powerful as Pihole. It's also annoying (involving some awk scripting) to add new lists that weren't added by the dev.
MT7621-based devices can all handle GbE and are supported by OpenWRT. Xiaomi among others sell routers based on this chipset, but personally I'd avoid them due to their locked-down bootloaders which complicate flashing. I myself purchased the YouHua WR1200JS, which is currently available rebranded on Aliexpress for under $35 USD as model "WR330" with OpenWRT pre-installed (although you should still reflash to be safe). It saturates LAN ethernet (~936mbps avg.) in my tests, but probably needs software/hardware offloading for Gb WAN, which I think is only enabled in snapshot versions of OpenWRT.
WLAN throughput is not so good, about 288mbps at 80mhz, but this was only one client device tested and not thoroughly because I don't know enough to diagnose bottlenecks. Regardless, this is more than enough to saturate my WAN connection - no issues with 6+ simultaneous wireless devices.
If these increase in popularity [0], I'm pretty sure DoH will be the goto workaround for web apps and native apps, alike, which would be unfortunate because DNS based ad-blocking is all encompassing and takes very little effort to setup [1]. That makes me wonder why DoH was even conceived, if not for the benefit of ad-networks [2]?
Thinking along similar lines, can't help but wonder if cert-pinning does more harm than good.
I don't see how DoH can be filtered at the firewall at request-level, since it looks like regular HTTPS traffic. Of course, MITMing HTTPS and then blocking particular DoH reqs and letting rest through would work, but apps that pin certificates might make MITMing an uncomfortable ordeal.
Blocking a DoH provider altogether might not be feasible.
Slack is just ICQ with a slick interface. Dropbox is just rsync with a slick interface. Sometimes a slick interface is the most important feature of a product.
> Are you just unwilling to pay for the ad-free versions of apps.
Note that paying and hiding ads doesn’t mean the app stops talking to the ad server. I had one app which pinged the Google Ads server even after paying (not going to name & shame as it’s a small independent developer so I’m leaning towards it being a legitimate bug).
Oh and don’t forget analytics which paying doesn’t work against at all.
Almost 60%... I don't install many apps, I use Firefox with Ublock Origin. Most of the blocked requests are to Google or Facebook.
At home I have it network-wide, and typically the block percentage stays under 10%. Until my partner opens his Windows 10 laptop, then the block graph goes up. Also my television talks to advertiser trackers (LG), which I can easily block from Pi-Hole.
Why it's better than just a hosts file? One is I can easily whitelist/blacklist domains from the UI or I can just disable all blocklists if I need for any reason. I also like the statistics it gives me.
I've picked up a few things that were making a crazy amount of requests. I don't know what Alexa is up to but over 1000 requests to device-metrics-us.amazon.com blocked each day
I have three machines on my network that are used for web browsing, all running uBlock and Ghostery. Two phones that aren't used for much internet-wise. A couple other devices such as a HTPC.
I don't use Pihole, but I do use something similar for pfSense. I rarely mindlessly-browse the internet and I don't have any IoT junk. That said, my stats for 30 days:
Note: The top three are various IP blacklists (99% of blocked is ingress). The bottom three are DNS blacklists.
It blows my mind every time I look at these stats and see how much they've increased... The data these companies would otherwise have on me. The data these companies have on everyone else. How much has actually gone through / missed / not blocked and rendered any of these efforts meaningless.
At the end of the day, I don't really care, but it's all pretty neat!
> Are you using apps/services on these devices that don’t offer an ad-free option, if so why?
Maybe the HN crowd avoids Instagram, Snapchat, and Facebook but most people don't, and can't unless they want to socially cut themselves off from parts of their social circles.
It's about being in control, not ads. My network belongs to me. I get to say what data leaves my network. I get to choose who I support by leaving ads on. I get to choose who gets blacklisted because they take advantage.
It's so ridiculous because instead of wasting money buying unnecessary hardware people could just use already existing DNS servers that do the same thing.
I block ads in my home because it's just a nicer web experience for everyone. Plus when I play games on my phone I don't get a barrage of ads every time I die.
Pihole is much more configurable: you can whitelist or blacklist domains with ease. However, Pihole can also be run on existing hardware like a home server, if you don't want a standalone pi.
I use https://github.com/dan-v/algo fork which has Wireguard VPN and PiHole combined. It takes minutes to spin up a Digital Ocean VPN and have it working on all my devices. I'm very happy with this setup.
Pi-hole dashboard is quite useful to see what's being blocked and add new domains/lists easily. For example, I also add all facebook domains (https://github.com/jmdugan/blocklists/blob/master/corporatio...) and sometimes Hacker news when I want to be productive.
algo is great for an automated setup of a secure Wireguard(and IPsec) server with ad-blocking capabilities. DNS adblocking is necessary to block tracking in iOS apps. Content Blockers only work with Safari.
I setup a similar system but with IPSec (https://github.com/jawj/IKEv2-setup) and Pi-Hole on DO. The best part is that the linked IPSec setup is trivial to install and also generates profile files that leverage the OS VPN capability in any iOS device without needing to install extra apps (and also force VPN connectivity by default so you don't need to remember to enable it)
I wrote a couple of bash scripts to easily configure WireGuard server and hosts. Automatically generates keys and puts them in correct configs. Adds client info to the server config as an option. As a bonus it can configure some iptables to enable NAT, vpn tracking, etc.
I did something similar by installing Wireguard as part of Streisand and then PiHole on a VPS. One caveat was this combination accepted public DNS queries by default. You would need to block it on your own. Otherwise the experience was good for various connection scenarios and adblocking was a breeze.
Now I am using Algo + Steven's hosts files for the similar idea. No complaint thus far yet.
I setup PiHole and removed it about 2-3 days later. UBlock Origin is perfect for laptops, but I wanted to see if it'd block YouTube ads and similar on my Smart TV and mobile devices - it didn't. If anything it just caused me grief by interfering with non-ad web services, so I canned it and everything started working again.
I did. I should point out I work for an ISP, so my day job is managing proxy farms, DNS, and so on. It just didn't seem to stop ads on the Smart TV with any variation of whitelist/blacklist.
You can use an iptables rule on your router to rewrite the address to your custom server, which I have done specifically because Google devices were ignoring DHCP.
That still doesn't fix the ad situation. There's still stuff that goes on at the JS level that a browser ad blocker catches, but a DNS blocker won't. That's been my experience after installing Pihole on the home network.
I wish PiHole could be more fine-grained. Being able to respond differently depending on which device made the request would be an amazing addition.
For example, I want to blackhole all X-related stuff because I don't use service X and don't want them to track me, but my girlfriend wants to access and use service X. So either we each get an instance of PiHole and tailor it to our specific needs, or we share an instance and one of us is unhappy.
For me it blocks most of the time. Sometimes I notice the advertisement coming directly from Youtube servers and passing through. This only seems to happen when using native YouTube clients...
I'm using this setup as well (I have Ansible to do it), but for some websites those use Akamai CDN will block you if your exit IP is from well-known networks like VPN providers, AWS, DO, etc.
Yes this has been my trouble as well, to the point that sometimes I switch off the VPN with piHole and just rely on a commercial VPN that rolls IPs + adblocker software for those occasions. Imperfect, and expensive comparatively
I'm surprised that commercial vpns don't offer an ad blocking option; from their perspective, it's not only a great feature to market, but it reduces their bandwidth costs.
I also tried to go one step further and setup mitm-proxy to man in the middle all of my traffic to see if I could do more invasive but thorough ad filtering. Certificate pinning from the likes of instagram, facebook, apple, and google really stymied this approach. So all in all, I don't see much benefit from DNS adblocking instead of ublock origin.
1 - https://github.com/iamadamdev/bypass-paywalls-chrome