I recently completed a "software pilgrimage", as I decided to call it in explaining to friends why I did it, of writing my own DNS recursive resolver. I decided to implement it purely from RFC 1034/1035, with no library code aside from TCP and UDP sockets and a JSON config parser, those kinds of things. No library help for anything relating to the RFC. It's part of a homelab-focused DNS solution that I plan to release under the AGPLv3 once I dogfood it for a while. Basically "DNS for impatient homelabbers/SMB's who don't want to have to learn how to write zonefiles and don't want some crazy enterprise omnibus thing either".
Anyway, in the process of doing this and putting it in my DHCP config and seeing the traffic from two smart TVs of different brands, my RIPE atlas probe, various phones and laptops, smart thermostat, etc etc, I've noticed that there is a TON of "garbage" dns requests. Like, the chart [the article] shows - where about 70% of queries result in a name error - that totally meshes with what I see on a much smaller scale. Right now, prometheus tells me that since last restart (which was about a week ago at this point?), I've answered 80437 queries with NO_ERROR, 52014 with NAME_ERROR, and 242 with SERVER_FAILURE (funnily enough, when spot checking these, 8.8.8.8 also SERVER_FAILUREs these same requests - usually devices with presumably-buggy dns libraries not correctly specifying lengths of variable-length fields).
It really surprised my initial suspicions that so many DNS requests would be coming up with what I was originally considering to be an error condition. But I guess sometimes the absence of a DNS record is just as meaningful as the presence.
Incidentally, I also noticed these chromium dns requests, and they had me worried for a bit because I wondered if they were malware trying to exploit some kind of vuln in dns servers. Took a bit of googling to figure it out. I do think they make up a decent % of the name errors I see, though I hadn't gotten around to having prometheus split them out to measure.
That's a very sweet idea. What are some other "software pilgrimage" projects for ambitious hackers with a finite amount of free time?
Buying something in bitcoin using only the protocol spec and man pages? Tweeting from a Linux from scratch install? Writing a correct tar command on the first try? Writing a quine on your favorite language without google?
Heh. I was inspired by a movement I saw a while ago to read old CS academic research, since e.g. quicksort is still very relevant nowadays but is also still approachable for someone with a B.Sc. in CS. (Whereas more recent papers all require fairly deep knowledge of the research niche).
My take on it was that lots of this old software was implemented by candlelight with a magnetized needle, a 512KiB HDD platter and a steady hand; so I should be able to reimplement it in a modern language with modern tooling in a lot less time, while learning a lot about the system. Like, dig command output means a lot more to me now than it did at the start, and I now appreciate what articles like the OP mean now.
One of my friends is building a collection of such exercises aimed at experienced & intermediate programmers, where you build Redis / Docker / Git / React from scratch with guidance.
The project goals are not as lofty as those proposed by nishanth_v's friend, but Mr. Heaton goes the extra mile to turn each project idea into a step by step mini-curriculum with lots of extension points.
Then, he goes another extra mile by allowing readers to email in their buggy projects, and running a companion series where he teaches people how to debug / fix / improve their code by refactoring reader's attempts at the "...Advanced Beginners" projects.
Not quite the same as I'm not following or reimplementing any established standard, but it's interesting and fun to Greenfield a c++ project using minimal dependencies to do things like:
- invent and parse a DSL
- modularise components and connections between them
- possibly cyclic directed graph traversal
- efficient near-real-time audio processing (working in chunks within a time budget)
- multithreading / parallel module processing
- spit the outputs into an audio device and/or wav files
- eventually put a GUI on top
My personal pilgrimage seems to always end up designing designing a stack virtual machine and writing a compiler for it for a toy programming language. I tend to pick this up every time I learn a new language to get a feel for it.
I usually get it to run a simple loop to print hello world and do the inevitable Fibonacci generator and get bored. No fancy things like functions!
One of these things was crudely repurposed into a domain specific language for a job as well which was handy.
> Basically "DNS for impatient homelabbers/SMB's who don't want to have to learn how to write zonefiles and don't want some crazy enterprise omnibus thing either".
Workable but no, not suitable. PowerDNS has a lot of features and isn't terrifically simple to deploy. DNSMasq is easier to deploy, but still has a lot of features and isn't easy to configure.
The niche I'm aiming for is for people who just want to map hostnames to IP addresses. And note this isn't a commercial offering, just something I built for myself that I'm going to put up a website for and maybe some other people will find interesting. I do think I have some product-market fit because even among my dev friends, who are all quite talented, many of them have mentioned to me "yeah I threw bind9 on a server but decided I didn't hate typing IPs enough to actually set up and operate it".
I'm aiming for zero onboarding time and zero maintenance headache. You run a docker container, map the ports, and then there's a nice web UI that takes you the rest of the way.
Try running 15k containers, dnsmasq and powerdns fall over pretty fast. I have to run a BIND cluster of 3 machines to support the days when I am running at-scale testing of software (low load, but high instance count is a better fit for my work these days). I use 4th gen i7 machines as DNS resolver/caching servers, and using dnsmasq on the same machines I start to have issues around 400 clients, massive latency, broken responses, tons of crazy stuff. One BIND instance and I can get to about 7500 clients before it starts to get squeezed.
The difference is huge in other ways, though. Configuration, operations, and maintenance of the BIND cluster is 20 fold more involved, and when things break, they can break in really confusing ways. Such is the world of computing though, the better something performs, the more places it has where you can mess it up.
That sounds pretty different from the audience they said they're targeting → "impatient homelabbers/SMB".
As another commenter points out though, the load you're talking about definitely sounds like something PowerDNS should handle without much problem. Given reasonably spec-ed, hardware and software that is.
I'm surprised 60 billion DNS queries is only 41% of the daily norm. With a billion people between the US and Europe alone - the vast majority of which use the world wide web directly multiple times a day (nevermind other online services) - and the amount of domains a typical website loads resources from, I figured daily DNS queries would have broken into the trillions by now.
I suppose that's probably thanks to caching, dedicated apps for many websites, and most users sticking to a relatively small selection of websites.
Well, probably (at least) half of those billion people are using one of the large public DNS servers (Google, Cloudflare, et al.).
Plus, the answers returned by the root servers more often than not include resource records with very high TTL values (e.g., NS RRs with a TTL of two days). These then get cached for that long by the recursive resolvers that are used (directly) by end users.
The root servers aren't responding to requests for the A RR for google.com from Joe Schmoe or the MX RR for gmail.com from Outlook running on his desktop -- both of which (without checking) likely have TTLs measured in a two- or (at the most) three-digit number of seconds.
Can someone explain why this couldn't have been implemented purely client side and validating if what was entered was first even a valid hostname? Or were the DNS queries only run after they passed an initial check?
I think this is one of the key problems that emerged by merging the search functionality into the location bar. I still now always enable the separate search bar for Firefox and avoid running searches in the so-called Awesome bar. I still consider a search and host lookup very different operations in my mind.
I believe it was mainly to detect when a user is trying to navigate to an intranet site with a single-word name (for example http://fileserver). To do that, Chrome needs to make a DNS request for the word and check if it's valid.
However, you can have misbehaving ISPs that replaces all NXDOMAIN responses with something (producing false positives for what constitutes "valid" hostnames), or a guest portal that is hijacking DNS responses, so Chrome will make requests for random DNS hostnames that are unlikely to exist to detect if DNS hijacking is occurring and disable the intranet detection if it is.
If that was the goal, it does a pretty poor job. The amount of times I've accidentally Googled things like 'internalserver', 'fileserver:80', or 'dockerhost:8000' over the years is way too high.
And when you do eventually reach any of them via manually prepending 'http://', they make the 'http://' not visible in the bar. What confusing signaling!
Hint for Chrome: if I'm appending a port and the DNS name exists I probably don't want to Google it.
> If that was the goal, it does a pretty poor job.
Their goal is to treat the query as search, and then try to detect if a host named "internalserver" exists. If so, Chrome displays a banner saying "do you want to visit internalserver/". So they don't have to delay the search until the probe finishes, thus it never automatically make it work without another click.
Personally I don't like it either, so I usually type "internalserver/" (with a tailing slash) to skip the search.
I've searched for internal servers so many times I've just gotten into the habit of suffixing with '/' so 'internalserver/' goes to http://internalserver/
I took a look at the sqlite db but I didn't see anything that stood out but I might be looking at the wrong file. Is it databases.db? You can get that tab-complete behavior with custom search engines though
Keyword: l, URL: localhost:%s
Then it works how you're describing with l<tab> and input the port. I personally define all the different versions of my servers that way so v1<enter> goes to something like internalserver:9001, v2<enter> goes to internalserver:9002, etc.
This. I understand why some people don't know (or care about) the difference between the search box and the address bar, but why engineers accept the merged input is beyond me.
How does it save keystrokes? I press ctrl+L to enter a URL or ctrl+K to enter a search, the same number of keystrokes regardless of which box I want to focus.
I was going to argue that it doesn't save keystrokes when I frequently have to go back and fix the bad guess. But then I realized that you're right - the time saved over the years when I actually do want to Google a specific term likely *far* exceeds the time I've spent fixing the bad guesses.
I guess I'll just accept the flaws that come with it.
The APNIC post[0] backs this up, saying it's the "Intranet Redirect Detector". [1] has more details on how that system works (which is basically what you said, but slightly more in-depth).
Worth noting that this doesn't actually affect the behaviour of a single user at this point in time.
> Thanks, Matt, that's great to see. Note that that's the effect of the change in comment 37, which only affects Android. The desktop change is blocked on a planned experiment in M88. A wide rollout probably won't happen until late February or early March, at which point we expect an additional reduction in root traffic. [0]
They were doing the DNS interception check on Android, but a different path of code was actually doing the checks (in a different way). Android was basically sending out useless requests.
Which amazes me. Chrome does this check every time network settings change, including a new IP. So your Android phone connecting to various networks just spammed this stuff for no reason. The fact that this fixed dropped root server requests by nearly 50% is astounding.
Normal users don't enter hostnames directly any longer. Even for the most common sites like YouTube.com, Facebook.com and Amazon.com they will largely search the name and then click on the top search result.
This behaviour actually suits Google well, because their incredibly lucrative "promoted results" are shown before the real destination site. You can't charge an advertising tax on users who browse directly to their destination.
Can you point to the specific piece of client side code that does this? It doesn’t sound like it inherently fixes the problem of either going to “myserver” (as in, http://myserver/) and “myserver” (searching the web).
So is this functionality just "gone"? I remember one of the original reasons for this functionality was to know if you were on an ISP that performed any kind of DNS hijacking (I don't fully understand the why of it, but that's separate), and I'm curious if that's still available and if so, where are those queries actually going now?
At first blush the fact that all these searches were sent over DNS (which is plain text by default) this was a gross privacy violation. Congress has passed bills allowing ISPs to use your DNS and browsing history and sell it to 3rd parties [1].
With all these searchs being included in DNS queries chrome was basically allowing everyone who wanted that data that you were worried about that mole or that you have that weird fetish.
Anyway, in the process of doing this and putting it in my DHCP config and seeing the traffic from two smart TVs of different brands, my RIPE atlas probe, various phones and laptops, smart thermostat, etc etc, I've noticed that there is a TON of "garbage" dns requests. Like, the chart [the article] shows - where about 70% of queries result in a name error - that totally meshes with what I see on a much smaller scale. Right now, prometheus tells me that since last restart (which was about a week ago at this point?), I've answered 80437 queries with NO_ERROR, 52014 with NAME_ERROR, and 242 with SERVER_FAILURE (funnily enough, when spot checking these, 8.8.8.8 also SERVER_FAILUREs these same requests - usually devices with presumably-buggy dns libraries not correctly specifying lengths of variable-length fields).
It really surprised my initial suspicions that so many DNS requests would be coming up with what I was originally considering to be an error condition. But I guess sometimes the absence of a DNS record is just as meaningful as the presence.
Incidentally, I also noticed these chromium dns requests, and they had me worried for a bit because I wondered if they were malware trying to exploit some kind of vuln in dns servers. Took a bit of googling to figure it out. I do think they make up a decent % of the name errors I see, though I hadn't gotten around to having prometheus split them out to measure.