This is an interesting little example of how "systemic" happens, or can happen.
The list exists initially to support a feature, a utilitarian reason. Then, being in the list has SEO value or some such. The list becomes a legitimacy or authority test. It affects traffic, sales.
Maybe other stuff piggybacks this list. Anyone who needs a list of important domain names uses it. Ordering search results, caching, testing, crawling... More dynamics unrelated to Apple's original purpose.
Meanwhile, Apple continues to treat the list as a relatively unimportant thing... Just a clunky little hack that enables autocomplete. At some point, willfully ignorant of a whole industry of professional services out there promising to get you on list.
Yes, nice to see we've come full circle at last from Karp, Mills and
Postel maintaining the official HOSTS.TXT throgh the "free internet
era" and back to a nice sanitised, official list of what's on the
internet.
In summary: I don't have any indication it's in current use. I saw this URL in intercepted traffic from an iOS 13 device. I suspect it was used at some point but not anymore. I bet iOS 15 still uses something similar but doesn't use this particular URL (perhaps one with more obscurity/protection).
It's a binary file, but if you look at the hex dump there's a list of vaguely "bad" words near the end including "whorehouse," "Zipperheads," "unabomber". I have no idea what format it is, though.
My competitor is in this list and I'm not, even though my site is more popular. There even are some small Dutch campings, municipalities and regional football clubs on this list. It's a strange list for sure.
Which is the cause, and which the effect? Was apple's incorrect autocorrect causing them to go to the wrong place initially, or did they autocorrect list get built from the incorrect attempts?
Periodically searching this list for purchasable domains seems like something scammers would be very very interested in. Honestly, apple should be making sure every one of these URLs goes to a live website and that the website is still exactly the website they intended it to be.
One of my old domains is on the list. It was a relatively popular blog years ago, once it expired it immediately got picked up by a spammer. This list is garbage.
So I guess the question is, what is the alternative?
The only realistic option I can think of is some combination of:
• Make autocomplete operate on a blacklist instead of whitelist, with a more limited goal of only removing e.g. known porn sites.
• Make the list of potential matches machine-generated, without human intervention. (Aside, are we sure the current list isn't just the 250K most-visited sites on the internet, or something like that?)
Either of these would remove culpability since it's no longer a curated list. And yet, would that make it more safe in a meaningful way?
Why should Apple avoid autocompletion of porn site domains? That’s an area rampant with scam sites, especially similar domains. Plus “everyone” looks at porn so the benefits would be widespread.
If Apple wants to protect users then autocompleting porn site domains seems like the place to start, not avoid.
The absence of porn domains raises the question of Apple’s intent.
> Why should Apple avoid autocompletion of porn site domains?
Because if I’m sharing my screen on a business call, and I start typing something into my browsers’s address bar, I don’t want it to autocomplete something nsfw which just happened to share the same first letter.
There's no such thing as "machine-generated, without human intervention". Even something as seemingly simple as "most-visited websites" involves measuring choices. (Fundamentally, this is indeed about responsibility. Until a non-human gets some form of citizenship, they have none.)
Furthermore, we now know that in practice "machine-generated" seems to be even worse, because too many people are fooled by the "the machine did it" 'excuse'. (Like you seem to be doing here ?)
Billions in quarterly revenue doesn't allow Apple to solve the halting problem. I can't begin to imagine how they would do what you're suggesting. They need to detect when a website changes in kind, but ignore day-to-day changes or normal UI revamps.
I'm honestly unclear on how you can't see how Apple could solve this with billions. It is definitely a "throw money at it" situation, no question.
Moderation is a hard problem because it isn't just a matter of someone filtering between the polite posts and the less polite posts, it's a matter of filtering between the polite posts and the content that will sear your soul, no joke.
But that's not what this is. This is just, is the website still there and look correct? With the correct software setup it's roughly a person-month by my estimate to gets eyes on every site in the list.
(Though most people usually don't set write this sort of software very well, making someone laboriously click this, scroll around some, click some more, click a tiny radio button, click the tiny submit button, wait for the next thing to load, etc. It'll be longer & more work with this style. Someday I hope to have the chance to write some sort of classification program and implement the UI I've wanted for a while, which amounts to "right -> ham, left -> spam", and everything as pre-rendered as I can get it before it gets to the human. I'm sure some people out there have done something like this, but it makes me honestly sad how few I've seen.)
There is a service called Visualping that basically does this. It takes a screenshot and sends you a “diff”. You can set it and say by what % things need to change.
They could use a similar tool plus human review to maintain the list.
mailboxapp.com (long dead/euthanised) is there, but mailbox.org (kicking) isn’t.
It might be a list that was made by each company request? Based on some form or authentication or as such? And the companies who didn’t approach Apple aren’t there. Just guessing. But I can imagine Apple doing such hacky things, though most of those remain behind the orchard curtain.
Judging by the quality of some of the websites the chance that their webmasters / marketing teams will ever know the existence of a list like this is close to 0%.
Neither I nor my largest competitor are on the list, which is fair enough. However a now-defunct domain which my competitor previously used for their hiring portal is on the list. Weird!
You should be careful with the cron intervals because GitHub eventually just disabled one of the actions that I wrote that worked exactly like this. I was generating a list of the YouTube's dynamic ad-serving domains.
That’s actually a bug, thank you. It will break as soon as the last modification to the data file is more than 20 commits behind HEAD. I used 20 because it made the workflow function (it was working on my local machine but not when I pushed to GitHub because the default depth is 1). I need to update it to check for the last commit that modified the json file rather than searching through the entire commit history.
Ha! This explains why I always get "cookieandkate.com" suggested when I want to visit another site that starts with "cookie". I was extremely frustrated that it suggested that site with higher priotity than the one I actually visit mostly every day. And I had never myself gone to that site.
That's very strange, you'd expect Apple to provide a good UX by suggesting a website from your bookmarks, history and then their list. I wonder whether that's an oversight or an intentional implementation.
Safari does use your bookmarks and your history, but at some point the history expires so some sites I go to rarely don’t autocomplete and I add bookmarks when I notice.
Also it sometimes is too slow to pick up that I don’t want to go where it autocompleted, I hit go, then immediately hit back and typed more characters in to get the actual site I wanted. I could go and edit my history, but that process is a little clunky on my phone.
Interesting side-note: there are only two *.ai domains on this entire list:
1. "tempo.ai", which doesn't actually resolve
2. "web.ai", an http-only site that looks kind of like a cross between a geocities website and a parked spam page, with the content allegedly last revised almost 20 years ago (2003)
Fascinating. I know of an organization that abandoned (other than redirects) their original domain name about 4 years ago and adopted a new domain name. The old domain name is in the list (with about 30 subdomains!), but the new domain is not in the list at all. This makes me wonder how (if) the list is updated.
hah I see we both discovered the same lists! Did you get these from traffic interception too? I didn't find any hits for these on Google but not surprised you beat me to it of course. I'm a long time fan of your apple-hacking blog posts.
Also I just realized that the source of all these doesn't require auth either and you can just view it here: https://api.smoot.apple.com/bag (but requires a spoofed user-agent if you use cURL)
I noticed that tiktok.com is missing, so this list might have been compiled sometime before 2019. I wonder if you could write a script that pinpoints the exact month or even day.
This attempts to scan WHOIS records for random domains in the list and try to find the most recent "registered at" date for a domain.
It's very unreliable. The whois-parser Ruby gem thought that giants.it was created on 2020-11-03 09:00:01, but I checked the record and it looks like it was actually created on 2009-09-28 11:00:43.
It looks like nature-in-art.org.uk was first registered on 21-Jul-2019
I don't know if this would provide any useful information. It would take at least a year for a brand new domain to make it into this list. And I don't know if the "created at" date can be trusted, maybe it gets reset whenever a domain is transferred, or lapses and then gets renewed.
In conclusion, I don't think this idea works at all.
This an interesting question. Skimming the list it seems that - at least - it is not based solely on popularity (like the Alexa 1 million list for example).
I see lots of websites for small German towns, which should only have a couple of visitors per week.
Of the blogs and personal websites I read - some popular, some less so - only paulgraham.com is there.
I suspect the list is biased to official and very old domains.
My hunch was that it originated well before iOS, and could be traced back to forking KHTML, but if that hunch was right I can’t find any way to validate it.
Agreed. Some of the domains there were popular 10-15 years ago, but are now relatively obscure. Meanwhile, high traffic domains that have emerged over the past few years are missing. Compare "pricerunner.com" and "tiktok.com" for example.
I've found wiibrew.org and hackmii.com in there which are both Wii homebrew sites that became popular around 2008/2009 and probably declined in popularity starting in ~2012/2013.
Then there's also wiiu-developers.nintendo.com and wiiudaily.com which probably didn't exist before late 2012 or early 2013 when the WiiU was released.
The strangest thing about this list for me is that it contains multiple .onion addresses. It also seems to have been at least somewhat filtered for "objectionable" content (porn etc.).
I assume it must have come from user searches at some point?
My guess actually is that iOS/Apple no longer uses this particular list. I found this by intercepting traffic on an older jailbroken iOS 13 device. Given how old some of the domains in the list are, I'd guess that iOS 15 no longer uses it. I also never actually saw my phone fetch this list, I just saw this URL in another configuration response.
tl;dr; I bet it'll never get updated. (but some similar list at a more hidden/protected location likely will be)
Oddly enough, tier 1 cell carriers that have agreements with Apple aren't immune to being forgotten by Apple.
The list has "orange.co.il" (now NXDOMAIN) but not "partner.co.il". Partner (one of 4 main cell networks in Israel) used to be called Orange but they terminated that agreement with Orange FR years ago (which I'm guessing is why they can't redirect "orange.co.il" either).
I'm curious if this control over auto-complete can help direct users to websites that are favored or approved by Apple, and eventually change people's opinion of which sites are "good".
In a way it's a slight step towards allowing or disallowing visiting certain websites. Is the owner of the phone allowed some control here?
Could be further immortalization of Oliver Smoot, “a fraternity pledge to Lambda Chi Alpha, who in October 1958 lay down repeatedly on the Harvard Bridge (between Boston and Cambridge, Massachusetts) so that his fraternity brothers could use his height to measure the length of the bridge, which summed to "364.4 Smoots ± 1 ear"
Raised tariffs, not taxes. It's an important difference, doubly so with regards to Smoot-Hawley because the whole point was to protect American companies (and therefore jobs) from foreign competition (of course, it did not work...).
I have the very strong feeling that this is is powered by https://register.apple.com/placesonmaps/ – at least as a hint. As soon as you have officially registered your Apple Maps entry (with an website URL) that website gets on the list.
e: This is reinforced since we updated our domain/URLs from a ccTLD to a gTLD not long ago, also claiming the Maps entry, and the list reflects the gTLD domain!
Ugh, I guess that's why the iOS browsers all suck so hard with autocomplete?
I always long for Firefox's url bar, it easily beats everything else since Firefox 2 came out or so. Nowadays you have to un-configure two layers of "let's put a search bar in your URL bar" but then it's still just amazing. Especially compared to google, which obviously doesn't have an incentive to have you skip searching on google.com... and apparently apple doesn't want to you complete just any old urls, but only apple approved ones?! This seems like such a useless kind of (soft, but still) gate-keeping...
> Ugh, I guess that's why the iOS browsers all suck so hard with autocomplete?
No. Contrary to popular belief, browsers on iOS aren’t all Safari skins. They all use WebKit, but there’s a vast amount of functionality in a web browser that isn’t handled by the rendering engine. This is one example.
Non-Safari browsers on iOS are free to use whatever address bar implementation they like. If Firefox on iOS has a crap address bar, that is 100% down to Firefox.
>No. Contrary to popular belief, browsers on iOS aren’t all Safari skins.
I know some people hate Apple for both good and bad reasons, but it's pretty wild to me people think Apple does this. I mean right out the gate that would be pretty illegal, no?
It would not be illegal. In fact the iPhone launched with only Safari.
Because Apple is selling the hardware device, they can decide what software they ship on it. This is not specific to phones, it’s true of all hardware. There’s no feature to load arbitrary browsers onto a PlayStation, for example.
I specifically meant letting you download, say, Google Chrome, and it's not actually Chrome but Safari. Unless I'm misunderstanding what that person described, which is totally possible.
Every browser you use is forced by App Store policy to run off of the system WebView, which is Safari...other features being part of the browser doesn't change being forced into using that engine.
right, it's just about the rendering part. so yeah, firefox can do that. I haven't really paid attention to this since I don't personally use apple devices. I'll make sure to pay attention to it though next time I use my girlfiend's ipad. I already installed firefox on there just to have a sane browsing experience, but it wasn't specifically for the autocompletion.
It seems like a useful privacy feature to me. I don't want Apple or Firefox to pass every site I visit every time I visit it to Google, Apple or any other company under US law.
Yes, although in principle this kind of list should only be necessary for fresh browsing profiles (plus people who don't keep a local browsing history). Once you've used the browser for a while, autocomplete should predominantly be powered by your local browsing history anyway.
Firefox on Android e.g. used to (or still has – I'm not sure what the state is nowadays after the rewrite) have a similar (although somewhat smaller) list of popular domains which it would use for providing autocomplete results if it couldn't find anything in your history, but local history always had priority.
I switched to Firefox about a year ago but I'm really missing the autocomplete from Chrome. I guess sharing my entire digital life with Google had some upside after all.
Joking aside - Firefox makes some strange choices and the I wish mobile and desktop could (optionally) share more of my browsing history. Maybe they do but the autocomplete doesn't behave the same across devices.
I was at first guessing this this might have had as a (partial) data source the hsts static preload list[1]. But, a domain I know we never submitted to it was on there while one which was submitted (but has since aged out) is not. Also these lists are so absurdly large
Even with `xz --best`, you can only compress this down to 25% of its original size. I guess a more specialized text compressor that can get this to less than 10% would allow 2-3x more domains to be included.
OP here. I found this URL by intercepting traffic on my older jailbroken iOS device. It's possible that a more recent version of iOS doesn't use it. This url, among others, was returned in some configuration-fetching call to https://api.smoot.apple.com/bag (which requires auth)
I assume it's for pre-populating the address bar suggestions. Even on a fresh browser with no history, users probably want 'nyti' to complete to 'nytimes.com', for example.
It looks like something not on this list gets suggested in mobile Safari, but not autocompleted (so on top of the screen it will show as a clickable link, but you will not see the complete url in the address field and be able to just go to it by hitting "enter").
My employer is in this list, along with their main competitor. However, said competitor's domain is incorrect and links to a much smaller unrelated page.
Various politicians have made their feelings known beyond any doubt. Note the lack of, say, pornhub etc in this list. Someone has sanitized it somehow and didn't pre-emptively censor wikileaks. That's a good thing, I guess?
There are several furry art communities listed, but Fur Affinity is not. Fur Affinity is the largest and most popular of these by a fair margin.
Unrelated to the furry websites the above, there are some nsfw websites listed but other, more popular ones are strangely missing. (The word "porn" does not appear in the list.)
redditenhancementsuite.com is also listed... which I did not expect.
I suspect this list is run through a keyword filter and a blacklist before anything is added.
There is however several sites that redirect to "erotic film" and blatant porn sites (the domain name also references porn, so it's not like someone scooped up another random domain here). So, probably not manually curated...
funny, I checked the different forums I hang around and the only one in the list is the smallest (in terms of user counts). Though maybe it's the most well-known/infamous outside of its direct niche?
1. Killed flash because it's insecure and drains battery.
2. Crippled and never let PWAs fully work because security and battery.
Your hypothesis raises two slightly ugly questions - "Why could't Apple make those things work on their platform in the past?" and "What's changed in the security and battery aspects of PWAs that means Apple are now moving to support them?" If you look at the changes for iOS 15.4 there are a lot of things that improve support for PWAs. If you're right, and the reasons that they didn't support PWAs in the past are security and battery life, then the new changes in iOS presumably mean that Apple is intentionally crippling their platform's security to support PWAs. That's a scary move, and should make every Apple device owner somewhat concerned.
I think it's significantly more likely that the reason why they killed Flash and haven't supported PWAs is because they represented a threat to the App store, but now Apple believe that supporting PWAs doesn't do that (either because they believe PWAs can't compete with apps, or because they believe users will choose a platform that supports PWAs). There was nothing about PWAs that meant they couldn't be secure, or that they had to drain the device battery. Apple just chose not to support the features because they didn't want to, and not because they couldn't.
Steve's sixth and most important reason in that essay is "We know from painful experience that letting a third party layer of software come between the platform and the developer ultimately results in sub-standard apps and hinders the enhancement and progress of the platform."
That's Steve saying he wants to 'protect' the app store, and justify the 30% fee. If people could pump out Flash games it would have put Apple users off buying apps...
I'm sorry if my comment came off like that but my point is we don't need such lists and more walled gardens - where is the selection criteria for this?
Who decides these domains? Why does my domain not equal to a domain of my competitor? Does it put me at a disadvantage as a small player if I'm not on this list?
And this isn't just one instance but it keeps happening over and over.. Apps getting rejected from play store due to arbitrary standards while there is news of big cos like google making secret deals with apple to get exceptions and special advantages which the small guy can't (1)
I think what the OP is getting at is that Apple has a "secret" list of domains that it has approved for suggesting. There's no means by which we can get on to that list. In an open source browser, this list would be public, and assumedly there would be some way of submitting a change request.
All that said, I think this is a little overblown. I doubt many people go to one site over another based on the existence of this list.
Darwin is open source and there’s nothing you can do to get your patch in (unless you’re the upstream of a package they include, that is). Apple Podcasts directory is closed source and you can submit your podcast as you wish. Open source and a submission channel a la hstspreload.org are orthogonal. Anyway, this is a generic tangent.
Nowhere in their comment they mention the topic at hand, it's just a jab at controversial things Apple has done. It doesn't add anything to the post but causes outrage for the sake of it.
It is a mainstream website mirroring the mainstream opinion in Japan so the target of your very valid criticism is the Japanese public, not Apple.
Similar would be criticizing Apple to not include foxnews.com in the list. It's not Apple's fault that American culture has brought us to such a website becoming mainstream.
Yes, that does make a difference and no, it’s not.
An autocomplete list is most likely generated based on statistics and contains sites that are being _used_, and an allow list is moderated and contains sites that are _condoned_.
Assuming this is a list for Apple's autocomplete feature on Safari and other apps, what damage can a hacker do if they were to maliciously update this file?
Not much if you use a password manager that checks the domain before auto filling and/or use U2F tokens as your 2nd factor.
The threat model isn’t too different from other things that can happen if a malicious user is on the same network as you. The scale would be different though.
Homoglyph Attacks come to mind. Replace legitimate domains like nytimes.com with ones that look identical but lead to phishing sites. (The hacker would have to build those convincing phishing sites as well, of course...)
That's a hoax just like the "ok hand gesture" = "white power" debate. It started as a joke on 4ch, now everyone (even big tech) believes in it because of a few thousand (social) media reposts.
The list exists initially to support a feature, a utilitarian reason. Then, being in the list has SEO value or some such. The list becomes a legitimacy or authority test. It affects traffic, sales.
Maybe other stuff piggybacks this list. Anyone who needs a list of important domain names uses it. Ordering search results, caching, testing, crawling... More dynamics unrelated to Apple's original purpose.
Meanwhile, Apple continues to treat the list as a relatively unimportant thing... Just a clunky little hack that enables autocomplete. At some point, willfully ignorant of a whole industry of professional services out there promising to get you on list.