This is amazing -- but every. single. update. disables it again
Why does the sudo file not persist between updates? This case is quite minor for personal computers, but what about companies that log in with yubikeys or smart cards? Do they have to reconfigure after every update too?
Thought on the second thought, I’ll continue to use the more “manual” method for now. As it gives me more control and it would be easier to switch off when touch ID sudo will be supported more officially.
This is a very macOS problem. Why does it reset every preference on updates? The one that kills me is wake-on-lan, that gets turned back on regardless. Hate!
Because, like ChromeOS/CoreOS, modern macOS now does image-based updates.
A macOS update isn’t “whatever was previously on your OS volume, plus arbitrary patch X”; rather it’s “a new, fresh OS disk image, written to a separate APFS volume, with a fixed SHA, with update transfer-size optimized by composing said image partially from files in your current OS, but only in such a way that the volume will still hash the same in the end.”
(In many ways, macOS’s update system now reuses the logic from iOS’s IPSW firmware-update system. You can even now download macOS updates from Apple as IPSW files, and then use Apple Configurator to push them to macOS devices.)
Unlike ChromeOS/CoreOS, after the first-round SHA verification of the volume, macOS will then patch the new OS-base-image volume with certain files from your current OS-base-image volume, if 1. they’re distinct from the ones it expected to be there, and 2. they appear on a whitelist of known-safe files.
If any of this patching happens, the OS volume’s metadata tree will then be re-hashed, and the new hash will be blessed by the volume-signing utility.
(If you’ve seen a “Recovered Files” directory on your desktop after an update, and it’s contained a “foo.system_default” file, that’s a copy of the file that macOS would have composed into the OS volume if it hadn’t found your known-safe customized file to use instead. /etc/shells is a usual trigger.)
Some preference files in /System/Library are known-safe; but others aren’t. It’s a whitelist, and it’s a conservative one.
Preference files in /Library live in the user volume, not the OS volume, and so will never be wiped. This is why most of the “trivial” preferences stick around.
Note that by “known-safe” here, I don’t mean “can’t be customized to malicious ends.” This isn’t a security thing. It’s an ABI stability thing. These files are safe in that they’re known to have the same ABI format between OS versions, and so keeping a customized version of them around won’t confuse newer versions of system daemons.
It’s about the same as if Linux had a set of known-safe DKMS modules that use interfaces that the kernel intentionally doesn’t change between releases, such that the kernel module files composed from those DKMS modules could be reused by the new kernel without requiring decompilation, rather than being tossed out at every single kernel update.
The thing that doesn't make sense here is that macOS discards randomly your settings, but it is also the only OS I use that regularly gets in a fucked up state where settings have to be nuked. I mean, here we are in year 37 of the mac and people are still zapping their PRAMs. The only improvement is you don't have to physically pull out the battery.
- If macOS is the only OS that you use regularly that gets in a fucked up state, then either you're not using Windows or it's gotten a lot better in the last few years. :) (I mean, it undeniably has gotten better, but I have Windows-using acquaintances who still kvetch about this issue pretty regularly.)
- I've been using Macs since 1999 and I don't think I've had to reset the PRAM to fix a problem since my Titanium MacBook Pro circa 2007. I've had to nuke other settings on occasion, but still generally have markedly fewer problems than I did in my Windows-using days. (Which were more recent than 1999, but still not that recent, so back to the "I'm sure it's better now" disclaimer.)
But, Apple definitely needs to have a better mechanism for managing config file updates than "yeah, we've put a few old config files in subfolders of this 'RecoveredFiles' folder, maybe they're useful, maybe they're not, good luck."
Windows definitely allows you to shoot yourself in the foot, but nowadays those settings are hidden away by non-flashy applications, which has now become synonymous with "advanced user shit I probably shouldn't be touching".
Though the difference is, if something gets royally screwed up on MacOS, at least there's a sane enough way to reset it. On Windows, it often entails a reformat.
And let's not get started on the Windows Updates. They too have the whole "reset settings" problem (e.g. "don't f&!king restart my computer randomly", which gets reset every somewhat major update).
Computing has changed since 2007, so you have to include Android, iOS, iPadOS, ChromeOS, and various flavors of Linux in that lineup to be complete, not just Windows and OS X.
Well, in my admittedly anecdotal experience, Linux has less of a chance of getting FUBARed up by mysterious settings glitches. (I have had a Linux install get spectacularly FUBARed at least once from a broken update process, but usually it's happened through operator error, albeit sometimes aided and abetted by some program's spectacularly bad UX.)
As for the rest, my suspicion is that the newer "mobile generation" OSes both tend to get messed up much less often this way and tend to have much less recourse to the user when they do. That's certainly true for iOS/iPadOS and I suspect it's true for ChromeOS; I suspect Android is probably somewhere between iOS and Linux on that scale.
> but it is also the only OS I use that regularly gets in a fucked up state where settings have to be nuked.
Windows gets into this state fairly quickly, too, but the failures are less dramatic so we just live with it.
At least with Debian, it only happens when I deliberately change stuff! Though the `xorg-server-video-intel` driver breaks features of post-2007 Intel cards, and it's installed by default, so… perhaps it isn't the perfect OS either.
True, but I guess 95% of mac users are still on Intel macs, are still being told by "genius" bar frontline support to reset the SMC / zap PRAM for any kind of issue, and this usually works, too. Jury is still out on whether the M1 auto-zapping architecture is better, or will just be more prone to being permanently messed up.
Well, the fact that nobody ever has to zap PRAM on an iOS device means that they’ve probably fixed that (or, since you can’t zap PRAM anymore, people just restore the device instead...)
>Unlike ChromeOS/CoreOS, after the first-round SHA verification of the volume, macOS will then patch the new OS-base-image volume with certain files from your current OS-base-image volume, if 1. they’re distinct from the ones it expected to be there, and 2. they appear on a whitelist of known-safe files.
Red Hat CoreOS (and other Linux systems managed with OSTree, such as Fedora Silverblue) now do something similar by merging the current /etc directory with the "upstream" version.
FreeBSD has an analogous mechanism called etcupdate[1]. I think Boot Environments[2] are pretty similar to SilerBlue but I don't know how BEs manage /etc.
Debian based systems use dpkg which has the option of confold and confnew. Difference is that the default is to ask the user what to do and unattended upgrades default to confold.
Why cant Apple use their TimeMachine tech to take a snapshot of the prefs on the existing, then do their image based update, then have it automatically apply the prefs from the timemachine mod-integration?
The richest company in the world can't think different?
They know exactly what files were customized in the previous OS-base-image. There’s a reason the whitelist is a whitelist, rather than a blacklist. Just guessing here, but it’s probably something to do with “iteration speed” vs. QA SLAs made to large enterprise customers like IBM.
In my hypothetical, there’s a set of preferences overrides Apple have been forced by contract to guarantee they’ll validate during their QA process, and then allow the OS to retain between updates. For any other file not forced by contract, they don’t bother with that QA process (“takes too long!”), and so default-distrust the ABI-stability of the file, and so don’t let the OS retain it between updates.
Just my opinion here, but I feel like Apple again doesn't really care about power users. 98% of their users won't be making any customizations like this, so they don't care to spend money working on those "edge cases." Their primary mission is to sell devices and apps to average people who aren't tech geeks. And I don't think Apple cares about general purpose computing. I see MacOS getting closer to iOS with every update, and I'm betting that at some point we'll lose filesystem access and sudo access.
Have you ever been FORCED to upgrade/update against your will for a new release whereby you say NO NO NO - then you make the mistake of falling asleep with your phone plugged into the charger to wake to a forced update to your device?
Yeah - fuck apple - and fuck Jonny Ive - they claim to be the masers of all aspects of designs, but really they're they masters of /r/assholedesign
Their hardware is in the top 70th % - but their UX is IMO <25%
(just look at how many clicks you need to do certain tasks, such as bluetooth. They have a slide-up menu to toggle BT and WIFI - but you have to go to desktop->settings->bluetooth->(toggle it on off/refresh)->find the device-> "cant connect" - Toggle BT on both phone and device -> attempt to connect...
But you cant up-swipe hit BT and have it show you the FN menu on screen.
You cant backup and manage all your prefs via icloud - such that you can apply profiles, save profiles, etc from your device to your linked cloud account and say "I always want my privacy to be thus, these are the networks I trust.
Their photos library mgmt is absolute garbage. You have no photo details available to you.
Their albums are garbage.
There are so many interactions that require like 5x more clicks than they should.
Their screens suck.
Their device accessory ecosystem sucks and punishes you for profit with impunity.
They exploit chinese slave labor.
They try to charge you for flaws in the HW provided (recall the balloon battery problem?
I had a macbook pro CATCH FIRE while I was asleep in bed. They said they "recognize that was a safety problem, but because our engineers (after two months of having the machine) determined that at one time the liquid sensor was set off, we cannot replace your machine EVEN THOUGH IT WAS UNDER RECALL FOR THE CLASS OF PROS THAT WERE RECALLED FOR CATCHING FIRE.
That’s a lot of anger and words over something that, in the grand scheme of things, is so utterly unimportant and almost contemptuously irrelevant.
Things are always a trade-off. There are big positives to keeping the ecosystem on a consistent, new and secure version of iOS. Similar positives apply to MacOS.
The problem I have with your argument, is you assume that my frustration is just over a single device from Apple.
I have literally had every single apple product since the Apple ][e -- aside from their server systems.
I have had every iPhone (even had pre-launch phones from them) since launch up until I stopped at the 6s+ which is the only phone I will ever own from them.
I have a history with apple that is not shallow and I have had ~35 years of judgement against them.
I recall getting into a talk with a fellow engineer at Lockheed where I said "Trust me, Apple will switch to intel procs - and this apple boi almost had a heart attack saying that would NEVER happen...
You already have to reboot into the Recovery environment and disable both System Integrity Protection (with csrutil) and OS root-volume signing (with bputil) in order to even (persistently) modify any of the files in /System now.
Malware can't do that, because there's no way for any executable that runs in the regular OS—and isn't signed by Apple—to get anything to automatically happen over in the Recovery OS.
(That's not to say you can't have persistent malware in macOS; just that it can't persist itself into the OS boot volume. It has to persist into the user volume—which is great, because that means the OS can, on boot, mount the user volume noexec and scan it for malware while running in a known-good base state. That's even without needing to boot into the recovery OS.)
So, other than OS updates, basically the only reason these files get modified is when people modify them manually. And the only people doing that are 1. people developing kernel extensions (or their friends, the Hackintosh community); and 2. enterprises burning low-level configuration changes into OS images for image-based deployment.
And both of those cases involve modifying things that are effectively "underneath the OS API abstraction", and therefore don't have the same ABI guarantees that the OS APIs themselves do. Thus the whitelist.
You know =what would be cool: If all apps had to register their pref config files into a single dir and that would be tracked and snapshotted - and then you could just have ALL apps look to the same dir for where their configs come from and you could have a single repo for ALL apps on your or ANY system - and then you could walk up to a terminal and plug in your "license key" which said what apps you had access to and what ones you had configs for and you could just run that app with all your input and prefs and mappings etc...
I actually wrote a white paper on just this in ~2003 or so - and met with several engineers from google and they said it was impossible.
The idea being that you only carried around with you your profile, and you could just come to a dumb terminal, plug in your key, three factor auth - and the terminal would give you access to the apps and resources they had...
This was basically the idea behind the Windows registry - a single configuration store. With mostly the same tree structure on Machine and User level, so your local prefs could override machine-level prefs. The 'user' part was portable between machines on a domain. And you have a single API to access or change settings
Opinions may differ on how well it was executed in practice. I'm not sure /etc/ with its hundreds of different file formats and Ansible or Chefs as an 'API' is that much better
> This was basically the idea behind the Windows registry.
Was it? I got the impression that the original (Windows 3.1) registry was a Windows-internal thing—a store of Windows settings, and a set of APIs to read and modify those Windows settings, e.g. COM/OLE class registrations. (See https://devblogs.microsoft.com/oldnewthing/20080117-00/?p=23...)
But then, third-party ISVs found the registry, and exploited it to store their own settings. And Microsoft being Microsoft, they accepted that unilateral design change and continued on with it.
Prior to the registry most settings were in files like WIN.INI, and ISVs put settings there. The registry took this idea, added more structure and an API so simultaneous writes don’t screw up.
With the registry, each vendor puts its stuff under a path like HKEY_LOCAL_MACHINE\Software\VendorName, even Microsoft.
Yep. The INI files were the original ways to store config info...
The registry was the next step. What would be great is an online serve for such - a machine gets booted and it asks for your config unam UUID etc - and then just slurps your snapshot down.
OK, disclaimer - I am going on over a decade since this happened:
There was a phone that came out (nokia??) that had a docking station and you could have a screen on it and a keyboard attachement etc...
I presented this in ~2004-ish as a white paper... (lost to hundreds of machines since, and poor data mgmt over time)
I met with a few engineers friends from google over this idea and they said they didnt think it was possible (which at the time it was not - even though I wrote about it in 2001) - and then there was that phone that came out which allowed you to have a phone docking station and use it as your primary computer.
This was at the same time whilst I was talking to my buddies at Intel about stacked procs - and they were doing 64 cores in ~2004 in test dies... and they worked.
It was in 1998 when I was at Intel that I asked "Why cant we just stack multiple CPUs on top of eachother?" and was laughed at...
I sat right next to Andy Grove, but I only spent all my time in the DRG Game lab testing games on AMD and Celeron procs to get subjective results on game perf.
Anyway... I posed a lot of things that were laughed at, which then became reality later.
I worked with a MIPS proc eng about Slot-Rack-Servers, in 1995 - this was deemed impossible... later we have HPE based systems... His name was Kent... he was one of the chief MIPS designers...
I was saying "lets make 'slots' that we can install a switch, a server, storage or whatever on the backplane..."
Yes - I am not lying - these were just things I imagined in the early-mid-late 90s....
I had a good career - but I cannot take any credit for these taking production due to I didnt ever implement any of these ideas... aside from expressing them earlier than others who were far more capable of executing.
Its just like IFTT - I literally whiteboarded IFTT for a bunch of engineers from Lockheed a few years before that existed... They have built a lot of what you interact daily with (netflix) among others...
My soul flaw - is that I think of something early, I have no ability to bring it to fruition and even though I have exceptional famous engineers as friends, I cant bring my ideas to market...
and then a few years later - things I designed hit the zeitgeist and hit market...
I have, as a consultant, made people MANY billions of dollars - and havent received anything in return.
The file in question is configuration and should not be part of the update image. (Better yet, Apple should surface this functionality somewhere reasonable, like the System Preferences.)
This is an excellent explanation. One last piece to clarify: there is no way to edit the known-safe list, is there? And/or there is no way to add a script after the update process that copies files like this, correct?
The list itself definitely lives in the OS base-image, so there’s a bit of a catch-22 there. (Imagine if it didn’t! That’d be a bit of an easy loophole for malware authors to exploit!)
I haven’t looked into it specifically, but I’m guessing that like many other basically-static system policy property-lists in macOS, the whitelist lives embedded in a Kernel Extension (kext)—probably the kext responsible for update validation. So you’d have to write your own kext to either replace that extension wholesale, or—hopefully—“inject” over just that element of it.
Of course, to run your own Kernel Extensions these days, you have to run your Mac in bputil --enable-kexts mode, which is “reduced security” from Apple’s perspective (reduced compared to the security model of iOS, they mean.)
On the other hand, the Hackintosh crowd have long agreed that it’s better to just “inject” data into the kernel/OS by either memory-patching the kernel during early boot (Clover), or patching the kernel as it’s being composed into a loadable image (OpenCore), rather than actually modifying anything in /System. (This was before modifying /System was even hard; they just didn’t like making a mess!)
I have a strong feeling that that same community that loves their little quality-of-life kext tweaks so very much, is pretty likely to come out with a way to use OpenCore as the boot-loader on real Apple-Silicon Macs, and thus to overlay arbitrary kexts into macOS without actually reducing macOS’s (OS-stage) security policy at all. (Provided you’re okay with the reduction in security of using a third-party bootloader, that is!)
It may be an unintentional side effect, but doing this may prevent some degree of "Win rot", which Macos (and iOS) has also suffered from to a degree in the past. A huge number of mysterious iOS and Macos problems are caused by an accumulation of issues when config files of various types are "upgraded" along with the OS from one version to the next. If you look at all of the minor and major versions that come out in a year, small issues become large issues after being compounded by a couple of dozen updates.
There was an issue with iOS a few years ago that caused higher than normal battery drain when I was managing a department that supported a lot of iOS devices. A lot of users thought they needed a new battery. The battery drain issue was resolved for 100% of our users by doing an "Erase all data" and NOT restoring the backup. This process will still fix an amazing number of weird iOS issues in 2021, especially issues with the device slowing down, getting hot, or draining battery. It works because you end up with default config files that were designed for that version of the OS. If you restore the backup, often it will "restore" the problem you were trying to fix because the iOS restore process puts some OS config files back.
It will be interesting to see if Macos becomes less susceptible to win-rot type issues because of this config file reset behavior, as annoying as it may seem.
Back in the day I used to tweak my Linux box, win3.1/95, vim, android phone, etc. endlessly.
After awhile they would all become unusable, and I would reinstall everything back to default and try again.
Great for leaning, lousy for day to day work.
Resetting things back to a known state can make life lot easier.
In 10 years I can’t recall needing to reset a Mac.
Why does the sudo file not persist between updates? This case is quite minor for personal computers, but what about companies that log in with yubikeys or smart cards? Do they have to reconfigure after every update too?