Hacker News new | past | comments | ask | show | jobs | submit login

A lot of this is a repeat of weird issues that we saw 8(!) years ago with OSX Lion. Perhaps I'm off the mark here, but I strongly suspect there's lack of a proper manual QA team at Apple.

QA at most so-called "modern" companies these days has turned into overly fetishising automated testing. Automation is valuable, but crucially also leads to situations where the least-common-denominator situations get tested since it's easier to automate the conditions for testing them. So for example you might write an automated test that verifies that the mail index is correctly upgraded from (say) El Capitan to Mojave to Catalina; but that doesn't show the whole picture. There could be 20 other complex factors that affect it when it's used as part of an OS upgrade, and none of those get tested.

Project managers, line managers, and other suits also seem to love automated testing, because you can show off some kind of real number to someone in an "Oh we wrote 200 tests this cycle, isn't it great that we can run them everyday" way, even if there are 0 issues found. Manual testing in the same case leaves you with no output that can be run in an automated fashion. Also, no one wants to argue against automation (even if it's taking away manual testing resources) because that can make you seem old-school and out of touch. And once this attitude gains steam, no one on the QA team will want to go against the grain and say "You know what, we should spend more time not writing automated tests".

All of this is to say -- sometimes during testing, you simply have to put yourself in a user's place (for as many different kinds of user you can imagine) and actually use the thing you're supposed to be testing. I strongly suspect that Apple is not investing enough, or at all in this kind of testing. Instead, they are pushing it off to those enthusiasts who install and use beta or newly released versions of their software. So far, this strategy seems to have paid off for them, but perhaps people will wise up and stop testing their stuff for free. I am certainly not upgrading my machines to Catalina any time soon.




Also, no one wants to argue against automation (even if it's taking away manual testing resources) because that can make you seem old-school and out of touch.

As someone who is "old-school" and has argued against automation countless times, it's more that your arguments against it will likely be dismissed because the management just wants to see the numbers on their pretty "quality metrics" dashboards going up; regardless of what it means for the actual quality of the product. They'll even ask you to "fix" automated tests by forcing them to pass, if they are failing for some reason you don't know yet. Truly a case of "you get what you measure". It's only when you start showing them the bugs that customers come back with and why they couldn't possibly be discovered in automated testing (at least without an impossibly large and nearly-infinite set of test cases on a similarly large set of hardware and software) that they may start to listen to you. Sometimes.


When they’re ready to listen, what do you propose instead?


> lack of a proper manual QA team at Apple

Yeah, I think so.

Apple's apps have always been terrible, and it seems weird because their Mac OS (though of course buggy in absolute terms) has, for almost 20 years, far outshined any other desktop computing OS of similar complexity (meaning, basically, Windows and Ubuntu and equivalent Linux-plus-all-basic-apps type distros).

I think this is a side effect of their silo culture. App teams are tiny, often one person, and they work in permanent stealth mode. Then they finally release a developer preview, which includes the core OS plus like a hundred app releases.

Suddenly, they have thousands of beta testers: Mac developers paying Apple for the privilege. I've been one for the past 20 years or so. But we don't have time to file bugs (especially through their shitty, rude Radar system, although it's recently finally gotten... um, marginally less shitty). We spend our bug-filing time on the OS deficiencies that are fucking up our own applications.

So the OS itself gets stress tested by thousands of developers, whereas none of us really gives a shit about the app collection that will ship alongside the OS.

E.g., relevant to this issue: thousands and thousands of developers install every OS release and beat on it, but almost none of us would expose our actual email accounts to some developer preview release of Mail.app. We'd just use a different machine, or use webmail for a while while working with the new OS.

So the apps themselves actually get released with pretty minimal real user testing.


The reality is actually the opposite. They rely pretty much solely on manual QA and only in the past few years started writing unit tests, sometimes. Apple is actually quite an old school company some ways.


Not so much automated testing but too much focus on unit and shallow integrations and not enough the big picture testing.

Such as load testing, end-to-end testing, fuzzing. All of which can be automated to a large degree.


My hypothesis is that the difficulty of writing automated end-to-end tests where failure can reliably indicate actual brokenness, combined with some of the other factors I mentioned is the reason for all the focus on unit and shallow integrations that you mention.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: