I think a lot of modern web design suffers from the same AB testing failures that got Pepsi into trouble back in the coke/pepsi days of the 80's. I may be misremembering the details, but the gist was that Pepsi was a bit sweeter, and so in the tiny amounts that people tasted during taste tests, they usually prefer the sweeter drink. In normal use, however, that flavor of Pepsi was too sweet, and sales tanked.
I see the same problem with user tests. People almost always pick the simpler of two options, because at the time they have no reason to pick anything else, and that looks easier. But then you start trying to get actual work done, and the Fisher Priced interface is too simplified.
And I know it's not just you or me that prefer information dense websites. Look at how well-known McMaster's site is, among people who actually need to use it to get work done.
Even in general public terms, things have gone too far. I set up a lot of iPads for elderly folks for use in assistive communication, and I have to go out of my way to use ones that have buttons. The modern ones without a home button are just too complicated for a whole swath of our population, and I am not exaggerating.
> But then you start trying to get actual work done, and the Fisher Priced interface is too simplified.
That's exactly my thinking as well. To give another example:
This week I tried to use an iPad and an Apple Pencil for day-to-day note taking. If you handwrite 100% plain English prose like, The quick brown fox jumps over the lazy dog, it instantly converts it to text and corrects any typos. For simple stuff, it seems perfect.
But it's insufferable once you start on real work. It doesn't understand a lot of punctuation, tries to correct foreign words (likes names and locations) I mix in with English, tries to spell correct serial numbers (N07 becomes NOT). Having to go back and correct each conversion error turned out to be much slower -- and frustrating. I went back to a keyboard.
The most notorious example I remember was an update to data entry software in the medical field. The old one ran on some Unix iirc, had a text mode interface, and was entirely controlled by keyboard. Staff could use it in their sleep, enter text, tab, enter more text, tab tab, and so on.
The new version ran on Windows, had a lot more features for sure, but suddenly staff was entering data into a field, grabbed the mouse to move to the next field, entered data, and so on. The process became way slower. So I asked why they don't just ignore the mouse and keep using tab. They were surprised this even works, because it was never mentioned by the folks who trained them on the new software. It was all about how modern and slick it looks and how it has so many more features.
So then staff tried to use the tab key again, but got frustrated very quickly since the software was smart enough so enable and disable controls in the entry form depending on selections you made in earlier fields. So tabbing through the form would require different number of key presses all the time, making it hard to build muscle memory. I think they raised their complaints to management but whether that feedback ever reached the software vendor I have no idea. I guess they did get faster again over time because any new system takes time to get used to, but I'd be very surprised if it ever matched the ancient, simple system again.
I would add to that what I call "fractioning" of data (on multiple screens).
We used for years (decades) a DOS based tool, where in a "conventional" 80x25 mask there was enough space for all the data we had to input (navigating with TAB/Enter, etc.) before passing to the next (with F8).
On the new software the same data (in windows, on a modern screen) goes inside a teeny-tiny window that has not enough fields, so it is spread on three tabs and this breaks the flow.
The same happens a lot on forms on the web, fill one or two fields, click next, repeat.
This explains a tendency I've long had which I always assumed was just a look and feel thing. So back in 2021 I got myself a (relatively new) Blackberry. I loved it. Hardware buttons. Lots of them. I've had to upgrade to a newer Android for work reasons, and it has so many default settings I've had to turn off because they are infuriating. Pixel 7. Gestures. No standard bottom bar with the three buttons. Power button does something else. Why???? Even with all these turned off, the three button bar disappears if an app is full screen, and you have to do a gesture to get it back. Seems mad to me.
This was very jarring at first, but over time I've actually gotten used to the gestures and they feel more consistent than various makers of phones having the software buttons in different positions. It's almost like muscle memory at this point.
If you don't like gestures, however, you should be able to go to the system settings and enable the old style navigation. If they had removed it entirely, then I'd probably be more critical (a la Windows 11 vertical taskbar).
That's my experience with doing basically anything that's vaguely complex on mobile devices.
The iPhone camera is usually excellent, but there are some situations (for me it's autumn forests and funnily enough Disneyland at night) in which the auto white balance fails miserably. On my DSLM it's a simple question of just using manual white balance, but on the iPhone the only way to do it with the stock camera app is to do in post.
On that note, the Photos Mac app has a neat feature where you can fix white balance by calibrating on a neutral grey area, but that feature is missing on the mobile apps which just have a color temperature slider which does not give as good results.
That isn’t how AB tests work. In a (properly-run) AB test, a user will only see one variant. AB tests have shortcomings, but the Pepsi critique isn’t one of them.
Combine A/B testing with exponential growth of the user-base and you will only measure newcomers. Medium, advanced, and expert user simply won't matter in your test results. Whether that is what you need depends on the situation.
The Pepsi critique can also happen with AB testing depending on what you're measuring. The problem that Pepsi had is that they measured user satisfaction after a small sample not a full can. This would show up even if you only present drinkers with one sample.
By ”the Pepsi critique” I specifically meant the exposure of both variants to a single user.
I agree with you that having users take a single sip is also a problem for construct validity. But it’s not clear from your post how this would this would happen in a properly run AB test.
Are you thinking of the case in which we ship a change that confers short term wins in user spend or churn, but comes at a long term cost that isn’t visible within the window of the test? This is a common problem, but it’s not clear to me that it’s the same problem Pepsi had. Here the problem is with choice of response variable, whereas Pepsi’s problem is choice of treatment.
My point about the Pepsi challenge is that it would have produced the same erroneous results if they ran it in a way that only exposed consumers to one or the other. This is because the flaw was that they were measuring satisfaction after a sip or whatever opposed to a full can.
It's about what you actually measure versus what you think you are measuring. In this case, the can satisfaction vs versus the sip.
Short term spend or churn could be examples. For an information serving website, examples might be mistaking time on the site or pages viewed for user satisfaction and finding the information they want.
That was a thought that I had when writing my post too. Test AB testing a focus group based on a 5 Second page view Bank of radically different answer than if you sat people down and asked them to perform a typical or moderately difficult task.
If this is indeed the case, I'm just shocked that developers and companies haven't figured it out. I suppose a lot of the time the performance impact simply isn't measured in their kpis
My other hypothesis is that once you move away from an orderly website architecture, organizations and devs are just as lost as the users and no one has an idea where information should reside. Without holistic architecture or vision, the best that people can do is throw a third or fourth hamburger on the home page, add a new page 10 links deep, or not linked at all and pray that a search engine will solve the problem.
I think at the end of the day, it's because the modern way of designing things is just easier for developers.
I remember when that started taking off; it was honestly kind of a godsend because one of the laziest way of doing things suddenly became associated with professionalism. On the other hand, it's getting pretty bland as a consumer now that every site is doing it.
Compared to images as buttons of the 2000s, something adjacent to material design seriously cuts down on the number of graphical assets necessary for develop anything. Even games seem to follow that trend: compare Crusader Kings 2 vs Crusader Kings 3, with image buttons vs a general UI toolkit in the two games.
The fact that the back button is completely useless now (at least on mobile) because every website wants to refresh every time you go back is probably one of the worst things to happen to web navigation in a long time.
Horrendous experience and it leaves me fumbling with opening everything in new tabs which can get super messy/annoying on a phone.
As far as easy access to tabs goes, Sleipnir browse (by Fenrir, a Japanese company) works. I do change some of the other default settings. Gestures, user-agent string, button layout, long-tap behaviour… It's fairly configurable. Uses its own extensions system, though.
Also, the desktop version of Sleipnir is…not recommended: quirky and buggy.
The modern web has been taken over by what you may call simulacrum driven design. It’s design by mimicry instead of design by fundamentals. I mentored a few design groups in SF before leaving the industry. The decision making I saw and were ridiculous, and were even being argued against usability. Oh well!
"Modern" websites aren't designed for the masses. It's not a matter of preference, the masses have little input into how the websites they visit are designed... These sites are designed to track the masses to serve ads or to trick the masses to sign up for some service, buy something, etc.
Try Unix, bro. Ubuntu is arguably a decent start coming from Windows. It gives you easiness of use, intuitive visual interface, simple software installation process, decent compatibility with hardware...
Then you can gradually get familiar with Linux and doing things on the command line. Once mature enough, you can explore pretty much any other distro.
Modern design is fueled by ego. The developer and all peripheral participants want to feel smart, it is no longer about the user and it hasn't been for a long time.
Ofen times it isn't even a balance between aesthetics an clarity, because the modern pages are broken or otherwise unusable.
It has gotten to the point where I wonder if maybe I'm not neurotypical. I'll take sites formatted like Craigslist 100% of the time.
Now I'm just waiting for windows OS to fully replace file navigation text with iconography and I won't be able to use a computer