Some time ago - though not as far back as this article was published - we did an experiment at a conference that we held in the demo facilities of a Very Well Known Audio Company.
We played a range of snippets of music - rock, classical, electronic, pop - at various qualities over what was quite possibly the best sound system in the world.
The audience was a significant number of record label executives, distribution execs and general audio/music industry experts.
We played pairs of the same snippet and asked people to tell us which was higher or lower quality.
One person got them all correct. Turned out he’d mastered one of the early tracks we played so had a good reference and then used that as a baseline for the others.
Everyone else it was completely scattershot.
It wasn’t a controlled experiment but it was definitely interesting.
This is why an ABX test is the way to go. If you don't know which version of a snippet is "right" there's no way to objectively say which one is "better" - maybe you like the distortions (cf the famous vinyl "warm sound").
wait, but if he could distinguish them from having mastered one of the tracks, doesn't that kind cast doubt on the "no human could possibly ever distinguish these things" mantra? as an aside, am i alone in feeling like the absolute certainty and hyperbole with which this argument tends to be stated potentially has a bit of a "doth protest too much" vibe to it? i definitely know it is a minority view, and that is ok-- i'm happy to be wrong and skeptical.
skeptical though i may be, i'm definitely not here to say that "audiophiles" aren't charlatans or anything like that, for the record. and while i don't totally understand the setup you describe in the sense that i don't get why insider knowledge on one track would tip all the rest of them (were the HQ tracks played either all first or all second, or something?), wouldn't one person's ability to completely discriminate between the two encodings seem to be very strong evidence that it is possible to tell the encodings apart? the kinds of differences between master recording and 16bit 44.1 kHz are exactly the kinds of things that would give away which encoding is higher quality, no?
i feel like there is this moving target thing that goes on sometimes, where the strong argument made loudly is "no human can tell the difference", and then the tests are more like "most people can't tell the difference between things that they have no reason to be attuned to well enough to have any chance of picking up these subtle differences.
forgive me if i misrepresent, or come across like some sort of audio quality chauvinist-- i ask all of this in earnest, and without having a strong opinion one way or the other.
Genuinely interested to know why - without being present - you think this would not have provided some sort of measurement of this. While it wasn’t scientifically devised it also wasn’t just “play a couple of things”.
The problem is that "what sounds better" isn't the same question as "which one is uncompressed".
The original CD version might have some high frequency stuff that's just on the edge of perception that you don't really know is there, but you can just sense a bit of discomfort when listening to it. After going through the MP3 process and that high frequency is removed because it contributes the least in reconstructing that signal, the resultant decompressed signal might sound "better" even though it's not the original, because you get a high quality reproduction but the thing that led to a slight discomfort when listening has now gone.
In this case, the sound engineer got them all right because he could tell the difference and knew what it was supposed to sound like. The rest of the people maybe could tell the difference or maybe couldn't (which was the claimed result of the test), but in fact, even if they could tell the difference, they had no idea which one was the uncompressed one and voted on which they thought sounded best.
As another comment has noted, it'd be a much better test if there was a "they sound the same" option as well as asking which one sounds best.
Without being present there, I can't analyze actual execution of the test, but test methodology can still be evaluated.
Playing A & B samples and asking which one is better/original requires much more from the listener that just hearing a difference between the two. It is possible to hear the difference, but not know which is which as that requires additional knowledge.
To avoid this issue you could:
Play (in random order) original twice and processed once and asking which one was different/processed.
Or play two sequences (in random order), [original, original] and [original, processed] and ask, if processed was in the first or second sequence.
Second option might focus better on short-term memory, because it has shorter sequences (2 samples vs 3 samples per sequence).
This would produce a better measurement of whether the difference is audible or not.
We actually did something more along the lines of your first suggestion - we were asking for difference not specifically “which is better” because actually “better” is entirely contextual (and actually this was part of the point - that some people actually prefer lower quality audio because it’s what they are used to). I think (it’s a long time ago now) that we played each set which was two or three samples twice and rather than asking which was “original” I think we asked them to score for clarity, richness and which they preferred - and then to make a guess about which was a higher/lower quality source file.
Like I say, it wasn’t a rigorously scientific experiment but it was in the context of a conference about evolution of audio standards and what that meant for audio delivery from labels/distributors to DSPs.
Some time ago - just a couple of years ago, I decided to get back to the roots and to listen some Iron Maiden. After a visit to the Brown Sector I made out with the full discography in FLAC but also I got a copy of Powerslave in AAC ripped from iTunes.
And all I can say what I really, really hear the difference.
Because iTunes version is mastered to sound good in iPods so it has a quite noticeable bass boost all over the album, which is extremely noticeable on my 2.0 acoustic which itself has a good bass boost, so this version sounds quite muffled compared to the FLAC version from the CD (CP32-5043).
But on the go, with my CX300-II / CX3.00 there is no noticeable difference.
This is exactly why vinyl actually does sound better sometimes. Because of the physical limitations of viny, vinyl masters were less affected by the "loudness wars" and they have greater dynamic range.
We played a range of snippets of music - rock, classical, electronic, pop - at various qualities over what was quite possibly the best sound system in the world.
The audience was a significant number of record label executives, distribution execs and general audio/music industry experts.
We played pairs of the same snippet and asked people to tell us which was higher or lower quality.
One person got them all correct. Turned out he’d mastered one of the early tracks we played so had a good reference and then used that as a baseline for the others.
Everyone else it was completely scattershot.
It wasn’t a controlled experiment but it was definitely interesting.