They do justify their methodology, I didn't think it sounded that unreasonable.
They flat out say that out of the two easily-measured factors, distortion and linear response, linear response correlates the most with subjective measures of audio reproduction according to prior research. I don't think they said that they had developed a foolproof methodology for absolutely determining the subjective reproductive quality of a headphone.
I think your description of the performance of headphones is based on soft, unscientific nonsense. Sure, there's more to headphones than single frequency response curves, but frequency response between the ear canal and the headphone is the only differing factor in audio reproduction quality.
If anything is flawed with the methodology, it would surely be with the lack of broad spectral testing or something equivalent. The fundamental characteristics of the driver are the diaphragm geometry, the mass of the driver, and the resistance of the suspension. The suspension changes, probably not linearly, with temperature. Frequency sweeps completely miss the point that the movement of a headphone driver is linear actuation, not some mystical frequency-domain process.
Oh, it's soft, unscientific nonsense. It is also direct experience as a semi-pro in the field.
Subjective experience is not science, but neither is it irrelevant. If someone's "science" does not explain observed subjective experience very well, then it shouldn't pretend that it does - and it really is not grounds to dismiss the subjective experience of experts by saying "BUT THOSE DUDES HAVE NUMBERS AND STUFF!!!!"
"Subjective experience is not science, but neither is it irrelevant."
Ah, anecdata.
Look, if you want to point to well controlled studies, etc, that say "these factors do not correlate well with subjective experience", that's awesome.
I'd then agree 100% "whatever we are measuring doesn't matter, we should measure something else".
But in your rage, you are conflating two issues here, and they shouldn't be conflated at all:
1. Was this study science, and properly performed science?
All available info seems to point to "yes, it was"
There is no reason for you to put air quotes around science, etc.
They set out what they are trying to measure and why: "This study quantifies variability of measured headphone response patterns and aims to uncover any correlations between headphone type, retail price, and frequency response."
They did not set out to say whether that has any bearing on subject experience.
In fact, they point out "The preferred response however seems to be listener, content, and headphone dependent <cite omitted>"
2. Does the thing they measured matter in any way to the subjective experience in the world?
You vehemently suggest "no".
I'm going to suggest if you want to convince people the answer is no, you should point them to data that says "the thing they measured, properly, doesn't matter", and not appeal to anecdote and authority.
They cite at least three studies thinking it matters:
"In particular, research suggests that the frequency (magnitude) response is a major factor in listener preference scores (Olive and Welti, 2012; Fleischmann et al., 2012; Olive et al., 2013)"
(and they are super careful not to suggest that listener preference scores completely correlate with subject experience)
but at the same time, admit
"Research suggests that factors influencing consumers' choice as to which model to purchase are mostly based on wireless functionality (Iyer and Jelisejeva, 2016) and attributes such as shape, design, and comfort (Jensen et al., 2016)."
They also admit the studies usually are small and that the body of work is not huge.
So, from my perspective, i feel like they are doing a fairly reasonable job of trying to present a relatively objective perspective on whether this matters or not.
I dunno. I think headphones track price-to-quality pretty well in the $20-250 range. Above that, it starts turning into luxury/status symbol stuff. This is my completely subjective opinion. What isn't opinion is that increases in quality are usually a matter of diminishing returns. It becomes increasingly expensive to get increasingly small incremental improvements.
Additionally, I think "sound quality" in headphones is very subjective. There are fine quality headphones that I really, really dislike (Grados, for example). I find a lot of expensive hi-fi headphones overly bright, too.
A good price/performance example is two headphones I keep around... my Beyerdynamic 880s, which I love, and the Sennheiser HD280. The 880s cost about twice as much, and fill the same role for me - closed-ear phones with very strong isolation, for performers tracking vocals or instruments. They can put a loud backing signal into the ear with only minimal leakage into the microphones, and easily block out other loud instruments in the same room. But the 880 sounds far better. It's flatter and more detailed. It's also much more comfortable for extended wear, better built, and more repairable. That's the $100 to $200 difference. But $200 to $400? Smaller change.
Fundamentally, there is nothing in the sound of a headphone other than specific spectral response (for a given head fit and ear). If you have an "ideal" headphone, you can emulate any headphone which is less than ideal.
If well-recorded music sounds bad on an ideal headphone, the recording is set up for a non-ideal headphone.
You'll notice in those response curves, there are common characteristics in the response of almost all the headphones they tested, within a fairly small margin (considering). Maybe for a mastering technician, the right thing to do is master for the average suboptimal headphone, and not for a linear response listening instrument; but that doesn't mean a headphone closer to the ideal is wrong, it just means that you need to filter audio that wasn't intended for its response profile.
There may not be a measurable criteria that drives price in all cases. Headphones are manufactured and priced for wildly different reasons. For instance, Beats are expensive because they're a status symbol. It has little to do with sound quality.
They flat out say that out of the two easily-measured factors, distortion and linear response, linear response correlates the most with subjective measures of audio reproduction according to prior research. I don't think they said that they had developed a foolproof methodology for absolutely determining the subjective reproductive quality of a headphone.
I think your description of the performance of headphones is based on soft, unscientific nonsense. Sure, there's more to headphones than single frequency response curves, but frequency response between the ear canal and the headphone is the only differing factor in audio reproduction quality.
If anything is flawed with the methodology, it would surely be with the lack of broad spectral testing or something equivalent. The fundamental characteristics of the driver are the diaphragm geometry, the mass of the driver, and the resistance of the suspension. The suspension changes, probably not linearly, with temperature. Frequency sweeps completely miss the point that the movement of a headphone driver is linear actuation, not some mystical frequency-domain process.