The silver lining is that the media hype probably got a lot of people interested in physics and materials science. Some of them will undoubtedly go on to make great discoveries and contributions to the field in the future.
I remembered the account of developers committing bugs so that reviewers were forced to find something instead of just giving the all clear.
Maybe there should be a group of researchers that submit papers that seem believable with known issues to avoid scientific journals stagnating. Maybe the names and institutions of the group submitting papers should be hidden so that more reputable/famous people and institutions get the same level of scrutiny as everyone else.
In terms of function though, the "hairy arm" is the complete opposite though: it's not to make sure reviewers are paying attention, it's to distract meddling reviewers from the changes you don't want them to veto.
> A feature added for no other reason than to draw management attention and be removed, thus avoiding unnecessary changes in other aspects of the product.
Seems clear to me: "change the system so that publications in big-name journals are not important for career advancement." There are many fields that don't have this requirement, or anything like it, and the suggestion is that academia join their ranks.
The "alternative solution" to having a thing that wacks people on the head every Tuesday is...not having it.
Businesses have money. Governments have votes. What measurable quantity do you propose for scientific work? Votes of colleagues? It's even more corrupt.
Well, simply, for senior researchers to stop making big-time publications so high-up on the list to hire people. I know quite a few senior researchers, and they do have some say in it...
A more benign example was the college professor who started each lecture saying he was going to say one wrong thing.
(it would keep the students alert)
or maybe Van Halen's "CRC check", where their venue contract would ask for a bunch of things set up in advance, including a bowl of M&Ms with the brown ones removed.
(it was a test of diligence, that they read the contract and knew the requirements)
Years before this current trend of LLM generated papers, there was a tool IIRC by some MIT students which would generate plausible looking papers which when actually read were obviously garbage. The title and abstract would have the sort of words you would expect of a research paper, but the resulting sentences would be nonsense.
The idea being that if the submission got published, the review process was obviously a total joke. I think they even had some examples that had actually been published.
You want to solve reproducibility? Apply forensic scrutiny to every publication?
That's a great idea, but who is going to pay for it?
Every scientist knows that it's more than possible to publish deliberately fraudulent work. Every non-scientist thinks that detecting this fraud is easy.
> Every non-scientist thinks that detecting this fraud is easy.
Eh, this working engineer doesn't think so. Freeform fraud detection is a hard problem in general, and when you're talking about (apparent) cutting-edge research, the pool of people who might spot specific details is tiny.
A low probability of detection does suggest that the penalty should be relatively high to have a deterrent effect.
Agreed. I probably won't catch a bug that the tests wouldn't also catch unless it's obvious. I'm looking for things like:
-Is this duplicating something
else?
-Should this be part of this
object?
-if I come back in a year, will
it take me 10 minutes or an
hour to understand it?
-is this necessary?
-should I approve this but add a
task to the tracking board to
figure out if this should be
done in the service instead of
the client?
-etc
If my understanding of the timeline is right, Dias already had a retraction when this was sent in. I don’t think reputation is to blame here, the reviewers just weren’t up to the task.
It might be worth pointing out that reviewers are only on board to offer their opinion with the material given at face value. After all they are probably not in a position to experimentally verify whether the results are reproducible.
The editor always has the final word on whether a manuscript gets accepted.
Weird, I was peer reviewing a manuscript earlier today and actually thought of something similar. The manuscript needs some work to improve its English and I wondered if some of the broken English passages were a test to see what nonsense a referee like me would put up with. I don't think it is a test but the thought did occur to me.
Careful not to repeat what the University of Minnesota did to the Linux kernel, when willing submitting security bugs to write out a paper about trust.
Because there's only one intended reader who is sufficiently educated and smart. All others of us are impostors who should not have pursued this career at all.
When it's "not even wrong" it's easy to dismiss something. But if it takes some work to disprove, there can be some cool insights and knowledge (and new avenues of exploration).
Nah. The reality is that very very few people will "make great discoveries and contributions" even for graduate students in those fields or after getting a PhD degree, and many people who are good enough to become professors do mostly mediocre research or publish papers that have very little impact. For this LK-99 thing, I doubt more than 5 people of all those that are interested will be making any significant contributions in 10 to 20 years.
As a scientist I see it the other way, where people will just ignore future discoveries and science and this goes on the trash pile with other bad science that makes the public distrust science. We saw the endgame of that with Covid and it was really ugly. The public doesn't understand the minutiae of each study and just gets really tired with "Carbs are bad" and "Now carbs are good again" headlines.
It really isn't. This paper was accepted and published in a peer-reviewed journal. Research could already be underway based on the results of this paper. The paper was retracted because the other co-authors raised concerns, not based on experiment or observation. Science working as intended would mean this paper wasn't published in the first place, nor would a researcher like this have his own lab.
Science historically works on the time scale of decades and centuries, not months and years imo. So I think it's working faster and better than ever before. Journals are not science, just a small tool for slightly better communication between scientists. I think/hope we'll make them obsolete in favor of something that looks more like git repos with lots of automated analysis of logic and tests etc eventually.
Peer review is a modern invention, so benchmarking whether or not science is working based on peer review is a bizarre notion. What we're seeing is indeed old-school science working as it always has.
But nonetheless there should be controls on such nonsense. The ease with which many academics get away with all this will make even a seasoned politician jealous.
Journals aren't intended to be pefect and only filled with papers that are never disputed or exposed as wrong.
There are ways of estimating percentages of bad papers, just as there are ways of estimating subsets of other populations.
This is science working as intended - the good reproducable work advances on the strength of the good papers and the bad papers fall away from lack of follow on.
The real question is why Nature keeps publishing this guy without any scrutiny. They really should have learned their lesson after the fiasco that was their last paper on superconductivity.
The vast majority of academic journal review is single-blind (reviewers are anonymous, but not authors). Double-blind review can be requested by authors submitting to Nature Journals, but it's rare. https://www.science.org/content/article/few-authors-choose-a...
So here I'm learning that Nature published their paper, which claimed an absolutely world-changing breakthrough, after the lead authors had already had two papers retracted?
the Springer Nature Group has a profit margin of 26% on annual revenue of ~€1.7bn [0]. If they were keen to publish this paper, they could have afforded to send an expert to the lab to actually see the process for themselves.
But that's not the kind of thing you do if your core business model is to publish splashy headlines...
that seems entirely too high for merely hosting and publishing papers. They don't do verification themselves, since scientific papers are meant to be peer reviewed (and unpaid peers, for that matter).
> In 2010, Elsevier’s scientific publishing arm reported profits of £724m on just over £2bn in revenue. It was a 36% margin – higher than Apple, Google, or Amazon posted that year.
It's the perfect rent-seeking business: grant bodies and governments judge scientists by approval of journals, scientists need to play the game of getting into 'reputable' journals, journals sit in the middle, outsource most of their work into unpaid work to scientists, and laugh all the way to the bank.
Then governments introduced mandatory open access for scientific research, for a moment it looked like journals' chokehold might fail, but they just adjusted: Nature journals charge authors ~$10,000 per open access paper. That's all taxpayer money going to for-profit organisations with little justification for the price.
> grant bodies and governments judge scientists by approval of journals
so this is the lynchpin - how is it that such judgement is reliant on the journals, when it is peer review that gives papers their credibility?
Therefore, shouldn't the peer review system be the judge of quality? Which means by exposing more peer review and making it way more open and participatory, the gov't will no longer need to rely on reputation in journals to judge grants or progress?
The first step for acceptance is the journal editor - in lower tier journals, that's an unpaid Professor, in higher tiers it's a paid staff member (usually academic background too) - who decides whether the journal thinks this paper is a good fit or not. Most papers die there, too low impact, too niche outcomes, not enough wide readership etc.
If it has passed the editor it goes to peer reviewers, unpaid academics, who make similar decisions; is it technically sound? Is it impactful?
Impact is measured by the number of citations this paper will have, a bit of an educated guess. A paper on an obscure seagrasses will have few citations, a paper on a novel cancer-defeating mechanism will have many.
High impact factor journals are selective in which papers they accept (via editors and journals) to keep up the high impact factor. That double-gated decision is what turned into a stamp of approval; it's the journal's brand which is the lynchpin for the academics' career, not the peer review
Springer is a huge publishing conglomerate that also deals in physical books, besides the payroll for editing and typesetting, the cut from retail channels, etc.
when a group stops large disclosed retractions and starts publishing the retraction somewhere in the Sports Section -- or worse yet : when the retractions stop.
Here's what I got when I asked it to finish the paragraph "In summary Matlab's rich set of features, mathematical capabilities ...":
"In summary, Matlab's versatile capabilities, including robust mathematical functions, extensive libraries, Simulink for dynamic system modeling, support for diverse data types, and its user-friendly programming language, collectively create an environment conducive to efficient numerical computations, effective data visualization, and the seamless handling of complex scientific and engineering tasks. Its popularity in both academic and industrial settings, coupled with an active user community, cross-platform compatibility, and support for parallel computing, positions Matlab as a top choice for professionals and researchers seeking a powerful and adaptable tool for their computational and analytical requirements."
"diverse data types", "user-friendly programming language" are exact matches in the linked snippet, and the rest of it is very similar in tone ("strong community support" vs "active user community").
ChatGPT also likes to end on "In Summary"; when I asked it to list Matlab features it did just that.
At a guess, it is copy/paste from MATLAB marketing materials, intended as a sort of appeal to authority – "we used MATLAB so our results must be good".
The disappointing part of this is the large parts of the population who will look at this and other recent cases (see: Francesca Gino) and think that "academia is broken" or "scientists can't be trusted". That narrative will even be amplified by many YouTubers and periodicals who were quick with those headlines. However, as soon as this work came out, multiple scientists voiced concerns, several even filed complaints with the journal. Those in the field even took steps to reproduce the work found in the paper. The fact that this error was caught and several were skeptical enough to comment is how the system should work. Arguing over results that are too good to be true, taking steps to try to reproduce it independently, and publicly taking it down is why science can be trusted. Research isn't going to be perfect every time but peer review and reproduction should weed out the less than credible.
Academia is broken, and for deeper reasons than the occasional implausible result. Indeed it's the papers that don't make waves that are dangerous. As well as the papers that don't get written.
I am not comfortable with that point of view. Growing up people used to always say "question authority" and that seems like the responsible thing to teach. But on the other hand, does it really make sense for everyone to do their own diligence in every matter?
The thing that has caused me to question this the most is crypto-currency scams. I don't mean generally, just the things we would all agree are clearly scams and kind of obviously so. They so often tell you: "do your own research". I think its a version of the poorly written Nigerian scam letter. The idea is a person with the right skill or knowledge will see its a scam right away and not bother them. People who are careless enough, dumb enough desperate enough will still be suckered and there are plenty of them. So the obvious scammers is completely comfortable saying "do your own research" and know that he filters out everyone but the suckers.
I wonder if sometimes most of us are better off just trusting the people who are experts most of the time? Skepticism is good but you can't really form your own exper opinion on every question you will face.
"But on the other hand, does it really make sense for everyone to do their own diligence in every matter?"
It probably depends on what is at stake. I don't particularly care about superconductors, but I do care about my knees, and when one doctor several years ago wanted to operate me, I sought a second opinion elsewhere. Sure enough, the other clinic told me that this can be treated conservatively.
I canceled the operation and, 6 years later, I have zero problems, zero pain and can walk just fine, even up/downstairs. I am not sure what the operation would do; knees are sensitive joints and don't like to be cut open.
> you can't really form your own exper[t] opinion on every question you will face.
noone can be an expert in all fields, but you can seek multiple opinions from multiple experts, and generally the consensus is likely to be correct. Esp. if those experts are far apart, and unlikely to be colluding or associated.
The trick is how to do it efficiently, and not to fall into confirmation bias (aka, seeking only experts that agree with your preconceived notions).
even just the task of finding multiple experts qualified to hold an opinion on a topic can be overwhelming. for a lot of things (like, for example, superconductivity) i'm not even qualified to determine if somebody should count as an expert or not.
this is the reason why things like scientific journals exist. it's embarrasing that Nature had to retract this paper, but on balance they're still doing a way better job of judging this subject matter than i would.
> The disappointing part of this is the large parts of the population who will look at this and other recent cases (see: Francesca Gino) and think that "academia is broken" or "scientists can't be trusted".
No, sorry. Just no. Stop. "Scientists" were clear, loud, and nearly unanimous in their calls for caution about these results. People on this very site were in these threads saying "we should wait a few weeks, guys" and getting downvoted into gray oblivion for their trouble by a horde of kids hopped up on "proof" they saw via (and I'm not kidding about this) a multimeter screen on TikTok.
You can't fix that with science. Science did everything right here. We had a tantalizing result reported, investigated, and disposed.
Saying "These are huge claims with insufficient evidence, let's wait for clarity" is hardly "being no-fun wet blankets". It's understanding how science works.
"And the only thing you wish went differently is that you squashed more people's enthusiasm??"
It's not about "enthusiasm". It's about scientific illiteracy. You can be excited all you want. "I saw a multimeter on TikTok, therefore it's true" and then shouting down folks with actual understanding for being cautious isn't enthusiasm, though. It's Dunning Kruger in action.
I agree with your assessment, of course, but I don’t think it invalidates the comment you are responding to. Large parts of the population could plausibly think those things, despite how clear and unambiguous “scientists” were about the results.
What's the problem? Lots of newspapers have separate news and editorial teams who don't talk to each other and only learn about what the other team is doing when the paper comes out. Nature may well be doing the same thing. I don't know if there is anything funny here
It is structurally impossible for these teams to actually be independent. All claims to the contrary are risibly and offensively false and should always be met with vocal contempt lest they start to sound credible to too many people.
