The proposed "solutions" of pre-registration and increase sample size are not going to work because they are not fixing the underlying problem of NHST use (where a researcher tests a nil null hypothesis of zero effect and then concludes their favorite explanation is correct if they reject this null hypothesis).
The solution is to go back to the old way of science (pre-1940):
1) Collect data and figure out what is repeatable and consistent (eg come up with "natural laws").
2) Come up with theories that explain the observations from #1 that also make precise predictions about some different type of data.
Reward researchers who figure out methods that lead to reliable, repeatable observations and those that make precise predictions (if your theory predicts nothing more than a positive/negative correlation between two variables it is too vague).
Your solution is historically inaccurate. Statistical tests of significance were created as a tool to improve the objectivity of evaluating results. Garbage in, garbage out still applies. Nor is it relevant. Statistical techniques are a symptom, not an underlying cause.
Indeed, everything in the article or in discussions of this that I see is always treated symptoms far from the causes. Let's do a quick five whys:
Why are publications not reproducible? Because positive results are being published as fast as possible with little checking on them.
Why are they publishing so? Because the people involved need publications to get out of their current transient position and secure another transient position, with hopes of someday being able to settle down somewhere.
Why do they need to secure their next transient position? Because they have been inculcated that this is the career to be in by their milieu and that is the only way to stay in it.
Why is that the only way to stay in it? Because there are no permanent positions that aren't at the very end of this gauntlet.
Why are there no permanent positions except at the end of the gauntlet? Because this is a pyramid scheme built on an apprenticeship system.
Don't get distracted by all the other stuff. It's the fundamental structure of academic science that leads to the rest of this.
This is exactly why I am aiming to bootstrap my own lab outside of academia and traditional research positions available private industry.
There's a lot of "BS" in these power structures that keep people from being able to dedicate resources and a proper amount of time to generate quality research.
I'm fairly "green" in my research career, but I've been in the game for a few years now in the Biology/Bioinformatics realm. I've seen and heard stories from plenty of colleagues about how when they first got started and well into their first P.I. positions (and even until this day when they're 10-20 years into their work) where they get abused by their institutions. Aside from "publish or perish" there's also your institution skimming 50-60% of your hard won grant funding to let you "rent" their facilities where you're fighting over shelf space with other P.I.s Absolutely insane.
Seeing this I decided I can bootstrap my own lab for much less and have true autonomy. For example a rough breakdown of my costs for setting up a bio/bioinformatics lab for myself and a few friends:
Land Zoned for Light Industrial and Scientific Research?
about $1000/acre or less in my area. (hint: buy tax liened property)
CO2 incubators? Don't pay $4k-8k new. I can buy refurbed or liquidated lab equipment from the unsustainable institution labs for literally pennies on the dollar. Try $100-200 for this equipment.
High RPM centrifuges? Once again instead of $2k-5k I can spend $300 for a 15k RPM centrifuge. and something like $20 for desktop centrifuges.
PCR machines? Try < $100 each. And I've even gotten "extras" with them in that price range.
Compute power/I.T.? I can get 256 core, 1tb ram blade centers for about $1k. Modern Core i5/Core i7 desktops for $80 each.
By my estimates I can buy the land, build a building(admittedly a very utilitarian one), and equip a new lab for sub $30k. Now my only costs are re-agents(if I'm doing research that requires it), power, and cheap land taxes.
You'll find that any large state university has dozens if not hundreds of little businesses that sprout up in exactly this way. You see them in the little business parks -- most of them lease rather than buy. You see them at the trade shows.
It goes without saying that the idea is just like any other business venture, with ways to succeed and to fail.
But a useful thing is to try and get a survey of what kinds of businesses are out there, what they are doing, and how well they are doing. Next time you're at a conference that has a trade show, go and look at the booths in the "low rent district," because it will give you an idea of what people are doing.
Oddly enough I work for a F500, and we buy second hand gear. One reason is that if somebody is advertising a particular instrument, it means that they actually have it, and you can get it right away. With new gear, sometimes it can take weeks or even months. Also, if it's under a couple grand, I can buy it on my purchasing card without management approval.
I have a business park a literal stones throw from my home. I thought about going that route, unfortunately for me I have some higher power requirements that I can't get in those units. I thought about leasing correctly zoned warehouse space with the right power options but the cost is actually more expensive in year 1 than buying and building in my area.
I'll definitely take a look at what some other small businesses are doing. I've seen some up close, but the trade show run through is a great idea.
And yea. I buy stuff at auction within 300 miles of my area and drive the same day to pick up any gear I buy. What kind of department do you work within inside your F500?
I'm in "advanced" R&D, playing with technologies that will be in future products. For my group, we often need something on a temporary basis. So it doesn't make sense to invest in a brand spanking new machine. Plus, we might be intending to modify it.
Could you link/explain how you get those hardware prices, specifically the servers and desktops? Are the available for the average consumer (ie not making a bulk purchase)? I have very little experience buying hardware, but those prices sound 4-8x lower than what I'd expect (and I'd be super happy to find that I'm wrong).
They are definitely available outside of a bulk purchase. I see about an 80/20 split between having to buy in small quantities versus having to buy in a bulk purchase.
I'd prefer not to share the info on a public forum to keep mass buyers out of the market(more than there already are)
Shoot me a message at ethan at dancecardrx dot-com and let me know what you're looking for and I can point you in the right direction.
There's a lot of consulting work available in traditional labs because of bureaucracy around hiring new employees full-time. So I can apply my skillsets in those labs as an outside consultant.
Ideally though this would be a pure-research endeavor. Which requires time and building up credentials and a funding base. So basically eventually income for the entire operation would be a mixture public/private grants and leasing out technology developed within the organization to others.
Are you worried that the necessity of getting those grants written/approved will push you into the same kind of incentive structure that you're trying to escape? Or is your theory that, yes, you'll have to deal with some of the same B.S. and incentive-skewing, but as long as you keep your burn low, it will be of a lower intensity?
He should be worried that the "skimming" he's talking about is, in many cases, legally mandated. Direct and indirect costs not being mixed isn't something that happened because universities wanted it that way.
I think that you will always have to deal with some amount of B.S. as long as you're spending other people's money. That's unavoidable.
The B.S. is amplified however when you're at a big institution. There's a lot more red-tape even if you have private financiers.
One way would be to yes, keep burn low. The idea there right would be that you're asking for less money, you're lowering expectations of those giving you money. If someone gives you $1000 they're going to micromanage you a lot less on how that money is used versus if they give you $10,000, $100,000, etc. The other idea is that if you have less stakeholders there's also less back and forth that needs to be maintained with them.
My idea would be that you'd avoid the incentive-skewing over the long term by doing some research that can be converted into revenue generators to cover the base costs of running a lab. Reinvesting profits into traditional finance products to get a yield on the lab's money. Even running a small scale cryptocurrency mining operation as a single part of revenue generation. If you combine this with low-overhead by being smart about what would typically be big ticket purchases that only the most well funded of labs can afford and finding people with a like mind who are willing to take a liveable(but smaller wage) to work in a more open environment then it should be sustainable. For me in particular I can balance research time with programming consulting time and make 80% of what I would make in Biology/BINF or at an academic institution without breaking a sweat.
Like I mentioned before about grant skimming by institutions.. If you're a PI at a state lab or university they will take a "generous" portion of your grant as soon as it arrives. I'm specifically thinking of one guy I know who recently had HALF of his grant taken for facility rental and upkeep. It was a large grant mind you. In another case a friend who had finished his postdoc and was a first time PI landed his first grant. As soon as it was in the lab managers tried to say "awesome, now we can pay your salary out of the grant" He had to play hardball with HR to state that his salary was guaranteed and not to be paid out of grant money intended for his research. Now I can understand why this happens. It's not free to run a research enterprise, and it's not particularly cheap when you're buying equipment at MSRP. Research equipment in the second hand market before it goes to professional refurbishers is literally cents on the dollar. It's so specialized, there's a small market for it, and big players are not buying this stuff because of the inherent risk! In my experience though the defect rate is so absurdly low that it's not an actual risk buying this equipment. But it's not something an institution with a multi-million dollar grant to setup a research center/core is going to waste their time on.
As for specifics of what I'm aiming to do. I intentionally picked a research area I'm going to delve into that has low costs. An operation like I'm talking about in my OP would not be feasible for high energy physics or high cost sequencing projects. I'm focusing on photobioreactor design and algae cultivation yields for animal feed, biodiesel refining, etc.
For that I need CO2 incubators, centrifuges, a 3d printer, algae samples, a sample freezer, turbidimeters, photospectrometers, pH meters, and then a revolving supply of plant fertilizers, CO2, and Nitrogen. But before I get deep into actual PBR design and cultivation techniques I'm going to work on computational models for PBR design, algae growth rates, and PBR monitoring software for data collection.
> One way would be to yes, keep burn low. The idea there right would be that you're asking for less money, you're lowering expectations of those giving you money. If someone gives you $1000 they're going to micromanage you a lot less on how that money is used versus if they give you $10,000, $100,000, etc. The other idea is that if you have less stakeholders there's also less back and forth that needs to be maintained with them.
You do have to be careful about the fixed costs for them administering the grant however. The time and effort of reviewing and approving / rejecting a $1000 request is not going to be a fraction of a percent of a much larger request. Lower perhaps but proportionally higher. And by promising much less, you might not really make it more appealing.
Again, indirect costs do not come out of the direct costs of running the project.
If you get a $250,000 a year NIH R01 (the largest you can get without PITA budget details) and your institution has negotiated a 50% rate, the NIH pays your institution $375,000 a year. Your $250K doesn't get reduced.
The only time I've ever had something like what you described happen was when a private funder had X total amount they wanted to spend, but hadn't bothered pre-specifying that they'd only pay a particular overhead rate. Which is mainly just a company wanting to get something for nothing, and doesn't leave me particularly sympathetic.
It's kind of a mix. Many agencies and calls say that the maximum amount you can budget is say, $500K over 3 years including indirect costs. So you'll say the project costs $350K and allocate $150K to indirects. If indirects were lower, you could up your direct costs.
It's not only private funders who do this. It's the biggest funding agencies in the United States. NSF, DARPA, Army, etc. In fact, I feel most grants operate this way. I'm not sure about NIH since I've only been Co-I on those.
The NIH is by far the largest funder of research in the U.S., so "most grants" has to include them.
You are correct that there are grants from agencies with a cap including indirect costs - I confess I have the opposite experience of you, I'm usually the Co-I on NSF-style grants, so I don't look at them much.
In that case, yes, if indirects were lower you could up your direct costs. I still contest the idea that the indirect cost system is somehow stealing "your" grant money though (where you = lab).
I didn't use the word "stealing", but they take a cut from the grant money allocated to the PI.
Maybe here is another perspective. NSF "Smalls" in my program are 500K, and Army "YIPs" are 360K. If I get a Small or a YIP, and my friend at another university with half the overhead gets the same Small or YIP, my friend will get more spending money than me.
I get 300K on the Small, and my friend gets 380K. To do the essentially same unit quantity of research. A university with a higher overhead takes 80K more from the grant that would have gone to the lab.
"there's also your institution skimming 50-60% of your hard won grant funding to let you "rent" their facilities"..
Is certainly a creative way to describe indirect costs.
For standard NIH/NSF/most Federal grants, they don't come out of "your" grant funding - they're a pre-negotiated rate, and the award amounts are based on indirect costs. If an institution cut their indirect rate to 25%, you wouldn't see that money back, it would just never be there.
I've also never been at an institution that's not willing to at least work with a grant giving agency/firm/etc. on lowering the overhead if it's a sticking point.
But I will say in my experience that I get a ton of value for my overhead rate - and that it's known in the institution to actually be slightly lower in most cases than the actual cost of doing research.
I'll admit it's definitely a creative way of describing indirect costs. I think we may have some different viewpoints on whether or not those costs are inflated justly/unjustly.
I don't think I can argue that the overhead at traditional research institutions is super inflated. It makes sense when you take into account the level of bureaucracy and growing amount of "extra" administration :). But I think it's disingenuous to say that if the costs were cut back to 25% that you'd NEVER see that money. If indirect costs were able to be lowered at these institutions there would be more money in the funding pipeline for direct costs to begin with right?
It's fair that you think you get a ton of value for your overhead rate. Different setups are better for different people. My ideal situation is definitely not ideal for most people. I just don't think I can stomach supporting what I see as administrative waste and needless redtape(not that I believe there should be NO redtape, no one sane would believe that?).
Also in your other comment about private donors not wanting to cover any indirect costs.. completely agree.
I have a friend who works in a flow cytometry company. Flow cytometers are one of those things that really need to be run day in and day out by experts to be reliable instruments, so they do assays for hire, contract research as part of grants and studies, and use that to fund their own research. It works quite well.
One thing to check on, though, is the requirements for waste disposal from your lab in your area. You may find yourself having to pay for special disposal.
Good point. I'll check that out. There are quite a few hazardous waste disposal companies in my city.. and we have a pretty large research footprint, hopefully the rates aren't awful!
Very astute. More permanent positions are needed, and grant agencies need to limit number of grants single professor can head. In fact NIH just did so to 3, I think. Good move.
>"Because positive results are being published as fast as possible with little checking on them."
Why do you think this is so? It is because they think the stats are telling them the probability their theory is true, etc. A significant p-value means the result is "real", etc.
Replicating a study is not something that is likely to get you published, and putting in twice the work to run an experiment twice is unlikely to give you twice the 'return' on your time investment (assuming the funder even supports spending twice as much).
Why, though? Assumably the replication would be the interesting part, and presumably both teams work would be equally valuable showing the same result.
Innovation is overrated; maybe we should just pay scientists more to do inglorious but immensely valuable work. I was certainly repulsed from lab sciences because I had no desire to do the work that's in demand—pulling results out of your ass to get more funding.
They are likely to not get cited as much, which would lower a journals impact factor, reducing the incentive to the journal. This makes it worse for the scientist as there's work without a high "impact" paper, and since they get largely to choose the work they do if it's less "fun" it might be done less. The funder also has to choose between funding new work or replicating previous work.
Something I'd be very interested in is paying independent labs to replicate studies. This removes the incentive issues surrounding the journals and researchers, leaving the funding issue.
Why would you need to cite a paper that confirms a result? If several people have confirmed it, do you need to cite them all every time?
If these were the case you'd have the opposite problem. People would be hugely incentivised to reproduce highly cited papers again, when what you want is to reproduce papers with low levels of confirmation.
Yes, we want to know what observations we can "hang our hats on", so that we can come up with actual theories (not vague crap like "this drug makes that disease happen less"). The more replications published the better until it is being done in undergrad/high school classrooms.
You can't rely on anything in the current half-assed, no replication environment. In many areas it is literally not worth coming up with an mathematical model to explain the data because it is all so questionable.
I'm not really sure how that addresses the issues raised.
You're saying it's better overall if more replications are published. I'm talking about the problems that are stopping us from getting to that point. The current incentives do not align well with the desire to have more replications done, and simple changes could easily backfire.
> (not vague crap like "this drug makes that disease happen less")
I simply do not agree that if you think this and find some results that point in that direction you should not publish. I see absolutely no reason to save electrons to improve some average of papers, I'd much rather work isn't re-done repeatedly. Perhaps publishing something vague with some backing (e.g. we think X does Y and the data is at least consistent with this, and we can't think of what else makes sense) gives enough for someone else to build on and do a more rigorous investigation.
In my initial post I distinguish between two steps. The first is data. If you collected some data and can describe the methods well enough for others to replicate it then go ahead and publish it. There is no need to start theorizing about things like "the drug caused the difference" vs "there was a confounder that caused the difference".
Explaining the observations is a separate step from coming up with a reliable way to generate a pattern in the data.
No one gets particularly large amounts of glory for confirmatory research because if it's successful than the first paper is still "The first paper to describe X".
I don't understand what studies you are referring to. Each additional paper should be improving whatever parameter estimates are being made, it makes no sense to only look at one result and ignore all the others.
While they'll all get swept up in meta-analyses, systematic reviews, etc. a huge portion of a paper's citation count can come from narrative sections, where that first paper will reap a lot of citations in "First describe by...", "As established in..." etc.
The Nobel Prize goes to the people who discovered something. Not the people who confirmed that they were right.
Not to mention that if your study is irreproducible, because you made a mistake in experiment design, or failed to control for some variable, running your experiments twice may not necessarily catch this issue.
Having someone else run the experiments is also required to get past something we have to deal with a lot in the tech world. The "it works on my machine" result.
You may have run the experiment twice, but can anyone else? Does your setup depend on something not specified in your paper? In the tech world, how many times has the setup documentation you wrote turned out to be missing some detail when a new starter actually tries it?
No setup will ever be perfect, but we can look at reducing common points of error.
context matters...in both the research and way it is reported.
correlation is not too vague, but implying causality from correlation without further tests is
conversely...
assuming that a paper that only reports correlation or reports correlation as well as further evidence of causality (i.e., prior work on causal mechanisms, qualitative evidence, etc.) is faulty simply because it has correlation is equally problematic.
The major problem of the article is that it displays the same flaws as many of the underlying problems in science itself: It makes massive generalizations without nuance or appreciation of context. The problems are paradigmatic (i.e., Kuhnian) and the fact that they appear in the research AND in the critiques of the research are far more problematic than a p of .05.
Fair point of clarification...I was specifically referring to the ontological and epistemological components of the argument. The perception of many scientists (especially young ones) that their research is inherently independent of them as a researcher is a big problem. These are issues that do not appear at surface or even values levels...they are base level assumptions about knowledge that manifest very subtly.
I'm not sure what you mean by results. If you mean the results of a thought process that does not go beyond predicting positive/negative correlations, then I see no point to publishing it.
If you have reliable measurements then that falls under my number 1. There is no need for any vague theorizing, but go ahead and include it in the speculation sections (Intro/Discussion) if you like.
Edit:
If all that gets published is a correlation coefficient and not the distribution, scatterplots, etc. Then no, that paper is worthless.
The solution is to go back to the old way of science (pre-1940):
1) Collect data and figure out what is repeatable and consistent (eg come up with "natural laws").
2) Come up with theories that explain the observations from #1 that also make precise predictions about some different type of data.
Reward researchers who figure out methods that lead to reliable, repeatable observations and those that make precise predictions (if your theory predicts nothing more than a positive/negative correlation between two variables it is too vague).