FWIW local demographics have a non-negligible effect on a hospital's specialty. The nearest hospital to a retirement community will get more practice at heart attacks and strokes and the one in the college town will be good at dealing with alcohol poisoning.
Risk-adjusted outcomes were calculated after adjusting for population factors, co-morbidities, and health system factors. Even after risk-adjustment, there exists large geographical variation in outcomes
That means they adjust for things like that heart attacks in 80-year-olds are more likely fatal than in 40-year-olds, so a hospital serving mostly the former is expected to have a lower survival rate for heart attack patients.
They are trying to measure differences in effectiveness for similar patients. GP gives a possible explanation for such a difference : the hospital probably gets good at the kinds of things it does a lot.
This is so frustrating for me personally - I recently had to choose between several hospitals without knowing anything about the historical patient outcomes for each hospital.
A hundred years ago a doctor named Ernest Codman suggested that hospitals should be rated by an "End Result System" [1] where paient outcomes were measured and compared, yet only now do we have the "first comprehensive study comparing how well individual hospitals treated a variety of medical conditions". And we still do not know exactly which hospital is best and which is worst.
To me the best way to judge a hospital is blindingly obvious: For each patient at admission, estimate the chance he/she will be alive after 3 years. For example, a 65-year-old woman who is a smoker, is overweight and has stage 2 lung cancer might have a 25% chance of being alive in 3 years. Then compare the predicted outcome with the actual outcome. For example, say both Hospital A and Hospital B both admit 100 patients with 20% predicted chance of surviving 3 years. If after 3 years there are 30 such patients from Hospital A still alive but only 10 such patients from Hospital B, it would strongly indicate that Hospital A is better. We could look then at Hospital A for ways to improve Hospital B.
What's so disheartening is that this process doesn't involve any special technology. It could have been implemented 100 years ago, albeit with less rigorous statistical methods. We spend a billion dollars on evaluating a drug which has a marginal, one-off benfit for a few hundred thousand people. But we ignore a process which is cheap and offers long-lasting benefits to millions of people. It seems that the medical establishment is too powerful in this case.
It's not that simple, and it only seems blindingly obvious to you because you haven't actually worked in healthcare. The trouble with using survival or outcome ratings is that they effectively punish providers who take on the hardest, riskiest cases. We simply don't have enough accurate data. And many of the input variables aren't independent. The reality is that's it's impossible to accurately quantify all the different factors which potentially impact survival. So some providers find ways to game the system through metrics arbitrage.
That said, it's still worthwhile to gather and use it for quality improvement. We just have to be careful to treat it with appropriate caveats and look at qualitative factors as well as quantified metrics.
Spot on analysis. The complexity of the human factor, for want of a better term, in healthcare is so hard to measure. I can only relate it to what I know. That is, I work with several paramedics that are just "good" at differential diagnosis. They would probably be excellent doctors, but life didn't deal those cards in. I also work with "meh" paramedics that, if given the choice, I wouldn't want taking care of my family. They do an adequate job to the bare minimum standard set, but they are not great by any stretch of the imagination. Both paramedics charge the same and carry the same title. Both could miss something critical, or not. The fact is they are both humans practicing paramedicine.
Until we automate the variability out of healthcare, we are stuck with inconsistent results. Consider that roughly 70 years ago, lobotomies were considered a valid medical procedure for psychiatric problems including anxiety and depression.
Sorry yes, I accept it is more complex than I described. There is the possibility that (a) the hospital admitting a patient can assess patient risk more accurately than a university professor who specialises in risk factors; (b) the size of the hospital advantage is significant when compared to the real differences in risk between hospitals; and (c) the hospital takes advantage of its better analysis by turning away patients where the professor risk rating of the patient is higher than the hospital risk rating.
This is subtly different to your analysis in two ways:
1. The trouble with using survival or outcome ratings is that they effectively punish providers who take on the hardest, riskiest cases. Not necessarily - if the professor underestimates the risk in easy cases and overestimates the risk in hard cases it would give the hospital an incentive to take on the hard cases and turn away the easy cases.
2. The reality is that's it's impossible to accurately quantify all the different factors which potentially impact survival. In some cases we have remarkably detailed understanding of a broad range of risk factors - take a look at table 3 from page 23 in this [1] report on stem cell transplants. There are detailed odds ratios for patient age, ethnicity, karnofsky score, CMV status, disease type and stage, interval between diagnosis and transplant, year of transplant, donor match type, donor/recipient sex parity and donor age. More importantly, the professor doesn't need to be particularly accurate in assessing patient risk - she just needs to be as accurate as the hospital. (Or close enough that the difference would not give the hospital a significant advantage)
Can the professor get close enough to hospital accuracy when estimating patient risk? Two relevant points spring to mind: (a) In the book Thinking Fast and Slow there is a description of how expert analysis can often be matched by simple rules of thumb. (b) The book Superforcasting describes how some lay people are able to make better predictions than the CIA about future geopolitical events, despite having no access to classified information.
My gut feel is that the hospital would not have a significant edge over the professor. But I'm willing to be proved wrong - let's conduct an experiment where a doctor estimates the 3-year chance of survival for each patient at the time the patient is admitted to hospital, and a professor also makes the same estimate based only on the patient's database record. The winner gets half a million dollars. Wouldn't this experiment give us valuable insight in determining whether hospitals can game an End Results system?
No rational hospital administrator would ever agree to be judged on 3-year survival rates. The sample sizes are too small and there are too many uncontrollable confounding factors impacting survival for this to ever give statistically significant results. Did the patient die because his heart bypass surgeon was incompetent, or because he went right back to eating Big Macs as soon as he was discharged from the hospital?
Well there are about 370 hospitals who agree to have patient outcomes on cardiac surgery published [1]. And there are 173 hospitals who agree to have patient outcomes on stem cell transplants published [2].
And if we keep the identity of the hospital confidential, the number of hospitals willing to provide data on patient outcome grows substantially. The study referenced by the New York Times article [3] had 22 million hospital admissions, and the Cystic Fibrosis Foundation has gathered detailed data from all US Cystic Fibrosis treatment centers for 50 years [4]. Although it would be better for all hospitals to be completely transparent, there are still significant benefits to judging hospitals in anonymous fashion, as described in [4] (which is a fascinating article in its own right).
As for statistically significant results, it's true that we will be unable to judge the effectiveness of smaller hospitals (unless they are extremely good or extremely bad). But for medium-sized and large hospitals this is not a problem - in the stem cell study [3] there were 43 out of 173 hospitals whose 2015 results were judged to be outside the 95% confidence interval (26 were better than predicted to a statistically significant degree, while 17 were worse). Of those 43 outliers, 26 were also outliers in 2014 which suggests this is not just a fluke.
Regarding your question: "Did the patient die because his heart bypass surgeon was incompetent, or because he went right back to eating Big Macs as soon as he was discharged from the hospital?": Keep in mind that we can control for body-mass index, so the question is whether people with similar BMI will be more likely to eat Big Macs after treatment at Hospital A, rather than Hospital B? Yes it's possible, but it seems to me more likely to be an effect of the hospital. Also note that it may not be down to the quality of the heart bypass surgeon - it may be simply that Hospital A is more effective at encouraging healthier eating after patients are discharged.
any time a problem of this magnitude hasn't been solved AND you think the answer is 'blindingly obvious', consider that you don't fully grok the problem. I've been in healthcare analytics for 10+ years and it's incredibly difficult to measure outcomes. Take your example of 3 year outcomes, are you saying that the entire reason why someone is alive is because of the hospital they went to 3 years ago? There are so many confounding variables that even trying to measure outcomes 90 days out is difficult.
On top of that, the other commenter mentioned the unintended consequences of simplistic measurements like the one you describe. Look at measuring patient satisfaction - 'what could possibly be wrong with measuring patient satisfaction', people say! hold docs/hospitals/etc. accountable. but when you look at the data, patient satisfaction scores are weakly correlated with evidence based care and healthy outcomes but it is strongly correlated with things like how much time the patient sat in the waiting room or if the doctor prescribed them a medication.
> There are so many confounding variables that even trying to measure outcomes 90 days out is difficult.
Very much this. Two patients, same doctor, same proceedure, with somewhat similar medical backgrounds. Nearly the same progression in the hospital, and released to go home, and possibly to doctors that don't interact with that hospital.
Perhaps one is poor and sometimes skips medication. Only one starts eating healthily and starts exercise. One has a stressful home life. Maybe one, when the recovery isn't going as quickly as possible, starts mixing in alternative medicines but doesn't tell his GP. Maybe they were both women, in the hospital to give birth - and one got pregnant again in the first year while the other didn't. These sorts of things greatly affect the outcomes, but none of them are under the hospital's controls.
> any time a problem of this magnitude hasn't been solved AND you think the answer is 'blindingly obvious', consider that you don't fully grok the problem.
That is a good point - this thread has turned out to be more complex than I expected, and had I been a bit more open-minded I would have avoided the phrase 'blindingly obvious'.
Still, I'm not yet convinced this is an intractable problem. The New York Times articles says that patients are 3 times more likely to die at one hospital than another, even after controlling for patient sickness, income and age. This may be due to (a) a confounding variable; (b) a real difference between the quality of care provided by the hospitals. How much weight should we place on each possibility? Is it 50/50, or 60/40, or 10/90?
If you have to have a medical procedure and were told the survival rate of Hospital A is 98% and at Hospital B is 94% would you say, "there are so many possible confounding variables, this statistic will have no bearing on my choice of hospital"?
Outcomes vary widely by doctor/surgeon as well, but the medical profession has successfully resisted meaningful measurement of results for many years now.
Most surgeons do not do enough cases over their entire careers to be meaningfully compared to their peers. There was a big paper about this recently. The ratings are a sham meant to instill false confidence.
Agreed! There are shit doctors and great doctors, but they all have one thing in common... the Dr. preceding their name! They have resisted for good reason. I mean, when you have a bad day at the office, some code or deal or possibly something else breaks. When a surgeon has a bad day, it is not so simple and it is easy to forget they are human.
I have worked countless full arrest codes in the field... countless. Some go a lot better than others, and some are even successful. Most codes are do not end well for the patient whose heart has stopped beating or the drive to breathe has stopped. I can't imagine having stats of the codes we work pushed in our face everyday. There are sooooo many variables that we don't control which would make the whole "measuring" endeavor meaningless to anyone that really cared. From the weight of the patient to the cleanliness of the house are all variables that may or may not matter. Good luck controlling for all that!
"There are shit doctors and great doctors, but they all have one thing in common... the Dr. preceding their name!"
I usually look at a doctor's reputation, experience level, education, publication, etc. as signals for "quality." It CAN vary drastically, as in, one doctor might overlook a diagnosis or even mis-diagnose something that another, more experienced doctor would catch correctly.
Every one of these proposed quality signals is prone to error. Publication history I would see as a negative quality signal. Less experience means more likely to be up-to-date on latest treatment techniques and algorithms. Reputation in the "Best Doctor" survey sense usually correlates with years in practice more than actual quality. The best indicator, in my experience, is to ask a nurse at that hospital who they would send a family member to for that specific problem. Even board certification does not hold great discriminatory power, in my experience. Analysis of these indicators would make for a worthwhile research project, but the more beneficial project would be to try to identify and disseminate those behaviors that form mechanisms of quality.
I have used a similar service, and found the results to be helpful in finding a general practicioner.
Tracking revision and mortality rates for surgeons would be much more helpful. I have been told (by engineers who have worked on joint replacements) that some surgeons have revision rates well over 50% (and frequently leave patients with one leg longer than the other), while others have revision rates well under 10%.
> The study did not disclose which hospitals had which results. Under the terms of the agreement to receive the data, the researchers agreed to keep the identities of the hospitals confidential.
Well that's not too useful for the patient oh I mean "consumer"
In metro LA, the top-tier hospitals (Cedars-Sinai, UCLA) have noticeably better outcomes than regular hospitals. The difference is like 98.7% of patients not dying versus 97.9% in a valve replacement ("AVR"). Flip it around and it's 1.3% versus 2.1% -- definitely noticeable.
OTOH, UCSF (presumably top-tier, SF residents correct me if I'm mistaken) has 2.3% chance of dying. Maybe it's not a top-tier facility for AVR?
And further, a small midwestern hospital in Hays, KS (pop. 21000) has a 3.6% chance of death. That's huge next to Cedars-Sinai at 1.3%!
These ratings are not great because they incentivize doctors not to take more complex cases. They claim to be risk-adjusted, but in reality you cannot adjust for all possible risk factors. Lots of things in medicine are very unique. There's very little that's more complex than human disease.
As a referral center for other referral centers, additionally UCSF will be getting some of the most complicated cases in the world.
In a previous life I worked for cardiac surgeons - one of them was on a board for the State of Florida to research outcomes and how to publish them without affecting which cases are taken.
In the case of one example where NY started publishing outcomes the difficult cases switched to the Cleveland Clinic. The outcomes appearred to increase in NY and decrease at the Cleveland Clinic, but risk adjusted rates didn’t actually change for NY hospitals that transferred cases or for the Cleveland Clinic.
This is absolutely something that happens and Cardiac Surgeons are all scientists to some degree. The results and risk adjustments are very well documented and they can determine them with a fair degree of accuracy. I’m not aware of the info for this particular case, but if they did risk adjust (the patient) it right - there are still a lot of other factors that determine quality of care (speed of delivery of service) and quantity of surgerys performed (if you don’t perform enough surgeries per year your quality suffers). Incidently, this is true for surgical staff (nurses) as well. If the surgeon travels to numerous to hospitals… the hospitals with the most surgeries performed will have better outcomes (after risk adjustment for them doing higher risk surgeries).
It's not that some surgeries are being shifted from one center to another, its that no one at all will do them, generally to the detriment of the patient.
Someone will do them. There is a business model for surgeons who only take the sickest patients. Sure, in that case your numbers are worse, but that's the way you market yourself.
I agree that numbers can be gamed, and it's possible that private hospitals might be particularly motivated to do so. Indeed, probably these numbers don't make much sense for anyone outside a small medical community of insiders to assess.
> These ratings are not great because they incentivize doctors not to take more complex cases.
This reminds me of a scene in the recent Dr. Strange movie where the main character -- a neurosurgeon -- turns down a case because it had a high risk of failure and he didn't want to tarnish his perfect record.
> They claim to be risk-adjusted, but in reality you cannot adjust for all possible risk factors. Lots of things in medicine are very unique. There's very little that's more complex than human disease.
Isn't it better to approach the problem by improving how we do these risk assessments so that the risk-adjusted ratings are fair, as opposed to leaving patients in the dark about where to get the best care?
It reminds me of a relative who couldn't find a surgeon for his problem, because the first 2 thought he was a risky patient. Then the 3rd charged more.
And thinking about it, I realized how many projects I jumped in while others jumped out because, you know, they were risky in not succeeding...
You seem to be saying that a hospital that doesn't want to take complex cases can more accurately adjust for risk factors than the people doing the study mentioned in the article. Is that right?
Surgeons talk about a "look test" that refers to that certain something not captured by objective risk measures. A patient who doesn't look too bad on paper may look in person like they might not survive a haircut, let alone a valve replacement.
The database can store both objective and subjective risk measures. If there is good evidence that a 'look test' can improve risk assessment, lets get the referring doctor to perform a look test and store the result in the database. In that way the hospital will not have an edge over the risk-adjustment model used in the study.
Edit: Many patients are already assessed by performance status [1]
Not the GP but I generally assume that a patient's actual doctors know more about that patient's physical body and how it will impact a procedure than some academics who are classifying/"coding" thousands of case files.
You also have to consider that the Hays, KS hospital, being a much smaller hospital, has a much smaller number of cases. The confidence interval around that is definitely larger as a result. Not sure how much larger, but likely enough to make the difference statistically insignificant (town pop is 21k, heart surgery # is likely very low).
Put another way, a single death out of a small population of patients will have a larger impact on the statistics.
Another data point: 30-day mortality rates for high-risk acute heart patients are lower during the dates of national cardiology meetings, i.e. when the top doctors are away.
Yes, this is what I had in mind. In addition to all that, treatment by medical "professionals" who may be quacks and/or have qualifications granted by diploma mills.
I got 6 stitches in Cambodia for a major cut on my arm. No wait, seen by 3 medical professionals (including a doctor), in and out in 15 minutes. Total cost: $15.
For me, that was a far better outcome than I would have received in the US for a similar condition. Of course, stitches are pretty well understood; I can't advocate for any other medical procedures in Cambodia.
Do you think the Secret Service knows, for example, which hospital is best for gunshot victims? For stroke? For heart attacks? Of course they do. And yet, when Hillary collapsed at the 9/11 Memorial, they took her to no hospital, but to her daughter's apartment instead. Seems odd. Very odd.
