I was "ringleader" (many diverse groups have been involved over the years) of the OpenEMR project in 2003-2005 and went on to create the open source ClearHealth and HealthCloud EMR (electronic medical record) systems. OpenEMR has a lot of dedicated folks in it and has been a project of some sort of another for ~25 years at this point.
You can read quite a bit about open source in healthcare in my book, Hacking Healthcare. A bit dated but still in print.
Unfortunately there are massive headwinds against open source in US healthcare settings. Regulation requires certifications that cost upwards of $100K first time, $10K with each release, just in fees. Licensed data sets also make for real difficulties in licensing. Required 3rd parties like SureScripts are openly hostile to open source. Most largest buyers of systems are institutional and most current interpretations of law make it so that open source systems cannot be sold as "sole source" which makes life very hard to close and keep those deals. Finally, until a business model emerges that favors open source and patient health, everyone makes more money with lock-in and so that perpetuates.
Ask me anything.
Can confirm, a little concerningly, that code I wrote 17 years ago is still widely present in the OpenEMR codebase including my old office number for test patients, lol.
I work for a large hosp. operations company and serve as the Dir. Engineering for our clinical operations group. Hacking Healthcare is required reading for new members of my team. It serves as an excellent introduction (with a healthy amount of critique) to the dynamics in the hc technology ecosystem. Thank you for providing this perspective on the industry and its challenges with tech.
We've been successful developing using open source technology internally. In fact, I take a fairly hard stance on disallowing proprietary healthcare specific "solutions" from working their way into our stack (aside from the EHR itself, it has staying power). We're lucky in that we are positioned as somewhat of a startup within a larger org, and are able to take that approach.
To avoid some of the issues you raise, we generally are working to reduce the surface area of the EHR to become simply the transactional backend which is then mirrored to a larger ecosystem of custom apps. This has the effect of boxing in the regulated entity. We focus on data integration (by spending $$$$ on custom HL7 interfaces, unfortunately not everyone can afford) to get outside of the walled garden. This means we can use the information/data for new and interesting purposes without worrying about the EHR vendor's roadblocks/tolls. More importantly to some people, we don't disrupt the billing cycle that originates from the EHR.
Do you notice any trends where healthcare operations/providers are starting to develop internal technology that integrates with the EHR to compliment vs. replace the core transactional system?
It's Duff (David) instead of Fred but thanks. Fred is doing great too. Are you a former CHL/TXR managed or sub-owned group or facility?
Unfortunately I see the opposite trend right now, more silos, more lip service to interoperability, more tolls. I think driven by the burden of regulatory overhead. Moving forward there could be a shift to a "patient owned" record where providers and facilities feed standardized formats into a patient owned/managed "personal cloud". I hope that continues to pick up steam.
Hi Duff! Apologies for the mixup, should have seen that in your handle. No, I'm not part of CHL/TXR.
While not open source by any stretch, I see the personal EHR space being ushered along by companies following Apple's lead. Aside from complex patients, I think the generally healthy/mild-chronic person is uninterested in owning/managing their health data unfortunately. Apple is contributing useful tools to understand fundamental determinants of health including cardiovascular, sleep, fitness at massive scale. It just happens to come with your watch and phone, and their vision for health is starting to come into focus. This puts the patient in the position of generating the primary data (sensors, etc.) and sharing it with their care team on their terms (more or less). As telehealth becomes more prominent, I suspect the patient will be required to engage with their data more often as it will be the means of conveying a shared understanding vs. observations recorded in the clinical setting and stashed away in centralized EHRs.
Furthermore, if labs and other diagnostics are available directly to consumers it puts the individual in a position of ownership. I think the default position is whoever generates the data owns it, and determines how easy/hard it is to share with others. If the individual is empowered to generate information about themselves- this will start to swing toward "patient owned." I too look forward to more of this, but it will have to come with more direct to consumer and digital offerings.
One of the coolest examples I've seen of individuals taking ownership in open source med tech is openaps.org . I'm not one, but T1D's are some of the most resourceful and resilient folks around. Good on them for building a community to solve real problems together. Shout out to the #wearenotwaiting crew.
Hey - I'm trying to get into health tech (EHR integrations, HL7, FHIR etc), do you have any additional "required reading" suggestions beyond Hacking Healthcare?
One of the reasons Hacking Healthcare happened is the huge dearth of material for just this question, unfortunately I don't have anything else on that still. One absolutely timeless work not directly about technology but that is still very relevant is "The House of God". It is "fiction" but it is probably the most realistic depiction of the actual happenings at hospitals that exists and as a result something technologists can benefit from.
Howdy there! I was one of the original authors of OpenEMR back in high school. I'm still good friends with at least one of the other authors. We're always stunned to see OpenEMR in the news, and watching it creep up on HackerNew today has been fun.
I've always been curious why OpenEMR seemed to dominate in the OSS space after we walked away from it. I can only theorize that the code was more approachable than other projects (PHP), and that the GPL kept the work from being captured by any one business. I can't imagine that the code was the best, I'm painfully aware of how poor the security practices must have been in hindsight.
You've given me the chance to ask a question I never knew who to ask: Why, back in 2003 (just after we stopped giving the project attention), was OpenEMR the project you decided to spend time on? What made it the attractive thing to invest in?
If you can tell me I'll bottle that elixir and pour it into every OSS effort I work on today.
Hi. James? I was CTO of Pennington Firm in that era and it was one of many industries where "internet modernization" was happening to a sort of sleepy status quo. OpenEMR with FreeB were the furthest along open source project at the time and so we started there. There were a lot of legacy type problems inherent in the OpenEMR codebase and I think the change to PHP 3 ultimately is what lead to starting fresh with ClearHealth. I'm dating myself but that's around the time that browser AJAX starting opening up a lot of UI possibilities.
Howdy! Nope, I'm one of the two Matts from the Synitech, the original publishers. IIRC the codebase as we left it was heavily into iframes. iframes and SQL injection attack surface.
I'm not sure it used CSS :-p in 2001 or 2002 I actually did a lot of systems work building a version of OpenEMR which booted from CDROM but wrote the database to an attached USB storage device. The idea was that small offices had to start thinking about HIPAA compliance, and could take the disks home from their server each evening for improved security. I think that was probably the last thing I was working on in OpenEMR.
> Unfortunately there are massive headwinds against open source in US healthcare settings.
This reminds me of one of the first articles I've read about Linux and open source in general. It was about a CEO (and largest shareholder) of Medsphere Systems Corp, who open sourced their tech stack (I believe called OpenVista) and was promptly sued by his own company (!)
Unfortunately it seems that the sands of time have eroded the original content (which was apparently hosted on linux-watch.com, which now redirects to a VPS provider), but I've still managed to find something [0] [1] [2]
There is a whole bunch to the behind the scenes of Medsphere and OpenVista. I am not sure what I can say except that a lot of big personalities were involved. WebVista by ClearHealth is still in use by a few very large hospital chains.
I recently worked on a very small, very custom open source 'EMR' system for a friend involved in prenatal care in a developing country, and so what I wonder is are there attempts to get OpenEMR/Clear Health etc into countries and settings where there is not the same huge regulatory barriers? To me it seems like the U.S. is largely lost for a generation for open source/patient-centric EMR, but maybe there is hope for other countries?
Excluding mexico where a ClearHealth derivative still, as far as I know, powers one of the large hospital chains there, no. In my personal experience the venn diagram of countries where there is not much regulation but yet it is advanced enough medically that EMR is a primary problem is pretty much zero. To put it another way most countries where regulation is low have much more pressing medical needs than software.
That's clearly true from talking to several foreigners I know who do medical work in developing countries.
Yet it also seems like there is a lane for a simple piece of software to do basic record keeping. For example, I wrote the app (https://github.com/russnewcomer/SeventyTwo) for my friend to solve the problem (somewhat specific to the culture they work in) where they don't have a clinic site or really the ability to make appointments but instead travel to homes or communities to do their work, and they have to cart around all their binders full of records. My simple app works for their use case, but this also feels like a spot where there are more opportunities to help.
Anyway, I definitely support open source EMR efforts, wherever they may lead, and I thank and applaud you for your service!
In hacking healthcare I talk about "the incredible bandwidth of paper". That's still true. Without really first world software and hardware in a very modern physical setting it is difficult if not impossible to solve medical problems better than pen and paper can. In the practice of medicine pen and paper are really adequate tools to deliver quality care. It is in the business of medicine where large scale data necessitates and benefits from electronification.
Thanks for that perspective. Helps me understand why their use case ( my friend talked about multiple nurses needing to have about 20in of records organized in binders and then coordinate between the nurses in case Nurse A saw someone in Place 1 but then Nurse B saw them in Place 2 a month later...) is helped by the computerization (along with the record keeping they needed to provide for grants/funding), but people working clinical settings that I've talked to are not really interested...
Hi could you please elaborate on how this would work?
There have been several attempts at treating paper as first class - scanning reports, using a digital pen etc. But none of these solutions is convenient. How would you interface paper with software?
My hypothesis—and I could be wrong—is that via innovations around 2-D programming languages, you could invent languages that are natural for casual users to write with pen and paper, that are identical to what a programmer would type.
You'd never get a casual user to write {"weight": 145} on paper, but you could teach them to write:
weight 145
And then it is really quite easy to write DSLs that parse things like that in a type checked way. Or, to put it another way, instead of trying to teach people JSON, why not teach programmers something new?
One thing we picked up on quickly when talking to doctors and nurses in the field is that most of them come up with their own short-hand dialects for scribbling notes (for later data entry, sometimes). Why not make it easy to scan these microDSLs as is, and even define them formally and cheaply, at the last mile that then compile into a more common tongue for ingestion into bigger EMRs? Epic I guess has something somewhat along those lines, but not taken to the extreme.
I'm not convinced that we can try to enforce any sort of format on blank paper for doctors. They fill templates also in weird and unexpected ways. They will write info out of order, in different orientations, outside the margins, etc.
I would be interested in discussing this in more detail. Let me know if I can mail you.
It's a good point. From what we learned each one had their own style, but the key thing was they all had a consistent style. If they are weird but consistently weird, should be okay. And we added things like anyfix to handle out of order.
And yet there is a need for software. I'm supporting a clinic in Asia that is using OpenEMR for records. The government is requiring reporting of diagnoses by category. This is exactly the sort of paperwork that an EMR system should be able to solve well. Since we haven't figured out how to do this with OpenEMR, the staff is doing it by hand. I hope to solve this problem soon, whether by figuring out how OpenEMR supports this or by adding this functionality.
I think there are plenty of open source projects with great UI but that aside I'm not sure I understand what you mean? What type of service for example?
HIPAA greatly complicates a lot of data sharing because of appropriate data privacy issues.
One of the most unpleasant things about working in the medical space is how tightly coupled and poorly modularized systems are.
Obviously, driven by the reasons and pressures you outlined in your parent comment. (Everything is sold and certified as a system, rather than a component)
It seems like there's an opportunity for OSS to eat shared functionality, that no vendor particularly liked implementing, and then allow for closed source UIs to be built on top.
E.g. EMR store/server being the open product, with {insert your preferred front end on top, for your specific use case}
I see. PACs which are the storage and index systems for medical "imaging" data have seen some pretty big in roads. HL7 processing has become dominated by open source. There is huge institutional inertia to overcome in any corner. You really need to offer something 10X better to get over the "no one ever gets fired for using EPIC" mentality and that's a very high bar.
Thanks for the keyword pointers. This is a bit outside my expertise: family on provider side, but I work more on the insurance side of the house.
In insurance, there seem to be some moves towards cracking monolithic systems into pieces for reasons of development agility. ACA limiting admin costs is a huge driver, as companies rightly identified lack of technical agility as a existential threat (no agility = no ability to update automated processing pipelines to changing requirements = manual processing = penalties for exceeding allowed admin costs).
Also, I think there's a generational shift at the executive level from "Buy and trust vendor" to "Own, develop, and operate."
not just ACA but general market pressure to lower admin/PMPM costs due to increased competition and providers beginning to taking risk on directly in insurance
I've noticed a lot of practitioners use PhraseExpander or some other shortcut writing tool to write their patient notes - I'm curious to know how they get around the HIPAA certifications, especially since they are a dedicated key logger on top of any OS.
Do you have any insight in this arena? I wonder if it is because they are not 'the record,' but instead are 'tools to create' the record that is eventually uploaded and stored in another platform?
Might be a tangential question, thanks for the patience
There is a lot of controversy in this area. Medicare rules are pretty clear that to the extent tools like that are used systematically to enhance bill-ability they are prohibited. Malpractice litigation is having a field day with computerized systems which is why so many states are being pressure to institute caps. Pretty much everyone tries to use templating tools to increase bill-ability to some extent. Healthcare is rife with conflicting goals.
The underlying problem is that we need an economical way for doctors to have more time to spend in the room with patients but no one, patients included, wants to pay for that.
I really hope "concierge" medicine, a lot of that now happening on the lower priced end not only for "luxury patients", continues to take off. You pay some cash out of pocket but get care that is dramatically better and more preventative.
Concierge medicine won't solve anything at scale. We have a shortage of physicians, especially in primary care, and it's only getting worse as the population ages.
What we really need is just more doctors. The federal government should remove the bottleneck by boosting funding for residency program slots.
Woa -- do you have a link to the regulation (chapter, but looking at Medicare fully now) to things that are designed to "systematically to enhance bill-ability"?
So you're saying if you use a templating tool to be more efficient and save time, it's explicitly not allowed AND you're opening up either yourself or the technology tool to malpractice litigation?!
Malpractice is a totally separate thing from Medicare Fraud. With respect to malpractice, having automated or semi-automated encounter notes with huge systematic similarity looks absolutely terrible if something goes wrong. Don't take my word for it, consult your attorneys.
With respect to Medicare fraud see the False Claims act and its interpretation in the enforcement actions against many parties beginning in 2016. Keywords would be like "not medically necessary" and "upcoding". I believe the monster enforcement action just this week against GHC also involves that.
Let us not forget “stand in your shoes” Stark provisions which made many a brother-in-law who owned a DME company very sad.
Hospices dropping off cookies with their contact information is kosher. Incentivizing a medical office to hit 100 nebulizer-orders, or cpap referrals -> federal prison.
My experiences in healthcare/EMR were focused on Medicare part B (home health); physicians, NPs, and PAs that made house calls.
This is a great question; being thrust into hippa compliance from 2000 - 2012 was the easy part, and can perhaps be summed up as
“
fax machines behind a door, paper charts and/or laptops/tablets locked in trunks*, all-in Blackberry enterprise services, Exhange over ssl xml-rpc endpoints, and ensuring EMR portal uptime was so much of the fun parts.
The hard part: training home health medical groups how to VPN - and how to never mix work tech and personal tech.”
I am unaware of hippa changes since the blackberry-era, and while I can’t imagine specific apps being named in amendments to the law, I can say without a doubt that hippa at the time meant having total control over all technical Devices used by your clinicians, prepping all laptops and equipment to use encrypted storage, ssl-encrypted email, VPNs, etc was always made easier by focusing on training, troubleshooting, and being able to mitigate the raw chaos monkey power of a doctor with a gadget.
With the passage of the HITECH act, being able to facilitate the automated logging of a doctor’s conversation with a patient’s family or caregiver towards the CPO (care plan oversight) and making the fulfillment of PQRI initiatives automatic... “so easy a doctor could do it”... meant (at the time) a very hefty pay raise for part-B collections.
TLDR: hippa was easy, training doctors to change their ways is hard. producing the tech that makes their lives easier means assuaging them of their worries that they are maximizing Medicare collections.
It is fun to imagine these days that transcription, location, and barcode technology has “made it” to the point we dreamed about ~10 years ago.
Unfortunately there are massive headwinds against open source in US healthcare settings. Regulation requires certifications that cost upwards of $100K first time, $10K with each release, just in fees.
Seems to me like this could be overcome by licencing but not in the general sense but more in a you need accreditation, join this other pool of people utilizing the system and buy an accreditation licence. Seems like there would still be a value proposition there.
As for the 3rd party seems that could be hit or miss again if it is money, build those out as modules that cover the licence fees the third party is looking to recoup. Maybe with a little bone in it for the open source developers as well.
Thanks for AMA. My previous startup was in self-insured employer space so I only saw the issues you're describing from a distance.
> "Finally, until a business model emerges that favors open source and patient health, everyone makes more money with lock-in and so that perpetuates"
I'm curious to know if you've thought about what such a business model might look like. The closest to a viable business model I've seen gives patients control of their data & allows them to monetize it. But that feels like a pipe dream at the moment because a) EHR vendors don't have incentives to share data, and b) there is no marketplace of buyers for said data.
As with anything in the b2b healthcare space, most of these systems suffer from quite a bit of legacy and at-best-average code quality. Despite that, many doctors, clinics and even small hospitals use them because the private solutions (think Epic [1], but smaller) aren't necessarily better code-wise (don't ask me how I know). I wish more FAANG-calibre devs would look into contributing to and evangelizing these platforms rather than writing yet another note-taking/"productivity management" app. It has a direct impact on the quality of care delivery in certain parts of the world and a positive impact on tool-related clinician burnout [2].
I've worked in the space for a few years and I'd discourage anyone from trying to build a career as a software dev in the Healthcare IT / EMR space. It's extremely sales driven (devs aren't valued at all), code quality is terrible and the systems you write are mostly for the benefit of insurance companies / compliance than doctors or patients.
I think there was a YC funded iPad EMR startup that tried to be cool / hip / provider first until they got smacked in the face by reality.
> It's extremely sales driven... and the systems you write are mostly for the benefit of insurance companies / compliance than doctors or patients.
This.
Once I worked making healthcare software that would basically save costs by avoiding complications. We were bidding into a large hospital network. After the long sales cycle, we checked the most boxes, clinicians liked the feel of ours the best, they said ours was the most intuitive, helped them do their job the fastest, etc.
The clinicians weren't the ones writing the check though.
Our competitor approached the main insurance provider in the area and convinced them they could save x% in additional claims if the hospital network would adopt their software. Competitor's deal was to split the bill between the insurance company and the hospital network. To the admins writing the checks, they had the choice between two vendors that more or less did the same thing but one came in at half the cost. Clinicians' preferences didn't matter, it was a no brainer for them.
No amount of software engineering would have saved that deal.
I have seen in healthcare more "bribery" than in any other industry I've ever worked. Whether quasi-legal in the shape of various kickbacks to outright illegal gifts of cash and goods. It is sadly endemic to the industry.
In part I think it happens more commonly because there is so much "other peoples money" in healthcare. Everyone is spending from someone elses checkbook and so transactions seem very distant.
Accountable care models are the next step now that HMOs are widespread. restoring financial burden to the patient seems backwards when considering systems like universal healthcare, but ends up reducing inefficiency (identifying genuine surplus, not things that end up costing in the end like reducing preventative care)
Not all healthcare systems are beholden to private insurance (my own country's included) and that shows in where these systems are deployed.
Also, compliance doesn't fully explain why competitors are able to convince clinicians to switch systems. Word-of-mouth means that people will know your EMR is a flaming piece of garbage, but (as you noted) companies would much rather cut up-front prices so they can milk an extended contract than improving product quality. All that said, I think there's a bit of a chicken-and-egg thing going on here. Good people don't join the space because the culture sucks and the pay is bad, but those are because those with the talent and drive all self-selected out of it. I get it, health IT is a huge drag and not at all sexy. But just look at how often folks on HN ask about "doing social good" and how many complaints there are about healthcare delivery. Trying to run a "disruptive" VC-backed startup is IMO pretty crazy, but contributing to an OSS project is far less risky and more achievable.
> Word-of-mouth means that people will know your EMR is a flaming piece of garbage
The problem is that the Venn diagram between the people who actually have to use or administer the EMR and the people who decide which EMR to implement is basically two entirely-disconnected circles.
That's often but not always true. With the caveat that is in a non-US country, hospital EHR purchases are usually done the way you describe, while doctors will often personally make EMR purchasing decisions for their own office or private practice. Government procurement and small business sales seem to be two very different beasts in this regard.
Yeah, my experience with healthcare IT was in a two-hospital rural provider in the US, so that definitely colors my observations there.
That said, I don't really recall the doctors themselves using an EMR much, either; they'd usually punt that to assistants and/or nurses. Probably different in smaller practices/clinics.
All these hip, cool EMRs always show a glamour of a patient. I have yet to see a medical practice or health care facility that has a photo studio as a side business!
Maybe not a side business, but it's not as ridiculous as it sounds! A friend of mine was an in-house photographer and graphic designer (I think he had some sort of managerial role, too) for the local hospital network until he got laid off when COVID hit. Basically, in addition to being a good photographer, he had to be aware of the various patient-privacy regulations involved and also have a practical working knowledge of the hospital so he and his team wouldn't be getting in the way.
> I wish more FAANG-calibre devs would look into contributing to and evangelizing these platforms rather than writing yet another note-taking/"productivity management" app.
I would like to see the US Digital Service continue to task technologists with improving EMR systems at CMS (Centers for Medicare and
Medicaid Services), but made free to use by all practitioners and citizens (and of course, open sourcing the resulting codebase). It seems sort of inefficient we keep reinventing the wheel (Epic and the like, which are crazy expensive, or self hosted solutions, when practitioners should not be spending time maintaining EMRs), when your records should be stored for your benefit by your government over the course of your life. This is where, imho, high calibre engineers provide the most leverage (one way ratchets on public goods at scale).
The VA sort-of does this, at least when I worked for them. The issue they had when I was there was depending on where you went, you may or may not have access to the record, because the VA didn't use a central system, it used many systems across the country, each with their own records(basically their own mainframe). We once added a hospital to our system, and had to have dual workstations because the systems couldn't be easily merged, and they had to look up patients in both systems.
Also, with Veterans Choice, I don't know how much there was an effort to bring this data back. Same thing with the DoD, for a while there was an agreement to send medical records for active duty to the VA, but then that got pulled for a time.
I believe there was a huge undertaking to consolidate these to fewer systems in the last few years, but Vista[0] (the VA's EMR) is pretty scary. I wouldn't wish it on anyone.
The VA has something called BlueButton that looks really cool (https://www.va.gov/bluebutton/) and I think should be standard practice across all EMS systems (one click export of all a patient's data to a single text file).
The file format itself seems like a bit tough to parse, but the concept I love.
Blue Button technically is an industry-wide standard though its origins were indeed in the government. I've seen it supported in a couple of the private healthcare systems I've used (though I haven't ever used the resulting data download unfortunately!).
TIL that the new system is Cerner. That's depressing. Thanks.
EDIT: Still a win I suppose if it improves care delivery over the status quo. Nice to know there's still some progress on this front [1] [2]. Looks like I owe OpenEMR a financial contribution.
That's imprecise shorthand on my part. s/"FANG Calibre"/"objectively talented and used to/capable of negotiating good compensation"/. There's a degree of Stockholm Syndrome in healthcare tech where people don't know what a well-developed product or codebase looks like. It's unlikely to change from the top, so getting more technical folks with higher leverage into the field is IMO the next-best option.
And yes things are changing at a glacial pace, but they are changing. For example, my province is developing a new patient portal [1] out in the open. AFAICT, they seem to be doing everything aboveboard: CI, code quality standards, documentation and proper testing, etc. Yet if you look at another team in the same org (ministry of health), you'll find non-existent dev practices, oodles of VBA, or (even worse) some slow+buggy third party system put in place by one of the procurement vampires (IBM, CGI, Deloitte, you know the bunch). The biggest difference? The former project has a dedicated, US Digital Service-style team of skilled and hopefully better-compensated dev(ops) people who know how to deliver good software.
I think, in fact, you have it exactly backwards. You are very unlikely to find a decision making authority (person or committee) in a medical practice who gives the slightest bit of consideration for the development practices or code quality of software they're considering.
These people have three things on their mind (in no particular order): 1) does this product meet this organization's requirements under our regulatory compliance policies; 2) does this product (including installation, maintenance, and training costs) fit within my budget; 3) is this product widely known and trusted by my peers at other medical practices.
Notice something? The word "software" doesn't appear in that even once. They literally don't care. The result is that companies develop products (software) on the cheap, and that results in the quality issues that exist.
Improving the quality of the code base and development practices is solving a problem the purse-holders (customers) don't have.
I agree that all the factors you've listed are in play, but they are far from the only ones involved in decision making. Perhaps this is a regional thing, I've never experienced (and have no wish to) what the US EMR market is like.
Some complaints/feedback I did receive from doctors and clinic admins while working for an EMR vendor:
1. Your system is buggy/unintuitive and we hate using it.
2. We're not upgrading or moving to your new system because of 1).
3. We're moving to competitor X because they have Y feature.
4. [conversely] We came from competitor X because their EMR is slow/buggy/lacks features.
5. We signed up because the docs/office assistants liked [hero feature] in the sales demo.
So yes, 0 mentions of the word "software". However, all of these are directly related to the software itself. There's a reason flashy new companies can swoop in and steal some market share (at least where I am). Even more importantly, there are many tech-related reasons why some companies start floundering and drop out of the market:
- bad foundations (most EMRs were created by doctors with limited dev experience)
- rampant tech debt driven by feature-driven development
- lack of knowledge about testing/CI
These are not theoretical problems. More than once, we incurred regulatory fines and SLA penalties in excess of the "cost of doing business" threshold. After a pretty major patient data screw-up, upper management even relented and gave the dev(ops) team time/money to clean up their act. Regulatory and bureaucratic inertia may insulate health IT companies from software engineering issues, but there's a limit to everything and they can sure as hell bleed.
It’s not about being sloppy. It’s about ego and hubris.
Just want to add: I work with (ex-)FAANG folks on a daily basis, too. Not all of them have egos bigger than their britches. But the ones who think “this industry just needs better software, and I can write it” sure as hell do.
I work in the medical research space and we have to integrate with EMR systems to get our data. I don't think software is the root problem, but rather the root problem is "there aren't incentives for good CMSes"--namely, there's no incentive for systems that talk to each other because healthcare consumers don't think about this when choosing a hospital and hospitals don't have any incentive to make it easier for their customers to leave their system (and EMS vendors certainly don't have that incentive). Ultimately the question is "why do we believe EMSes are valuable, but no one can figure out how to make money from making them better?".
I used to work in Bioinformatics. Getting Epic and Cerner to flow into our i2b2 or REDCap was a mess.
Epic was a real disappointment in terms of their willingness to implement simple features that would have truly helped researchers analyze more specific outcomes. But I guess you gotta make money some how.
Yes, I absolutely agree that there is a problem with greed, ego and hubris in sillicon valley. Just wanted to point out that not all engineers fall in that category. I think that confidence in writing solid software when paired with humility can lead to good outcomes. To me, it seems like the ego and hubris issue can cause the most amount of problems when an individual (e.g. zuckerberg) takes on a savior and machiavellian complex. It seems like most of them actually believe that they are making the world better and they don't see the harm that their self interested blood thirsty capitalist side does at all.
One of the hurdles of EMR systems is that there is a pretty significant minimum viable product due to various standards that can't be ignored. Thankfully, this space is far more open now than it used to be; HL7 for example used to require payment just to see the standards. Pretty much everything you would need access to (HL7, CCDA, SNOMED, LOINC, ICD-10, etc) is freely available now!
Generally just having an EMR system is not enough; you also need practice management, scheduling, billing, insurance claims, etc. Interoperability between separate software for these things is... tenuous at best, though some practices do manage to handle it, it can be very fragile. Hence integrated solutions are pretty much the best way to go, and also prevent disruption from competitors which may be better in one space but not another, since it's so hard to get them to talk to each other well.
> Pretty much everything you would need access to (HL7, CCDA, SNOMED, LOINC, ICD-10, etc) is freely available now!
The AMA’s CPT-4, incorporated as a component of HCPCS, is not free, and is the mandated code set for most professional procedure coding.
And while otherwise that may be true for most of what you need for core EMR functionality, everyone wants EMR and billing/insurance transaction handling to be modules of the same core system (because you are going to need both, and they need to interface smoothly to avoid a whole lot of operational friction), and most of the mandatory billing/insurance standards are decidedly not gratis; older versions of at least the X-12 standards in this space were subsidized by CMS and available for free, but that hasn't been the case for the versions required since 2010. And that's just basic transaction standards, a lot of the code set standards are also proprietary.
(In addition to not being free, the standards in this space are exceptionally poorly written, ambiguous, self-contradictory, and incorporate vast quantities of external material, often also not free, by reference—and often not hyperlinks, but “here is the name of the document and the postal address from which you can contact the entity from which you can order it.”)
I think what would help us the most is not writing software, but instead explaining the requirements in detail (like a specification). There are many people looking for a nice self-contained FOSS project to work on, but many don't know where to look and joining an existing codebase might be too daunting.
There will be an Uber of this space, someone who says "I understand the problems these laws are trying to solve, unfortunately they are a kludge and we can solve them much better with better technology". So complying with all these standards will be a second priority done for backwards compat for the person who comes in and disrupts this space.
Yeah its a total mess, most health systems I work with have 30-50 different vendors all with some various forms of integration with the EMR... It's always a mess, with no end in sights.
I was release coordinator 5-7 years ago for an EMR software (DIPS) in Norway, to one of the bigger hospitals (Kalnes).
It was said that there was a unwritten policy that they never use open source software, the only exception was for the ERP database running on Linux. There was more than 2000 systems in that portfolio to various hospitals under the same policy.
After checking with other employees if found the reason; they had to make sure the supplier did not have to many levels of sub contractors, and had to be close to the core development.
So not open source in it self, but open source was seen as a big red flag.
In this space it's worth mention OpenMRS https://openmrs.org/ which also aims to do the same thing. Most of its deployments are in developing countries (I was part of a team driving adoption of it in Kenya) but I'm yet to see a successful large scale deployment of the platform.
Experimented with it and seemed to scale well.
3x load balancers (nginx or httpd or haproxy or iis) https only
3x apache tomcats to deploy openmrs war ... https only. azul jre with -Xmx=2g. Tomcats were also as close to default configuration as possible.
3x mysql clustered ... https only
No issues on any of these OS's:
Windows Server 2008r2
Windows Server 2016
CentOS 7
CentOS 7.5
Raspian Lite
MACOS
OSes ran on virtualized (hyper-v, virtualbox, xen) and baremetal. Commodity hardware (old laptops, desktops with at least 4G ram. and PIs) and real server hardware. Purposely used mixed computing resources to simulate what a team say in Ndola, Zambia might have at their disposal.
By "scale" I'm not referring to OpenMRS capabilities t run on commodity hardware. I'm referring to a large deployment at a national level or at a large public hospital in Kenya. Most deployments have been pilots and at smaller establishments.
Ultimately it would depend on quality of the software. The community behind it appear to be very engaged so if an organization were deploy it at scale (national level or large org) and offer them amiable incentives to support the application, it might become the standard.
Obviously there are upsides and downsides to this, but IMO it significantly cuts down on clickbaity titles at the expense of some readability when the site being linked to chooses a less-than-descriptive title.
EMR is just an automated way of deploying open source components (Hadoop and co.) - there's some glue code there but the equivalent "open" version is probably the Hortonworks stuff (now owned by Cloudera): https://github.com/hortonworks
I clicked on "features", trying to find out what EMR means, and was greeted by a banner that really clarified things:
"ONC Certified HIT! 2014 Complete Edition EHR!"
I've been coming back to the project intermittently over the last few years and have been pleased to see the progress. The reason I came back this time was that our managing partner just told us we'll be paying next year in our small outpatient clinic--truly unreal. I dream of a day when we could just use our own system, maybe something that I could even manage myself (maybe not realistic?). Bravo to the contributors of this project!
1. Certification, in the USA if you want to get paid from the government you must be certified. This isn't a small undertaking at all and requires quite a bit of development and then the cost of actually doing the third party testing. It has driven companies out of business and forced consolidation even among the larger players.
2. Configurability. EMR's are crazy configurable to meet any hospitals requirements. This means lots of consultant hours to get things setup and running. Take a look at how much money Epic and Cerner make just from "consulting".
3. Interoperability. Again, there are standards like HL7 and FHIR are widely used but the data isn't always great. We are seeing more and more API endpoints all of this requires a level of customization.
All of this adds up to a ton of cost for a small-ish market with a large pool of no or low profit buyers and pretty much a replacement market.
Oh, and you are building software that could cause harm or death. I can't imagine why people don't want to come into this industry and really push the state of the art.
Open EMR was the first choice for our project, but it did fit our workflow needs and the UI/UX is also quite outdated. It's FHIR api also had minimal support. So we ended up building our own EMR on top of HAPI FHIR repository, (https://hapifhir.io).
There is definitely support, but it is nowhere near what is documented in FHIR website. You can look at the search capabilities for example : https://www.hl7.org/fhir/search.html.
That's still better than a lot of healthcare software. I have no issues with it running PHP, CVS is more of a hassle for devs. My current EMR only recently came out with a UI that works outside of IE and their proprietary ActiveX controls.
Many healthcare systems are a COBOL-dialect all the way at the bottom. Some of these had PHP layers shimmed in, when the web became a thing.
I've seen php scripts that shell out to .bat's, that interface with the COBOL engine. It's a mad world.
For context, a large amount of healthtech software was written in the 80s (kind of like fintech, the difference is that there's no competitive advantage to having better technology in health).
I showed someone from the Allscripts innovation group what I could do in an Elixir repl once, and his jaw hit the floor. Then I showed him how we wrote parsers. He said we'd never make it because we turned around new features too fast for anyone to trust us.
Haha, among the most popular EMR you'll find a snarl of Perl, PHP, VB, Mumps/M, C#, old Java, cobol, and several proprietary languages. There is a small number of people who die in the US every year do to medical mistakes attributable to software bugs.
Would it make you feel better to know your PID is being stored in a database language where the only data type is strings, and there is an intrinsic command to interpret any string as code?
Kaiser spent $4 billion on implementing an EMR with Epic, after abandoning their own project developed with IBM. Imagine if that kind of investment was directed towards a open source consortium. I realize these kinds of projects are incredibly complicated to do correctly even for a single customer, but the benefit of a shared medical record nationwide would be enormous.
I would not accept EMR unless my records are encrypted and can only be unlocked with a smartcard that remains in my posession. Or something close to that.
So (hope it never happens) if you have met with a bad accident, the first responders must break in to your house, call lockpickinglawyer (youtube lock picking guy) break your safe open, retrieve your smart card, then bring it to hospital and start your treatment? And I thought time is very critical. Privacy is important, but its implementation must be reasonable, come on.
You're often in that condition anyway. If you're taken to hospital in a coma or stroke condition, they won't necessarily know who you are nor will you be able to consent to any procedures. They proceed on good faith in that case.
You can read quite a bit about open source in healthcare in my book, Hacking Healthcare. A bit dated but still in print.
Unfortunately there are massive headwinds against open source in US healthcare settings. Regulation requires certifications that cost upwards of $100K first time, $10K with each release, just in fees. Licensed data sets also make for real difficulties in licensing. Required 3rd parties like SureScripts are openly hostile to open source. Most largest buyers of systems are institutional and most current interpretations of law make it so that open source systems cannot be sold as "sole source" which makes life very hard to close and keep those deals. Finally, until a business model emerges that favors open source and patient health, everyone makes more money with lock-in and so that perpetuates.
Ask me anything.
Can confirm, a little concerningly, that code I wrote 17 years ago is still widely present in the OpenEMR codebase including my old office number for test patients, lol.