Hacker News new | past | comments | ask | show | jobs | submit login
There is no “software supply chain” (iliana.fyi)
252 points by xena on Sept 19, 2022 | hide | past | favorite | 210 comments



I do software supply chain security consulting for several high risk companies and largely agree with this post that we must stop expecting devs to have any responsibility for code they produce. The responsibility is on those that consume it.

This will sound pretty harsh, but if your company chooses to use open source code that does not have capable, paid, full time professionals reviewing it for security and quality, then your company is signing up for that responsibility. If you make no reasonable attempt at vetting your supply chain and harm comes to users as a result, then IMO you should be liable for negligence just like a restaurant serving food with poisonous ingredients. Broadly normalized negligence is still negligence.

This should not be controversial, but it is. Washing hands in hospitals was once controversial too but those advocating for it had irrefutable evidence on their side. The medical industry did not want to pay the labor cost of hygiene, and we are seeing the same in the software industry.

https://www.nationalgeographic.com/history/article/handwashi...

Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something in-house focused on your needs on top of the standard library. Pick one. Both are real options. Some of my clients actually do (or pay others for) security review of every single NPM dependency they use in prod. If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Find a leaner path.

Companies must stop expecting others to do their software review job for them. OSS devs already wrote the code for free because, ostensibly, it was fun. You are an ass if you ask them to do anything that is not fun, for free, to make your company safer or more money. Such actions make it not fun anymore, and make them stop entirely.

I do not know why companies have code review policies for code written by peers, but if the code is 2 million lines of NPM dependencies essentially copy/pasted from randos on the internet it is suddenly okay to ship straight to prod and give said randos full control of the data or property of millions of people.

We need to start calling this out as the negligence that it is.


I want to first say: I agree with your comment.

I think the problem is that companies would say "okay, let's take the leaner path"; they then proceed to implement their own (say) HTTP library, riddled with bugs which will never get fixed, because it will never have the same person-hours sunk into it that a (major/respectable) FOSS HTTP library will.

(And the number of times I've watched a dev attempt to implement some standard while steadfastly refusing to read the standard. Then I proceed to find bug after bug, trivially … because I am reading the standard…)

But you're right with your hospital analogy: it's that the company does not want to put forth the resources to do the job right, either by doing it right themselves or by doing the verification work & upstreaming fixes; they'd rather put forward a bare minimum to do shoddy work.

And in all my years of experience, I still am no closer to understanding how to fix that.

> We need to start calling this out as the negligence that it is.

People absolutely hate this, IME.


> their own (say) HTTP library, riddled with bugs which will never get fixed, because it will never have the same person-hours sunk into it that a (major/respectable) FOSS HTTP library will.

The interesting trade-off here is that they probably aren't going to make the exact same mistakes as everyone else.

Which means more vulnerabilities, but fewer industry wide vulnerabilities where everyone gets pwned.

I'm not sure that's worth it, but it is something that should be mentioned when discussing the situation.

edit: me -> be


> (And the number of times I've watched a dev attempt to implement some standard while steadfastly refusing to read the standard. Then I proceed to find bug after bug, trivially … because I am reading the standard…)

This is a large part of why I expect the binary protocols HTTP/2 and HTTP/3 to be more reliable and less buggy than the textual HTTP/1 once all the dust settles: you won’t get far without reading at least parts of the specs, and they’re harder to implement, so there will be fewer implementations, with a higher average quality.


If you implement HTTP, surely your service is behind nginx. So it does not matter much, nginx is not going to send evil data to you.


The third option is to pay someone else to certify things (maybe the developers of the thing, or a third party entirely).

The problem gets out if people find out “Caddy is certified by Google” and then don’t bother doing their own.


> And in all my years of experience, I still am no closer to understanding how to fix that.

Does it need fixing? This may be a case of the cure being worse than the disease (the velocity loss outweighing the security risk).


> Companies must stop expecting others to do their job for them. OSS devs already wrote the code for free because, ostensibly, it was fun

This is not really true. Open source is much more than hobbyists doing things in their spare time for fun. A huge amount of open source is developed by and released by paid software professionals as part of their jobs. Some of those companies are directly doing it as part of their actual business offering. Others are doing it because they value owning the mind share in a space. Then, a huge amount of other OSS is developed by people who want to enhance their professional reputations.

EDIT: I see the article actually confines itself to OSS software that was written as a hobby. In that case I think my words above are probably out of context. But by the same token, the whole article scope is diluted a lot. Yes, if you have pinned your project on something someone wrote on the weekend for fun with no intention to maintain it, you've got problems. But let's not brand all of open source with that problem.


It doesn't matter if it's written as a hobby project or not. People need to stop making assumptions and just expecting work to be done for free for them.

No wonder why maintainers burn out and companies are hesitant to open-source projects... random people come by and make demands with zero plans to pay for it.


I think the reality is, if its a hobby project supported by one person then money doesn't really solve it. No company should put itself in the position of being reliant on some random individual on the internet whether they are willing to pay them or not.

Of course, adopting this stance takes away opportunity for many small time open source projects to become popular in the first place, and the general incentive and opportunity for open source to succeed.


Money solves this by paying the original person or someone else to do what they need and provide the level of service and guarantees that they need. The whole point of open source is that it doesn't have to be the original developer.


I'm glad you brought this up. It's really core to the value of open-source as an activity. It creates the possibility for a project to become much greater than what it would have been if it was limited to the person or organization that originated it.


If one good thing can be said about the crypto sphere, it is that it is the only corner of the internet where nobody expects to get anything for free. Access is already gated by a wallet containing real money; you're going to use it. This leads to some surprisingly healthy dynamics around services and software that cost money to run but would be hard to charge for in web2.


> random people come by and make demands with zero plans to pay for it.

Perhaps we should then oppose the whole "hustle culture" which such attempts are part of (i.e. make demands without willing to pay, see whether they stick).


I generally advise my clients to mostly trust open source with lots of well known and documented professional eyes on it like reproducible builds of programming language compilers, standard libraries, and well maintained OS kernels.

Where I normally have them focus their resources is on the often thousands of dependencies that are, mostly, written has hobby projects by randos.


This is a thought-provoking take (as someone who really thinks open source devs have no obligation to their users). It is clearly possible to abuse that -- just dump the software necessary to use your product into open source ecosystem, abdicating responsibility while still technically supporting your product.

I dunno. I'd say that this still doesn't establish a customer/provider type obligation -- the open source code is still offered without any quality guarantees -- but it is entirely fair (and reasonable) to be extremely skeptical of a company that doesn't offer a contract-defined paid relationship for support.


These are all good reasons to contribute to free software but I think you've missed the biggest one of all, the fundamental engine that makes the entire concept work: making the software you already use, more useful. If you need a feature or a bugfix, you can write it yourself, upstream it, and never worry about it again. That's a big part of why companies employ people to work on free software, and it applies just as much to individuals.


And part of ostensibly "corporate" OSS is the stuff that devs convinced management to put out there because they didn't want someone else to reinvent that particular wheel. It might be supported for as long as it's used internally (and if you're lucky, some devs will continue unofficially supporting it for a while after that - but at that point we're already into de facto hobby territory).


I frequently find myself advocating for better supply chain security. But if I asked developers (or their managers) to review/audit 2,000 NPM dependencies, I would almost certainly fail. No other company they know of is doing that, and asking them to start is predicated on the entire industry being wrong. That tends to be a tough sell - even though I agree with you, convincing others is a whole 'nother ball game!

What kind of arguments do the organizations you consult for find compelling? I find it extraordinarily difficult to convince others. There is a strong bias towards the status quo.


I often suggest that they either review it all, pay a company like mine to review it all, or do tree shaking to remove most of the deps and make it lean enough that they can afford to maintain reviews.

Failing those when a legacy codebase is just too big to simplify or review, I will generally pivot to retrofitting in accountability to something that can be trusted. This often means working with the team to design small, auditable, reproducibly buildable standalone services for security critical functions that can run in a TEE or HSM. These often take on critical signing, encryption, and policy enforcement responsibilities such that a compromise of any single employee or the larger legacy application can be tolerated.

Gotta have multiple knobs you can turn so the org can choose the cheapest path that honestly meets the required threat model.


It's a "thing" in the Medical Device industry. I don't recall if it's now mandated or merely suggested in the recent FDA Cybersecurity Guidelines (still a Draft, not yet a Regulation), but creating a Software Bill of Materials for OTS/SOUP that's in your project is now recommended. Doing this allows you to track and analyze the impact of discovered vulnerabilities that may be in your shipped product.

I'm far from a security expert, but the concept of creating and maintaining an SBOM seems to be gaining traction across a number of industries.


Why do you think this is controversial? Whether a company works with another software company that is bonded and/or a person uses OSS if something bad happens to the customer it still is reflective on the company using the software in a negative way. No company would refute that.

I rarely have seen court cases in regard to customer damage try to quantify negligence, because the court system is missing a lot of nuance in our industry. Pragmatically speaking the courts are ruling on the severity of the customer impact. There can and will always be an argument that is subjective about negligence in regard to how much you protect yourself from a malicious event vs the severity of said event. This isn't specific to software engineering either like concert venues that are mishandled and result in accidental death.

Your comments around npm dependencies not being reviewed and shows an engineering team is negligent seem contextually correct depending on the damage of said system the engineers are managing. If it's a bank system that leads to fraud then I agree. If it's a start up that runs a website; I hardly categorize this as negligent. Every company I have worked for has understood this trade off. If you are trying to be over zealous about the definition of negligence then I could understand how that would be controversial.


Do you think CTOs should be held criminally negligent if they lose data that is of national security concern?

Because that’s the only way I see us getting audited dependencies as a commonality.


I would agree fintech companies, or any company managing a lot of PII, must take supply chain security much more seriously than say a fashion blog or a video game. The level of negligence is anchored to the potential for harm, but it is IMO a rare case where a successful company manages without a lot of PII or payment details. Even Deezer, who /only/ sells booze, was sued for a data breach and had to give out thousands of $10 checks in a class action. Their negligence hurt them and their users.

For context, most of my clients are high risk with large PII footprints or various forms of fintech. Even in fintech and banking, dependency code review is unheard of, and supply chain attacks are happening in the wild targeting those orgs.

I kind of do intend to come off alarmist about this, because it is very alarming and is likely going to get a lot more people harmed than it already has.


I think you meant to refer to drizly. I would say this is an example of a company that put their priorities on growth rather than security and it worked out for them. It sounds like Drizly didn't think about security at all, and in the end it cost them 0.6% (worst case they settled for $7M and were aquired for over $1.1B) of their value. Looks like their executive team prioritized the right things to me.

Making a decision for or against more security is more about risk mitigation. If the courts are just going to slap companies on the wrists for data breaches I don't see a strong argument for intense security protocols for your run of the mill e-commerce business.


Ha ha. Yes. Drizly is what I meant to say. Too late to edit now.

I would say it caused them reputational harm as well. It would have likely been a lot less trouble to just hire a capable security engineer or two and do some basics.

To your point, we need these things to hurt a lot more, but it is a start.


People have been conditioned to ignore security. Too many big public incidents, too many emails telling them their data was exposed. They don't care anymore.

I think you're fighting an uphill battle here.


It took something like 100 years to normalize washing hands in the medical industry. I am in this for the long game.


Well of course security consultants believe every company should hire security consultants to review every dependency in the stack all the way down.

But for the people who sign the checks, do you have ROI numbers to support this approach?

How many exploitable vulnerabilities in open source dependencies that could have led to financial loss have your reviews found that an off-the-shelf SCA tool didn't find?


FWIIW I advocated for external review of security critical dependencies and hired external firms for this long before I started my own firm.

The ROI is not having your company ended by someone malicious that reviewed that code before someone acting in your interest does. That said, I normally work with companies that -really- need users to trust their security practices so this can become a sales and marketing win. When competitors are forced to admit they do not review the code that handles customer data or money, and you do because it is obviously the responsible thing to do... it helps customers choose you.

As to your practical example question, many companies depend on Terraform, for example. Until I reviewed the AWS password generation function it was producing passwords seeded by the current time. An attacker that knew what time an account was created had a path to guessing the password.

https://nvd.nist.gov/vuln/detail/CVE-2018-9057

Do we find things this impactful super often? No. But when we do find them all of our clients can get early warning and be recommended general practices that avoid this type of issue entirely.

We have also privately exploited supply chain attacks when legally possible more than once as demonstrations for our clients.

While the best examples are public, you do not have to look hard to realize supply chain attacks are all too often -easy-.

Researchers did a scan of all npm packages who have maintainer email addresses with expired domains. AKA packages anyone can buy their way into control of.

https://arxiv.org/abs/2112.10165

Thousands of them. I bought the maintainer email domain of the NPM foreach package just to prove a point.

Generally though we spend most of our time doing blue team work. If you architect your application well with well placed secure enclaves you can get away with any single system or human being comprised without a loss of user data or funds. Then it becomes possible to more freely use libraries and move fast in other parts of your stack with lower risk.

Reviewing 100% of all deps company-wide is not the only knob one can turn.


I have two thoughts to share about this from what you've said:

> we must stop expecting [Open Source] devs to have any responsibility for code they produce.

Yes, I agree with this. And something that's been in my head for a while, which often discussed, is how people often "magically think" something along the lines of:

"If we just paid these devs for support, then we'd solve the support problem!"

Unfortunately, I don't think is reality. If you're a dev working at Google and you toss a library on NPM, you probably can't be paid to support it because your employer forbids it.

The type of person writing libraries Open Source is not always going to be motivated by money!

> Some of my clients actually do (or pay others for) security review of every single NPM dependency they use in prod. If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Find a leaner path.

There was a post a few months ago[0] that has been talking about a way to "democratize" this review process. It's been sitting in my head and it's actually a part of a project that I'm working on now[1]. (My apologies for that not being documented anywhere yet though!)

Essentially, if every company on this planet has to review every Open Source dependency, that's a massive waste of human effort because there is a ton of overlap between that work.

Instead, what if you could have one person/entity do that work to review the dependencies and then you "buy" that dataset for a fraction of the price? If it takes 10 humans a year to review every dependency you have, then it's cheaper to pay 1 human salary to buy that data. (And we do the work once but then sell it 100+ times.)

That's the core principle that's been in my head for a while and that's how we're trying to build a company right now.

Unsexy problems aren't very "shiny" (like Stable Diffusion or NFTs) but they are important!

0: https://news.ycombinator.com/item?id=32037562

1: https://github.com/lunasec-io/lunasec


I also am spinning up some projects and tooling to have standardized, cryptographically signed, language agnostic, vcs agnostic, and publicly publishable/searchable code reviews for any open source project.

I have been thinking about this deeply for years, built internal tools like this for companies, and been pleading with package manager maintainers for collaboration.

Some of my clients have made said they would strongly consider publishing internal dependency reviews to such a platform if it existed.

Sounds like we have some common goals. Would love to keep in touch on new developments. Contact info on https://lance.dev


Have you looked at crev (see my other comment)?


I have. I actually worked a few desks down from dpc when he was creating it and we talked about it at length. I felt then and now that it has good goals but a very limiting implementation in that it does not pursue a portable spec and instead anchors a very opinionated format to git, and github, instead of cryptographic keys held in hardware controlled by reviewers. I want to see the same keys that sign git commits also sign reviews, for instance.

I think for broad adoption a review system should ask essentially the same questions as crev, but store them in a format like in-toto including signatures by the reviewers created with a user choice of pgp smartcards, ssh keys, or webauthn devices. These reviews would be anchored to hashes of a particular state of a particular tree of code and not to any type of VCS or distribution system. Important code is distributed via perforce, mercurial, cvs, subversion, and tar files depending if we are talking about big corps, old codebases, or linux distro building blocks. A good OSS review system should be also be usable by teams in their internal proprietary codebases too if we wish to see wide adoption. Even for OSS we may wish to share some reviews as standalone objects privately while security embargos are in place, etc. Proofs should also be verified standalone easily from local cache, when github is down, when original repos vanish, etc.

Something that meets these broader needs will make it easy for large orgs with very different internal setups to participate and play nice with other supply chain security tooling efforts by the OpenSSF using in-toto for Sigstore, Witness, etc.

My experience and visibility into internal review tooling of multiple companies tells me many need something very different than crev, but crev proved to me many people have real interest in this problem which I really thank dpc for.

The biggest blocker for starting this project is the human review spec settling in in-toto https://github.com/in-toto/attestation/issues/77

Some relevant discussion here too: https://gist.github.com/lrvick/d4b87c600cc074dfcd00a01ee6275...


Thats a great criticism of crev that definitely makes sense.

I guess you may have also read about `cargo vet`? I feel like that is a worse model than crev, but is likely to win over it unfortunately.

https://lwn.net/Articles/897435/


Like crev I feel cargo-vet is solving the right problem, and I am glad it exists as a stopgap today to encourage more review, but both designs lack portability and cryptographic signing.

They fail to address a major problem in supply chain security which is that malicious actors often get write access to repos, or their mirrors, allowing for fake commits or reviews.

Cargo itself, like most language package managers, shows complete disregard for supply chain security from the unsigned curl/bash recommended installation process. When I have needed trusted reproducible builds of cargo in my supply chains, I have had no choice but to build it myself, only to find you need a recent version of cargo to build cargo. It is a circular nightmare.

Debian at least does mostly best effort reproducible builds and signs them, so it seems like the least bad option today for security critical rust projects to build with. Linux/BSD package managers that rely on web of trust signing and reproducible builds often feel like the only responsible adults in the whole industry wide software supply chain story.

I have talked to multiple package manager teams and there is almost universal hate of the very concept of author/distributor/reviewer cryptographic signing citing that it is so hard for participants to learn that it even being offered as optional would put people off.

IMO if someone lacks will to tap a blinking Yubikey, TouchID, or similar to sign their security reviews and commits, they likely should not be doing security reviews or distributing security critical tooling.

My hope is to make signed review tooling easy enough that people will run out of excuses not to use or emulate it.


Thats incredible what you say about packaging people hating signing, I come from the Linux distro space (Debian) where support for it is almost universal. I OpenPGP sign all my git pushes, commits, tags, tarballs and uploads to Debian.

BTW, you might want to look into the Bootstrappable Builds project. They are working on starting from ~512 bytes of machine code plus a ton of source all the way up to a full distro. Started by Guix folks, hopefully it will eventually trickle down to more mainstream distros. Apparently rustc is bootstrappable via mrustc, which only requires a C/C++ toolchain. There is some discussion of that in the comments on an LWN article about the Rust/GCC projects.

https://bootstrappable.org/ https://lwn.net/Articles/841924/ https://lwn.net/Articles/907405/


That's what TideLift's goals are too. https://tidelift.com/


The comment on review overlap reminds me of crev, a socially scalable review system:

https://github.com/crev-dev/


Hey, it sounds like you'd like my essay on this: The Asymmetry of Open Source: https://matt.life/writing/the-asymmetry-of-open-source


I agree that the responsibility for quality and security of open-source lies with the consumers of that code. But legal liability for negligence is another thing altogether. If the ultimate customer or user wants their vendor to accept liability for failures or defects then they need to negotiate that up front in a binding contract. If customers don't want to pay for that then that's their problem, they can take whatever garbage their vendors shovel out and then deal with the consequences.


Even shorter:

An entity gets to demand exactly as much from me as we agreed it could demand.

My giving it more does not mean it can demand more.

Therefore, without an agreement, nobody is allowed to demand anything of me.

(Laws are another matter, of course, unless you bring the social contract into it.)


> Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something focused on your needs on stop of the standard library yourself.

Why stop at the standard library? What about the rest of the language toolchain? Compilers, interpreters. What about the OS?


Those much more often have full time paid people whose actual job is to review them. E.g the standard library of Go has security and quality review by Google. It is reasonable to assume they are investing more in supply chain security than you are in those portions.

The random dep you grabbed that some hobbyist hammered out in a weekend... may not have 2FA on github, or maybe their email domain is about to expire, and it is very likely no one reviewed it.

If you choose to use the latter because it does what you need, then you are now the first party accepting the additional responsibilities of reviewing and maintaining that code to the level of safety appropriate for your use case.

If you are making a video game for personal use, then maybe the worst case scenario is tolerable for you to not care. If you are handling PII or money of other people... then suddenly you /must/ care.


He who cashes the check shall be the defendant in the lawsuit. If things went well the people behind the compiler, interpreter and OS wouldn't have received a dime from your successful enterprise, now that it's gone sideways you want to share?


The idea is risk reduction, not removal.

An OS is backed by thousands of developers, and usually billions of dollars. Theres negligible risk that the momentum stops.

If that OSS library, that is fundamental to your business, has three developers, then maybe you do have a problem.


I think that a lot of people should be asking themselves about the state of vulns and other bugs in linux. In fact, I'd call the state of correctness of the linux kernel and surrounding systems to be something resembling an industry emergency. It shouldn't be possible for bugs to regress in the kernel, but they do! A lot of very ordinary best practices aren't actually followed in the kernel world.


The linux kernel is a security shit show, but it is the only compatible option in most cases. You can at least strip it to the bone, apply Kernel Self Protection Project guidelines, and run it offline in TEEs or HSMs for your most sensitive operations. This is approaching as good as we will ever get.

Linux is a massive C codebase one can never /fully/ trust or harden and high security applications with sufficient resources should favor formally verified microkernels like SeL4 whenever possible.


It is true that Linux rules the world and is absolutely the right platform for a huge number of projects. I'm not saying "Linux is bad, reach for something else." I'm saying "Linux is the foundation for basically the entire computing landscape so the industry needs to take a real hard look at finding a way to make it safe."

I think it can be done. But I think it needs industry-wide effort in a huge way.


I think it is going to be much easier to produce a formally verified Linux ABI compatible microkernel than to ever fix Linux itself at this point.


> on top of the standard library

The standard library should also not be trusted by default, but it doesn't change as frequently which I imagine was what you were thinking.


>Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something in-house focused on your needs on top of the standard library.

That's not the comparison, the comparison is review an open source program vs. implement + review an in-house program.


When I read these kinds of sentiments about OSS I always think of Windows desktops and Internet Explorer in places where they absolutely don't belong. Yes, it is bad to have an enormously deep and complex stack with a big fat attack surface, almost none of which is relevant to the application. That badness isn't mitigated by the stack being commercial and coming from a single vendor.


> we must stop expecting devs to have any responsibility for code they produce. The responsibility is on those that consume it.

Why talk about the future ?

Microsoft, Google and other SW makers do not take responsibility for the code they produce. "Oops, we have an RCE. We're sorry. Please update "


> If you make no reasonable attempt at vetting your supply chain and harm comes to users as a result, then IMO you should be liable for negligence just like a restaurant serving food with poisonous ingredients.

You trying to kill FOSS for end users here?

> If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies.

Occasional security issues can just be the cost of doing business. And that's OK as long as your company isn't doing something critical like running the banking system. Needing to restore your shopping site from backup isn't the end of the world.

There's also the separate and even rarer problem of possible malicious non-internet dependant code in dependencies which would be relevant if running an xray machine or a nuclear power plant (e.g. things disconnected from the internet) but Ive never heard of a malicious non-internet dependant (so not something that tries to connect to a remote server) code in a dependency.


FOSS for end users is free and their own personal responsibility so I never see an issue here. When a business runs the code for others, then responsibility and liability kicks in.

Per my other comments, fintech we can all agree on, but if your organization manages PII then it has an obligation to make a best effort to ensure randos from the internet cannot execute any code they want against the databases.

If your organization has no PII and no financial exchange then I would agree the risk and need for review is much lower, but if those things are true then it is probably a nonprofit or a hobby.


> on top of the standard library

What makes the standard library so special? Or do you advocate fully developing that in house as well?


See my other comments on this. I tend to have people focus their cycles on easy targets for supply chain attacks such as libraries where there is no evidence -anyone- is doing review.

Most programming language standard libraries generally have at least some first and third party reviews from large organizations like Google, etc. That may not be perfect, but it makes those a much more difficult target than phishing some student programmers github account.


How did you get into this type of consulting?


My software engineering and sysadmin background overlapped a lot with security, and I found flaws and recommended security solutions at every company... so I pivoted my career to full time security engineering for multiple companies. I eventually saw enough massive oversights and targeted attacks to operate as though any person or system that lacks strong accountability for their every action is compromised.

Most companies can not afford to meet this kind of zeroish-trust threat model, so I moved to roles in fintech companies where they -must- think this way as they are highly targeted.

I realized though the highest value I provided to employers in addressing security problems happens in the first few weeks, then spot check pentests and advice a few hours a month after that while I build tooling. I also got tired of writing internal-only security tooling and practices I knew so many companies really need. I concluded the only assured way for me to be able to open source my tools and practices and get them refined by collaboration and exposure with a lot more organizations was to start a company with that mission.

I founded Distrust and my full time employer at the time graciously agreed to be my first part time retainer client. Turns out, many companies want pentesting and full stack security consulting retainers where someone can integrate into their team and help architect practical solutions unique to each company. Even companies with their own security teams benefit from a part timer with a perspective on security problems and solutions at many other companies with similar threat models.

Word of mouth has kept my schedule full enough to raise rates and justify building a small team last year. Best career choice I ever made.


The author of an open source project usually makes NO guarantees and assumes no liability. They've tossed a blueprint into the world.

When you decided to use it, you took on all of its technical debt and design decisions as your own. You can't fault the author if you use it in an environment they didn't have, or didn't intend it to be used in, or in a manner that wasn't intended. They left an artifact, they did NOT make an agreement to support it.

If someone makes a mistake in a book, there's no obligation for the author to correct it. If someone misunderstands the ideas and goes on to do some horrible deeds using it as justification, it's not the author's fault.

Open source software is, for the most part, an unsupported gift economy. It isn't commercial, and it certainly isn't a chain... there's no delivery to be had... something was tossed over the wall, and it may never happen again.

We Tims have been waiting for the next episode of Hello Internet for a while now... it may never come. That is also the nature of open source software.


Funny thing, most commercial software also comes with T&Cs that say the make no guarantees and assume no liability. e.g., Microsoft Windows 10 retail license:

> Neither Microsoft, nor the device manufacturer or installer, gives any other express warranties, guarantees, or conditions.

And their liability is limited to...

> at its election, either: (i) repair or replace the software at no charge, or (ii) accept return of the software (or at its election the device on which the software was preinstalled) for a refund of the amount paid, if any.

So, maybe they'll fix, or maybe they'll refund you your money and you discontinue your use of the software. Maybe they'll issue a fix that you incur the cost of installing and testing.

[1] https://www.microsoft.com/en-us/Useterms/Retail/Windows/10/U...


I can be on board with that. If you don't like the software I provided for free, then I'll refund every dime that you paid me.


> You can't fault the author if you use it in an environment they didn't have, or didn't intend it to be used in, or in a manner that wasn't intended.

It goes even farther than that: you can’t fault the author even if you did use it in the intended manner.


Practically speaking, I'm going to assert that there is no team of developers, let alone single developers, who are capable of taking on both the burden of their own software and the software that they import. I simply can not imagine even a whole fortune 500 company supporting their own business logic written on top of Spring, which includes log4j and a whole host of other open source dependencies.

And what's the commercial alternative? I'm personally not aware of one.

Also, the liability thing is a grey area - it works for the US (mostly, Tornado cash is currently testing these waters) - since there are open issues with developers of "questionable" software traveling to some countries. Heck, even the US has pulled some OSS developers aside at customs because of their work on OSS projects.


> I simply can not imagine even a whole fortune 500 company supporting their own business logic written on top of Spring, which includes log4j and a whole host of other open source dependencies.

> And what's the commercial alternative? I'm personally not aware of one.

There's commercial support for Spring:

https://spring.io/support https://tanzu.vmware.com/support/oss


Wouldn't you have to start with the JVM? Or possibly with the Linux kernel?


Not necessarily. If you _purchase_ software that includes the guarantees that you desire. (e.g. of correctness, fitness for purpose, and support), then you don’t have to maintain the entire stack yourself.


Even if you purchase the software from a commercial vendor that doesn't mean that the vendor will indemnify you for damages caused by their failures, or that they have sufficient financial resources to pay for such damages (counterparty risk).


At that point, just buy insurance


What about the hardware? Who will take on liability for Row Hammer[1] vulnerabilities? What about Meltdown [2] or Spectre?

[1] https://en.wikipedia.org/wiki/Row_hammer

[2] https://en.wikipedia.org/wiki/Meltdown_(security_vulnerabili...


Rowhammer when it first came out, or rowhammer today? I'd say any memory made in the last five years that doesn't properly implement Target Row Refresh should put liability on the manufacturer.


RIP Hello Internet. Cortex is great, but it just isn't the same. Grey always said he would probably take the Irish Exit on that show, and that's exactly what he ended up doing


We the Tims accept this fact and stand waiting.


Software repositories have every right to set terms and conditions for their use.

Nothing about tossing a blueprint into the world entitles you to have blueprint hosting for free and without requirements.

Folks who don't like that there are terms and conditions on hosting software in repositories are free to host it themselves.


I don't really understand this argument: if GitHub allows hosting things that completely disclaim warranty and people are using things from GitHub, don't you effectively have this "tossing a blueprint into the world" problem? Sure, you aren't entitled to it, but there are plenty of places that offer this kind of service and plenty of consumers who get software from these services.


I was talking about the complaints that Github allows PRs.


I could easily self host my projects on a rpi and rate limit all the companies that download the same thing over and over every second. Companies can and should set up internal mirrors.

The advantage of github is that people have accounts and it's easy for them to send patches or discover projects.

But I'm fully capable of using git via email.


Is there a software repository that requires the developers to assume the liabilities?


The hosting costs are almost zero, and plenty of people are willing to do it for free. How is this supposed to change anything?


That's nice in principle, but definitely not how it operates in practice.


This is a historically narrow view of the situation.

We are all living in a post-war era. That war was over whether and how much of the software in the world should be locked up behind proprietary licenses and paywalls. To win the war, Open Source had to produce software that was less awful than the proprietary software it was trying to unseat. It did. Companies went out of business or adapted.

Now everyone is trying to pretend that the choice all along was between someone else's software and making your own. That wasn't the choice, and it's revisionist history to claim otherwise. A war was one, territory was occupied, and the 'treaty' at the end of that war was "make stuff that isn't awful", which is being peeled away layer by layer.

The above is the opening to my thesis as to why SaaS is now trying to eat Open Source: FOSS is abdicating responsibility for quality, and so the only way to get quality is to harass FOSS authors who think we are ungrateful whiners, or pay for improvements on the backs of paying customers who just want software that works, not to be told to put up or shut up.

The only sane way to defend against the re-privatization of FOSS software is to stop treating your users like they need to grow up and realize that you're deflecting. Unless authors accept that you have an ethical responsibility that comes with occupying a corner of the collective consciousness, it's going to be walled gardens everywhere you look.

We aren't paying people to 'steal' your code. We're paying people to act like professionals.


You can also get quality by... doing it yourself. That's the Free Software core idea. Don't like what exists? You're free to change it. In fact, you're encouraged to change it, and publish your changes, so others can use your higher-quality version.

It's supposed to be a gift economy, not a plunder economy. You're given something for free, with little to no strings attached. All that is politely expected of you is to A) not be a dick, and B) reciprocate in kind, if the situation arises.


Or you can get "quality" (or support) by paying the author (or some other developer) to support / improve said software within the legal terms of whatever license that software is released under.


Which we used to be able to do a lot more. Many of those companies are out of business or merged with a mega corporation.


> and the 'treaty' at the end of that war was "make stuff that isn't awful", which is being peeled away layer by layer.

There never was a treaty (not even an informal one!).

In businesses, the reason why open source software was adopted was typically because it enabled cost savings (e.g. saving/reducing license costs).


> This is a historically narrow view of the situation.

> We aren't paying people to 'steal' your code. We're paying people to act like professionals.

I think you have a narrow view of the situation, or you are trying to misrepresent the reality we are living in.

We are signing up for OSS-derived SaaS mostly because of the existence of cloud-computing network effects. There have been many fights fought by companies 'acting like professionals' that could accept payment. They are re-licensing their software because developers and companies are utilizing a forked cloud-vendor-specific clones of their work. Forking software to provide it to a captive audience doesn't seem professional, it seems anti-competitive.

That said, most popular licenses allow it, so it is legal, but I don't think it is ethical.


> FOSS is abdicating responsibility for quality, and so the only way to get quality is to harass FOSS authors who think we are ungrateful whiners, or pay for improvements on the backs of paying customers who just want software that works, not to be told to put up or shut up.

I think you are assigning things broadly to "FOSS authors" that were explicitly disclaimed by most such authors from before companies started using FOSS.


Then we’re going to keep seeing an erosion of Free software until there’s a renaissance where people begin actively courting the general developer public to vote with their feet again.

FOSS won people over by saying we are entitled to useful software without having to mortgage ourselves to corporations. Well, it got what it wanted and now it calls users entitled. You’re damn right they are.


> FOSS won people over by saying we are entitled to useful software without having to mortgage ourselves to corporations. Well, it got what it wanted and now it calls users entitled. You’re damn right they are.

It's not the users that are acting too entitled. It's software producers that are doing it, by failing to perform their essential role of making sure the software they use is adequate for their purposes, and demanding that the FOSS components they're using do their their work for them.


We didn't get it… what we got are gigantic corporations using free software to make proprietary software. Nobody wanted that.

Can I download the sources and easily patch the gmail client on my phone as easily as I do that on debian? No.

q.e.d.


Yes, because it’s been twenty years of FOSS with zero DevEx. People ran back to corporate walled gardens. Which is just a rephrasing of what I said above.

This isn’t “users” and corporations making a mess of the software world together. It’s all of us. As you guys are so fond of telling everyone else, if you don’t like it, change.


You're revising history here.

There was no war. There was only "I'm not using your code unless I can see what it does and do whatever I want with it." It wasn't a statement of conflict, it was one of choosing ones own way without compromise. Then some people said "I like that idea, here, I made this, do what you want with it, but it's not mine, it's yours, problems and all."

That's all there is to any of this. There is no ethical responsibility. People doing something for fun for free dont owe anyone any degree of professionalism. If you want to burden yourself with the expectations of others that is your right, but just because you do it doesn't make it some standard everyone has to live up to.


> There is no ethical responsibility.

Then stop calling yourself a software engineer. Engineering has an ethical responsibility associated with all of their work.

Go be a software developer with your no responsibility. Work on Dark Patterns. Work on spying on your customers. Work on making “Developer” an epithet so the rest of us can use you as an example to push for actual Engineering status and the protections that go with it.

Why do we expect more grace and generosity from the hosts of social gatherings than we do with software that will touch the lives of millions of users? That’s madness.

“Oh you don’t like that there are no snacks? Well then bring your own.” would get you blackballed in any other setting. It’s antisocial, which kind of goes with our stereotype. Most of us don’t like it, some of us don’t see how we feed into it.


You dropped the crux of what i said: "People who do something for fun for free..." It's much harder to paint my position as one of antisocial uncaring evil in light of that.


Jeepers, you seem to be making up some argument against a bunch of stuff betwixtthewires didn’t say. Where did they call themselves an engineer? Strawman? They talk about developing as a hobby, not killing civilisation, nor working for Metagoobooksoftle.


The metaphor “war” is invalid methinks.

Ecological systems or economic competition are far better metaphors, because they are emergent and decentralised, rather than patriotic and authoritarian. “Nature red in tooth and claw”, or monopoly rent and marginal price → 0.

Revisionist? The 1T$ Microsoft has not “lost the war”. The biggest FOSS wins have serious corporate sponsorship (OSes, databases, browsers, dev tools, languages, etcetera).


I think it’s lost on the kids how many people came to open source because they hated Microsoft. That went on for a dozen years at least. That’s the same time period when Eric Raymond and Linus Torvalds and Richard Stallman and a dozen others would have public debates on which flavor OSS should have. Those people all were visionaries, and wrote like it. We had long talks about gift economies too, which even showed up in sci-fi (Kim Stanley Robinson, for one).

Linus is the most practical of the three, which is probably why we still talk about him more than the other two. But we wouldn’t talk about any of them if “practical” was the only adjective we had for them.

If you want to celebrate Microsoft becoming open source as an ally, knock yourself out. But the older people still don’t trust Microsoft (see also the Scorpion and the Frog), so it’s more of a war trophy. This is probably as “beaten” as MS gets, but they can always wake up.


I am suitably middle-aged and FOSS supporting to find your insinuations offensive. I didn’t say anything about supporting Microsoft, just stating some facts that are hard to argue against (and I note that you don’t argue against them). I think that jumping to conclusions about others is nasty behaviour.


This isn't entirely true because many people don't just write software, they also promote it. For example, they create websites encouraging adoption, much in the same way companies do, with instructions on how to install it easily, implying that you should install it. And maybe they'll give talks and make videos?

But it gets worse. Often the people promoting the software aren't the people who wrote it, and they may make claims that the authors never intended.

None of these things mean that the authors accept formal liability, but morally speaking, if you're encouraging people to use your software and you're participating in an ecosystem promoting its usage, and it turns out to have bad security flaws, then maybe you shouldn't be so encouraging? Be upfront that it's just something you threw together. Add a few speedbumps. Maybe don't publish it on npm? Disrupt the hype train.

Disclaiming all responsibility should imply that you also don't hype it up and set people up for failure.


It's kind of amazing that 95% of gripes about open source projects and who is supposed to support them to what extent can be solved by open source authors being upfront about those expectations.

If you're releasing something and have no intention of supporting it, simply put "NOT INTENTED FOR USE IN PRODUCTION" at the very top of the readme file. It's that easy, imho.


Wouldn't a LICENSE file already do this for the most part? MIT: "THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED ..." and so forth.


This doesn't tell the reader anything because solid open source projects that can be used in production and those that are just experiments will often have the same LICENSE. All open source licenses disclaim liability.

So if you really want to discourage usage, you should say something in the README.


Or, abide the license. Asses for yourself if it can be used in production in your business. Be prepared to be wrong about that. If you have a hard time discerning, abide the license, don't use code in production that disclaims liability, ever. If it disclaims liability the onus is on you.


> Asses for yourself if it can be used in production in your business.

So waste everyone's time instead of just telling them?

> If you have a hard time discerning, abide the license, don't use code in production that disclaims liability, ever.

That's basically all code so that's not useful advice.


Since when it is a waste of time to carefully figure out if you should base your important business on someone else's work? You know what's a waste of time? Telling people something you already told them in your license. If we did as you ask, someone would drop a project expecting not to when they started, or someone would maintain one with your warning, and people would say "mean it! This guy duplicated your code expecting you not to maintain it, and you did, so you wasted his time! Put a third warning in!" Adding a warning that's already in the license does not convey useful information, it does not help anyone discern anything.

> not useful advice.

It's very useful advice. If you can't discern whether something FOSS will be useful to your business or not then don't base your business on FOSS code. Simple.


>You know what's a waste of time? Telling people something you already told them in your license.

Uh, the whole point of the comment was that license doesn't say how serious the project is.

It's not about boilerplate disclaimers...

> It's very useful advice. If you can't discern whether something FOSS will be useful to your business or not then don't base your business on FOSS code. Simple.

"don't use code in production that disclaims liability" is useless. All code disclaims liability. The "if you can't discern if it's useful" doesn't convert such a pointless phrase into good advice.

It's like saying "if you can't determine which foods are healthy, don't eat food that comes in amounts". If someone can determine, the advice doesn't apply, and if they can't determine... you just told them to starve? That advice helps nobody.


Food, that analogy doesn't apply because nobody is dying here.

And it is about boilerplate disclaimers. That's my entire point. "Add a boilerplate disclaimer reiterating what your license says if you really really mean it" is what your argument boils down to. How do you make it not boilerplate? Require then to write a 200 page report in their own words? If the license disclaimer means nothing, no number of disclaimers will mean anything more. If the advice helps nobody so will an additional disclaimer.


> Food, that analogy doesn't apply because nobody is dying here.

A typical business is going to die if it can't get software.

> And it is about boilerplate disclaimers. That's my entire point. "Add a boilerplate disclaimer reiterating what your license says if you really really mean it" is what your argument boils down to.

No, please read what people are saying.

The request is for a basic description of how serious the project is. Whether it's a super slapdash proof of concept, or a basic implementation with half the features stubbed out, or a mostly complete but unstable project that, or something battle-tested and heavily documented and maintained full-time, or other places in between or outside those lines. Whether they even want bug reports; whether they even want patch submissions.

All of those very different projects could have the exact same license. The exact same disclaimers.

The request is for information that does not belong in a license, and is not put into a license. Nobody is asking for a restatement or rewrite of the license.


A business that would die without software will write it's own software. A business that would die without getting that software for free is not a viable business.

I can see putting some message of intent as a developer, but people change their minds. Lots of the problems we see today that we are discussing involve people who started out with an intent to maintain full time but then for burnt out or otherwise lost interest. A message of intent like you say does nothing for a business already dependent on their work. This scenario is the more common problem with these dependency chains and what I thought we were discussing, most slapdash proof of concept projects say so in a README. If you can't discern whether you can depend on something, then you need to be prepared to take over the project or live without it.


> A business that would die without software will write it's own software. A business that would die without getting that software for free is not a viable business.

A business will not write its own OS, and commercial OSes have the same disclaimers. You didn't say "if you have a hard time discerning, don't use FOSS code that disclaims liability". You said "if you have a hard time discerning, don't use code that disclaims liability". (slightly paraphrased)

> I can see putting some message of intent as a developer, but people change their minds.

And then you can update the message. All the more reason not to tie it to license files.

> most slapdash proof of concept projects say so in a README

Sometimes. Less often can you tell the difference between a mildly serious project and a very serious project at a glance.

> A message of intent like you say does nothing for a business already dependent on their work.

> If you can't discern whether you can depend on something, then you need to be prepared to take over the project or live without it.

Anything could change, but knowing the intent gives you a much better starting position. And it's better to put it somewhere it can be seen in under a minute rather than after a chunk of time in investigation, especially because there might be a whole pile of dependencies to examine.



This sounds like most of web scraping and "integrations" generally, by commercial companies, largely black or grayboxed: they just don't read the README, because they're busy scraping.


> There is no formal agreement between a maintainer and its downstream users.

But there usually is, that is what software licenses are for, with typical statements like

> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT


Maybe the annoyance comes from releasing software with no license.md or a license that says "DO WHATEVER THE F--- YOU WANT WITH THIS S---" then getting a bunch of corporates saying it's not properly licensed while a bunch of "free as in freedom" people are complaining that the license isn't free as in freedom enough.


The WTFPL is FSF-approved enough for me:

http://www.wtfpl.net/faq/


Big corps can't use WTFPL-licensed software, at least the one I work at.


(In which case, I propose a new slogan, "don't tread on my crap.")


"I dare you to use this."


> A package repository might deem your software “critical”, adding requirements to publishing updates that you might not want to or be able to comply with.

I discovered last week that Github no longer allows you to push changes via https, and thus I have not yet published a minor fix for an insignificant piece of open source software I have been lackadaisically maintaining for the last eight years. Perhaps some day I will get around to jumping through their new security hoops... or perhaps I won't. In this case nobody is likely to care, but it makes me think these kinds of organizations ought to be extremely cautious about introducing extra friction to the workflows of people who are giving away their time for free.


Not sure I understand how having an ssh key is "jumping through hoops" though...? Presumably you already have one set up (even if you don't, it's literally just a few seconds of work to create a new one), so just add the public key to your account's SSH keys list, and done. Update your remote urls from https to the git@github.com:yourusername format and push whatever you want to push.

Or, heck, if no build is required: why use git at all, just use github itself. You can edit, create PRs, on new branches, all without ever needing your own desktop. Perfect for small code changes (especially typo fixes).


Pushing via HTTPS is the only way I use GitHub, so I find this claim questionable.

It's working fine for me.


Huh! I wonder why your experience is different. I get this error:

  remote: Support for password authentication was removed on August 13, 2021.
  remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.


You need to read the url and follow the instructions on creating a personal access token (PAT) and then you can push via HTTPS again.

You can't push via HTTPS Authentication user/password anymore.


Yep! That's the situation, all right. The system I used for many years doesn't work anymore; to keep updating that repo I would have to jump through some hoops dealing with the new security apparatus, which was created for Github's benefit not mine. Maybe some day I'll get around to it, or perhaps I'll never bother. We'll see!

The point here is not that my personal experience matters, but to illustrate an idea from the linked article: every time you introduce some new point of friction to an open-source distribution process, someone is going to wonder "do I care enough to deal with this"?, and sometimes the answer will be "no".


I don’t understand this attitude at all, it’s significantly more secure than password auth, takes less than 5 minutes to correct, and you only need to do it once. It takes approximately as long as you typing out your comment.

The entire software industry has occasional change, and I have a hard time believing that 5 minutes to update your account to use a more secure system is that much of a burden (they even provide you a link to the docs!). You seem to have some other reason for really taking a stand about this.


You don't have to understand my attitude, but you might want to be aware that this is a real phenomenon, occurring regularly in the world around you, and take it into account if you happen to be thinking about so-called "software supply chain" issues.

I'm not actually taking a stand about anything; github can do whatever they please. I'm just sharing a funny story from my recent life which illustrates a point from this article: there is no software supply chain, there is only an army of volunteers bearing gifts. If you try to demand that these volunteers raise their standard of gift-wrapping, no matter how small and reasonable a request that may seem to you, some of 'em just won't show up anymore.

Of course I have spent more time here talking about the problem than it would have taken to fix. So what? This conversation is fun! I am happy to spend my free time this way. When I choose to spend some of my precious free time on grungy, irritating chores, however, there's a long list I would rather deal with before this nuisance from github would come up: washing the dishes, taking out the trash, running the laundry, sweeping the floor, scrubbing out the toilets... those all being activities which improve my environment in ways I actually care about.


But it's not more secure than password auth. My passwords are not guessable, by anyone. Period. You are assuming passwords are guessable; they are not. They are not reused. They are not crackable. It's utter bullshit that an SSH key pair is more secure than passwords done right.

"But what about a keylogger / what if your password manager is compromised / etc." if someone has root access to my machine and can read my encrypted documents or log my keystrokes, the game is up for an SSH private key as well. There's no reason to say my SSH private key sitting on my hard drive is more secure than my GitHub-specific password sitting (encrypted) on my hard drive.

SSH key pairs are more secure on average because passwords are used by everybody and SSH key pairs are used by security nerds. That's it. That's all there is.


The post's definition of 'supply chain' is wrong, and it leads to confusion. If I forage for mushrooms and give them to a local chef, I'm part of the supply chain for that night's dinner, even without money changing hands or quality guarantees.

That said, to opt out of companies' supply chains, stop publishing software with permissive licenses.


I'm certainly not going to stop publishing software with permissive licenses. I'll just adhere to the disclaimer I include with that software: I provide no warranty that it is fit for any particular use. Use it as you will, but don't expect any guarantee that I'll support whatever needs your supply chain has.


That's the current state that the author considers untenable, right?


My read of TFA is that author is complaining that companies are trying to push responsibilities onto FOSS authors that don't belong with the FOSS authors.

He includes complaints about Github there, but those are easily resolved by not using Github.

I believe that the author and I are on the same page here, generally. Have I misunderstood his essay?


Or license your work under GPL, which is kryptonite to most businesses. The mega-globo-corp I work for makes commercial license deals with some open source software vendors who dual license their software with GPL and a commercial license.


> which is kryptonite to most businesses

Absolutely not. There's plenty of software under GPL used everywhere, including the Linux kernel, GCC and tons more.


Right. GPL is an example of a non-permissive license.


The author is just flat wrong.

Unless one thinks there are no supply chains in Gene Roddenberry's Star Trek, which is obviously wrong.

Money isn't what makes it a supply chain. Pull dependencies on provenance (including origination, transformations, even aggregation) make it a supply chain.


If you want or need support or maintenance for my project in your business, offer me cash. If your first thought is, “hah, as if finance would ever green light that…” then I encourage you to fork it and tell your manager it will cost your company two to three times as much to do it yourself.

I think both are perfectly valid options. Just make informed decisions and in doing so, demonstrate business competence.


I sell both my software as OSS, and as commercial software with a delivery through a corporate appstore. Companies are happy to pay.


The word these thought leaders are looking for is "provenance", which is not marketing-friendly and they are too scared that this will have people reaching for the dictionary or worse, ignoring them, so they resort to existing terms.


Provenance is fine. People use it, particularly in narrow settings around the literal provenance of software assets. I have been in this space for a while and I have come across zero arguments that "provenance" should be ditched due to dictionary obstacles.


You won't have come across those arguments. They are had by marketing types.


Why would marketing even care about any of that?


The author is taking the term too literally.

Try explaining software packaging and distribution to a non-technical person.

Using analogies like "software supply chain" helps people better frame a problem/concept in their heads.


Not just "too literally" but outright incorrectly.

"A supply chain is a network of individuals and companies who are involved in creating a product and delivering it to the consumer. Links on the chain begin with the producers of the raw materials and end when the van delivers the finished product to the end user. "

https://www.investopedia.com/terms/s/supplychain.asp

The exchange of money, guarantees, quality assurance... None of it is required.


Further, the apparently requirement for "money to change hands" falls flat on it's face.

The last node in the supply chain for virtually any real-world product is "human beings extract the material from the earth in some way/shape/form". No human being paid "the earth" money for this, yet it is a fundamental component of supply chains (and, recently, a fundamental component of many supply chain issues). "The earth" also doesn't offer any guarantees or quality assurance.

Attempting to interpret these processes as requirements to be part of a "supply chain" excludes the core foundation of traditional supply chains.


It's investopedia. The exchange of money is taken as read. You might as well say that a vendor of exotic fish not mentioning water is proof that the fish don't need it.


I am used to investopedia. The exchange of money is not "taken as read"


I think the author is rather raising the issue that this analogy gives people false expectations since they take it too literally. I don't think this post would have been made (and certainly not frontpaged) if this was just semantic pedantry.


I mostly agree with the article, but the title is downright wrong.

Even a relatively literal read of the words "supply chain" is reasonably appropriate, even for FOSS. Wikipedia's current definition for supply chain is:

> In commerce, a supply chain refers to the network of organizations, people, activities, information, and resources involved in delivering a product or service to a consumer.

There's nothing in that definition that necessitates a formal agreement, support commitment, etc.


Agreed. There are some interesting points made in the last third of the article, but the rigid interpretation in the beginning doesn't add anything to them.

If the author scraped the title and just focused on how they are large companies attempting to force their will on open source maintainers to protect their supply chain, they'd have a stronger piece.


If you want a software supply chain that includes formal agreements, use Java or C#, or maybe C++. Those ecosystems have plenty of vendors who will sell you libraries, including phone support and liability.

So far enterprises are the only ones willing to pay for their "software supply chain", if you want this kind of "solution" just use what they use.


The sensitive nature of our business forced the equation for us. We are basically a pure Microsoft shop because it makes survival of audits feasible. 100% of our B2B clients are already doing business with them, so the conversation is fairly simple.


> You still cannot disable pull requests on a GitHub repository. A package repository might deem your software “critical”, adding requirements to publishing updates that you might not want to or be able to comply with. Google even wanted to disallow anonymous individuals from maintaining critical software and wanted to police the identities of others.

That's just whining from people who host their stuff on other people's servers and sites.

If other people package your software into a bigger system and then later improve their standards so that the bar is above your stuff now (e.g. you break it left and right with careless, untested changes, or whatever, whereas downstream is doing actual QA) that's downstream's prerogative. You don't have to do anything; they can fork your stuff and develop it themselves, or stick with the old version from you that worked, ignoring the newer releases.


The problem is that they don't fork it. Instead they harass you.


People should give that a good try; maintaining forks sucks.

In the past I've forked at the sign of any resistance.

I have a fork of GNU Awk called Enhanced GNU Awk (egawk).

https://www.kylheku.com/cgit/egawk/about/

It has a let construct for lexical variables.

Chances are it's not packaged for your favorite OS, so you'd have to build it yourself.

The next thing on the plate for that project is to support the GNU C preprocessor line annotations so that if Egawk is handed preprocessed code, it will report errors against the original file and line number.


I do feel its indicative that most of the authors issues stem from GitHub, the facebook of foss, big social media for coders. If you don't want to engage with the social trends and aspects of the ecosystem (which is perfectly understandable) then GH seems particularly poor choice of platform to publish your software on. I feel the author would have far better chance of "to be left the hell alone" if they chose literally any other platform.


Not the author, but I wouldn't say it's just (or even primarily) Github's pull requests or "critical security advisory" that's the problem here. PyPI requiring 2FA for maintainers of popular software has had more real-world impact (one maintainer took down and recreated their project, erasing old releases), and Google calling for deanonymizing (doxxing) maintainers of open-source software is more terrifying.

I'd argue that the problem isn't that "software supply chains do not exist", but "you using a program or library without pay does not and should not mean the software's author is now responsible for fulfilling your use cases and paperwork requirements".


iliana is an active Mastodon/Activity Pub user, so surprisingly apt metaphor and probably a fair assessment.


The could be a supply chain though.

I had a running joke at a place I used to work that I should just quit and start my own US based company that "sold" open-source software so that it'd have a US-flagged company behind it. IMO easy way to branch out to commercial instead of just government contracts would be to audit the source code. Probably save a few companies from a left-pad incident (or worse) since they'd have to pull from my servers (or cache mine) and I'm smart enough to recognize that updating leftpad to be nothing would not be helpful to any client


> start my own US based company that "sold" open-source software so that it'd have a US-flagged company

That is the business model for Redhat, Ubuntu, SUSE, and other GNU/Linux companies. You can download it for free or pay a subscription to have emails and phonecalls answered.


You can download Red Hat for free, but you won't be able to use it without registering. Rocky Linux has done a good job of making a free Red Hat though. Been using it for my RHCSA recently


Developer RHEL doesn’t cost any money. RHEL is based on Centos Stream and Stream comes from Fedora.


The difference is really I wouldn't be developing any of the software. I'm literally selling things you can get off of say GitHub.

Although I'd probably need to not be completely blunt about it. It'd just be mostly an open secret where any of the dev teams would know that's exactly what I'm doing (and any sort of LICENSE file would make it dead obvious) but to procurement and whatnot it just looks like you're funneling money to me in exchange for some software.


Frankly I'm surprised a company like this hasn't been created yet.


Anaconda is pretty much selling a redistribution of popular foss python packages.


In my case, I basically never use anyone else's code, unless there's a real need for it, and if I'm quite confident in its veracity.

I also take full Responsibility and Accountability for that code, when it is running in my execution thread.

The result of this policy, is that I have written almost every single dependency that I use.

It has garnered me a lot of sneers from the "in crowd."

I don't particularly care.

WFM. YMMV.

[EDIT] Also, I like this:

    my name is always spelled lowercase and is pronounced /ɪliˈɑnə ɛˈtiːn/.

    please consider making the internet a weirder place today

    note: if you send me unsolicited email based on my github profile:

    i will screenshot it and share in my discord server
    we will make fun of you
    you will not receive a response*


I'm not trying to be flippant, but where do you draw the line of "never use anyone else's code"? You're writing that software using an operating system and editor someone else wrote. You're compiling it with software someone else wrote. If you're deploying it to users, it's hosted on someone else's code and will be run by someone else's code in that runtime.


OP claims responsibility for code when it is running in my execution thread. That seems to be the boundary. So no need to write their OS, though maybe their own interface for making syscallz.


I know. That's the classic argument that I get. Basically "what about..?"

Like I said, I know what works for me. I'm not gonna argue about it. I do know that some find my stance offensive, which I don't actually understand. I'm not attacking anyone. I'm just saying how I do it, in my own work.

I write lots of packages that I integrate into my own work. Each package is a standalone project, with tests and docs. Because they are written and maintained by a single person, I know that some folks believe this makes them "worthless," but not to me.

It's the way that I work, on my own stuff. I'm not competing with anyone here, so I fail to see the problem. If I'm just some clueless old boomer, making a hash of things, then God bless my little heart, and I'll just be tilting at windmills, over here.


So you never used any library? Not even the standard library?


Sigh... The traditional argument is that every dependency is of the same quality and trustworthiness of the language Standard Library.

If I use the SL, then I should also have no problem using some lashed-up chimera that has a dependency hierarchy that spans three continents.

Like I said, I'll do things my way.

For the record, here's the Swift Foundation Library: https://github.com/apple/swift-corelibs-foundation (I mostly write in Swift).

It has plenty of open issues: https://github.com/apple/swift-corelibs-foundation/issues?q=...

Also, for the record, here's a peek at some of the "worthless" packages that I use in my own work: https://github.com/RiftValleySoftware

If every dependency chain can match these, then I'll be open to considering them.

As it is, I do use the occasional external package, but I'm picky.


This is just nitpicking over the fact that a specific metaphor doesn’t map 1-to-1 with the target domain. No metaphor does, that’s not the point of using one.

The metaphor in this case is pretty straightforward as are its limits, and it doesn’t appear to be causing confusion with people misidentifying attributes of the metaphor that don’t map to the software domain.

(And to boot: there are actual software supply chains with a traditional supplier/vendor relationship…)


> The metaphor in this case is pretty straightforward as are its limits, and it doesn’t appear to be causing confusion with people misidentifying attributes of the metaphor that don’t map to the software domain.

One of the central points that the author makes is that this is in his opinion not true, and he provides evidence for this.


When, for example, Google open sources their client libraries, it absolutely is a supply chain issue for their customers. The people working on those projects for Google aren’t doing it as a hobby. Their customers might not be paying them directly for those client libs, but they sure as heck are being billed at some point.


> I just want to publish software that I think is neat so that other hobbyists can use and learn from it, and I otherwise want to be left the hell alone.

Me, too. Which is why I pretty much abandoned the formal "OSS" world and don't use Github and the like. I still make software, I still provide a free license for anyone to use it, and I'll even maintain it for as long as it interests me.

But it's a hobby, not a job. If someone uses my software commercially, it's on them to make sure that it's properly maintained. They have the source, they can do it.


"I just want to publish software that I think is neat so that other hobbyists can use and learn from it, and I otherwise want to be left the hell alone. I should be allowed to decide if something I wrote is “done”. The focus on securing the “software supply chain” has made it even more likely that releasing software for others to use will just mean more work for me that I don’t benefit from. I reject the idea that a concept so tenuous can be secured in the first place."

K. The rest of us are gonna keep using the phrase though.


>You still cannot disable pull requests on a GitHub repository.

>I just want to publish software that I think is neat so that other hobbyists can use and learn from it, and I otherwise want to be left the hell alone.

Don't use GitHub. I don't. Random dipshits aren't even aware of me.

>To continue the inclusive nature of open source, we need to be able to trust a wide range of identities, but still with verified integrity. This implies a federated model for identities, perhaps similar to how we support federated SSL certificates today

Oh yes, of course Google supports TLS-flavoured snake oil to match the TLS snake oil.

I'm shocked MicroSoft is extending open source after embracing it so well, shocked.

I'm now recycling a previous comment of mine on this topic:

Companies used to have employees write code, rather than stitch together random garbage written by random dipshits who could be tricked into using loose licenses. That's one cause for concern. The only reason open source receives support is because it helps corporations defang Free Software and get gratis labor.

All of this new security theatre is always about trust and reputation, and not trusting those disgusting lone programmers such as me or other silly things; it's always really about doing anything but truly auditing that yucky code.


This sort of term is made up by c suites for c suites, arguing about semantics is rarely productive and I'm inclined to believe that's the case here.

I also think this battle is lost as GH and others will always yeld to the needs of folks that atually bring money to their business.

"Pure hobbist FOSS" mantainers likely need to start a Wikipedia ish non profit for software hosting or think about something else.


> This sort of term is made up by c suites for c suites, arguing about semantics is rarely productive and I'm inclined to believe that's the case here.

That's the point of the article, yes. It's about the mismatch of expectations and behaviors between C suite, which expects every business to do its duty on pains of lawsuit, and hobbyist FOSS programmers, who don't see themselves as in business and who have less than zero duty to FrobnitzWare/Invisible Corporation kabushiki gaisha LLC Limited AG despite the fact that August Corporate Personage decided to base Essential Business Processes on their work.

I mean, fundamentally, the programmers are in the right, in that the company cannot, in point of fact, rely on a contract it never drew up or got anyone to sign, but the companies will do the legal equivalent of holding their breath and stamping their feet and this is something people will, every so often, write blog posts about. Heck, maybe, just this once, a blog post will inform some C suite suits about the power gradient here (corporations: none, rando programmers: quite a bit, in fact) and maybe nudge some of them to, maybe, not make software that collapses when left-pad's author has another Big Idea.


Do you have an example of the C suites stamping their feet? I am in this space and I have seen no such foot stamping from anyone.


Just one I found quickly:

> If you are a multi billion dollar company and are concerned about log4j, why not just email OSS authors you never paid anything and demand a response for free within 24 hours with lots of info? (company name redacted for my peace of mind)

https://twitter.com/bagder/status/1484672924036616195

https://daniel.haxx.se/blog/2022/01/24/logj4-security-inquir...

Maybe "Stamping their feet" is a bit hyperbolic for this case, but it demonstrates the cultural mismatch very well: The email assumes the Log4J package comes from the "Log4J Company" such that a business requesting the results of an internal audit would be met with something other than an annoyed FOSS developer Tweeting about this clueless moo sending a form email to a random programmer they have no prior relationship with.

As Daniel himself says in the follow-up Tweet:

> I replied saying I'd be happy to answer all the questions as soon as we have a support contract.

There was, deep inside the company, an internal process which blindly assumed a contract would be present, and acted accordingly.



This isn't a C suite case though. It's just a procurement staffer going through the motions.


Maybe Github and repository maintainers care about society at large.

If the bet could be objectively settled I'd happily stake my entire savings on the question, because I've already talked to the folks making those decisions.


I'd hazard an inability to imagine oneself as "not the main character", to recognize that a majority likely have other priorities, causes a sizable share of debates on HN.

It's the opening question in PG's essay linked today.


Am I missing some background? Author gives out software "WITHOUT WARRANTY" and can just ignore PRs etc. Why is the author complaining?


> and can just ignore PRs

This complaint is in my opinion rather specific to GitHub (and could thus likely be solved by switching to a more repository provider, or hosting the repository oneself), but not being able to disable PRs at your own repositories is like having a blog hosted by a provider that does not offer the option to disable blog comments.


Eh, sounds like a really minor annoyance. It's clear that the PRs aren't your own content, unlike comments which you're sorta responsible for moderating or else you'll often end up with vile stuff all over your family-friendly blog post.


> It's clear that the PRs aren't your own content, unlike comments which you're sorta responsible for moderating.

I see this differently: you have some own content

* a blog post

* a Git repository

that is hosted by some provider. "Everybody" can add their own "scribblings" next to it:

* comments

* PRs

In both cases it is clear that these "scribblings" are not your own content.

Why do you thus make the difference that in one case one is some kind of responsible for moderating, but not in the other case?


Cause PRs aren't presented very upfront to visitors and are far less frequently abused. A lot of the open source world relies purely on goodwill, just like the author says.

Also, blogs are on your own domain name usually, which at least gives the illusion that it's your own website rather than just your little tenancy on a blog platform. Some platforms don't give you a domain technically but a / instead, like FB Pages and Reddit, and they explicitly assign you the responsibility of moderating your own page. If their own moderators have to step in, it often leads to deletion.


> "WITHOUT WARRANTY"

Liability disclaimers are a legal grey area. Even the US pulls aside some developers at customs because of their work on OSS projects. Imagine what some of the least tolerant countries do.


Yeah, but that doesn't seem to be the complaint here. It's more about GitHub's workings.


I must disagree. Software supply chain is very different than supply chain of physical goods, but, it is still a chain of supplies in the form of software code.

Looking at it strictly from an open source perspective is having too narrow of a focus. If I purchase O365 licenses from a reseller and they buy it from Microsoft, that's a supply chain. If I have an embedded system that has a proprietary OS made by a 3 person company who use a proprietary firmware from other vendors, that is a supply chain.

The npm example is a good example of a convoluted and unreliable supply chain. OSS is an example of a supply chain that is by design unreliable, the suppliers disavow all responsibility to support or guarantee its features. OSS supply chain being unreliable does not mean the supply chain does not exist, it just means you have to compensate for the unreliability yourself. The real life equivalent would be sourcing materials from guys standing at a street corner selling stuff that "fell off a truck" or "made in their garage".


The author is being even too polite IMHO, since there IS a "contract" (legal working agreement, or license) when involving open source. And it CLEARLY stipulates, usually even in capital letters, exactly what the author is conveying in the article! That the software is "as-is" and you should not expect -let alone demand- anything from authors. Anything beyond that is good will from open source authors, which historically has varied wildly from project to project.

I fully agree with the author though; it was even funnier when NPM sarcastically blamed me for losing my password on the reset screen, since it was THEM who lost their users passwords and force-reset them all, so now I had to reset my password, forced to use 2FA and being mocked while doing all of that ONLY to be able to share my work for free with the world. I was very, very close to say fuck NPM I'm out, unfortunately they have a de-facto monopoly.


so publish on your own platform then?

don't expect to get to publish on someone else's platform then not play by their rules regarding how published material is handled.


There's a business opportunity here for making a "slow copy" of npm (or PyPI or whatever): start with a copy and then just only upgrade packages after you've checked the changes. You could sign and manage the BOM, perhaps even ensuring that $BIG_ENTERPRISE only got the versions they had asked for no matter what request they make (oh, sorry for the dev trying to get their job done -- go ask IT).


I've heard of software pipelines and software dependency but this is the first I've ever heard of software supply chain.

I don't think the metaphor is apt. Dependencies are fungible you can branch, tag, freeze, lock, etc whatever your requirements are and continue trucking along. Physical Supply chains are not that at all. In some cases you literally cannot go back to the previous revision ever again.


Physical supply chains are full of fungible products such as wheat, pork bellies or gold.

Such products are what the word "fungible" was first used to describe.


After the Log4J vulnerability, people started using the term "software supply chain".

I think this comic demonstrates the "problem": https://xkcd.com/2347/


I've seen the case made that engineers who develop FOSS don't want to be paid for their work specifically because once they accept payment they feel a stronger obligation to maintain the software and respond to issues from those paying entities rather than focusing on the issues they want to or generally doing work as they see fit.

We all use FOSS - its one of the most unique aspects of software engineering compared to other industries IMO. The world would literally not have advanced nearly as far without it.

I'm incredibly grateful to work in a field that views FOSS maintainers so well and thankful to the FOSS for all the incredible and fun technology I get to use everyday.


So fwiw I think the whole concept of the "software supply chain" actually stems from "supply chain" attacks. It's kind of like once that turn of speech entered the fray, suddenly there was a software supply chain (for OSS).


I personally like the metaphor (but then again I work on a major package manager).

I like to think of it this way...

- we download code

- from the internet

- written by unknown individuals

- that we haven't read

- which we execute

- with full permissions

- on our trusted devices

- where we keep our most important data

(someone wrote this that I found in the SSC space)

Given a supply chain attack can happen at many of those steps above, but is also extended to source code, build systems, and production services, it's a great metaphor for visualizing the complex "software supply chain".


interesting points xie is making: 1) it's not a supply chain bc no money is changing hands, 2) therefore no contracts or enforcement, 3) many market participants are hobbyists

I think this speaks to the fundamental weirdness of the information economy -- it's highly deflationary, with some extreme cases where hobbyists outperform paid software companies

Hobbyists + open groups beat companies in cases where IP concerns makes the software worse, like if they're making something a saas that should be a 100-line library. And also in cases where everyone in industry needs a thing and they're happy to have a great standard free one rather than paying three shitty vendors.

I don't know this, but I think a chunk of major OSS projects are maintained by people at big companies or universities. (Postgres may have been associated with fujitsu for a while, guido van rossum + linus torvalds parked in various companies, some fraction of kube committers are at bigcos or corporate-backed CNCF). I have nadia eghbal's 'building in public' on my list to read, guessing she knows the answer to this question.


I respectfully disagree with the author.

Once you put stuff on someone else’s servers, they can do whatever they want to the package as long as it doesn’t break the license. That includes marking the package as critical or even making a fork the canonical repo. If you want that kind of control, you need to become your own package provider. Or distribute your code with a more restrictive license.


In fact, it's just as simple as that: every package today has a license. If the license says that the author has no responsibility and liability for it - so it is. If the package has no license - you cannot use it in commercial software anyway.


This article is hyperbole. Everyone involved knows that it's an imperfect metaphor, but that is in the nature of metaphors. Referring to the metaphor as "dehumanizing" is just silly.


> The author of an open source project usually makes NO guarantees and assumes no liability.

That's pretty much true of commercial software as well.


168 comments and nobody pointed out the role of Linux distributions...


There are several forms of OSS and it seems people often conflate them.

There is:

* "open source as a reference"

* "open source as a line item"

* "open source as a funnel"

It seems like our industry favors line item OSS. Personally, I think the other two are far more useful.

Line item OSS productizes a high-value project inside a company and releases it publicly for no charge. The goal of open sourcing is generally to steer the industry in a direction compatible with your company's architecture. Often times companies are trying to:

* Cultivate engineering talent trained on their architecture that they can hire in the future

* Increase maturity of their software stack by exposing it to environments outside their current architecture

* Recruit maintainers to drive down the cost of maintaining an internal pattern

* Cultivate an ecosystem of plugins and modules they can rely on for future engineering projects

* Cultivate good will in the developer community as a recruiting/dev-rel tactic

* etc.

Line-item OSS projects typically carry a high cost, they don't just become successful when you push to a public repository. These projects require devrel presence to meet 3rd party engineers where they are (conferences, blogs, documentation, branding, logos, news aggregator sites, etc.). They require maintainers to mentor 3rd party engineers through pull requests and issues. You have to navigate community building with codes of conduct, boards, foundations, etc. This all requires salaried employee's time. This high cost raises the bar for contributing to the commons: the return has to cover the salaried time spent on open sourcing the project. If the return doesn't cover your investment, ROI is negative and you're expecting companies to contribute for charity. What's worse is that failing on a line item OSS project often results in the opposite of what you want: a perception that your company is bad at OSS or not a great community participant. Every line item OSS project is a high-cost bet carrying risk to your reputation.

Open source as a reference, in contrast, is closer to publishing a conference paper. You describe how you are doing something, publish the source code as a reference, and call it a day. No community building. You don't commit to merging random PRs. You don't commit to fielding questions on issue trackers. It's a statement that what you are doing is working for you, that other engineers might benefit from similar patterns, and a jump start for motivated engineers to figure out how to apply the pattern themselves. You're contributing to the commons under a permissive license for the cost of a conference road show and a "cleanup" of the code to ensure you aren't leaking anything sensitive in your SCM history. I feel a lot more software would make it into the commons if we normalized this form of OSS.

"Open source as a funnel" is probably the most sustainable OSS model IMH(umble)O. "OSS funnel projects are associated with a vendor that sells services and products in that ecosystem. The services are particularly useful for companies building on top of OSS. If you are a small-ish org, every dependency you're bringing in is a pretty big risk. You have to have internal talent capable of maintaining the entire software stack. "OSS as a line item" and "OSS as a reference" do not give you any guarantees from the project. The maintainers aren't your employees and they owe you nothing. You often have no way to pay maintainers to solve your problems. A show stopper bug in those projects can bury your company. In contrast, "OSS as a funnel" has a vendor you can reach out to for consulting/contracting services to elastically scale out your engineering org for bugs in the stack. In the Node.js ecosystem, NearForm is a great example IMHO. They are maintainers on a pretty large catalogue of software (i.e. fastify). You can bring that into your architecture knowing that, if you end up blocked on a bug your team doesn't have the expertise to solve, you have a vendor you can reach out to and pay to solve your problem.

Sometimes line item OSS projects foster an ecosystem of vendors which in turn checks the OSS as a funnel box. For example, with GraphQL and React, you can't pay Facebook but someone out there will take your money to solve your problems.

I do consulting/contracting in the Node.js space and, when I bring modules into my client's architecture, having a vendor associated with the module is front-of-mind for me. I know I'm not leaving them with technical debt they have no way out of if something goes sideways. They always have me and they have a 3rd party vendor they can reach out to as well.


V




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: