> There is no way for changes in the outside world—such as a new version of a dependency being published—to automatically affect a Go build.
> Unlike most other package managers files, Go modules don’t have a separate list of constraints and a lock file pinning specific versions. The version of every dependency contributing to any Go build is fully determined by the go.mod file of the main module.
I don't know if this was intentional on the author's part, but this reads to me like it's implying that with package managers that do use lockfiles a new version of a dependency can automatically affect a build.
The purpose of a lockfile is to make that false. If you have a valid lockfile, then fetching dependencies is 100% deterministic across machines and the existence of new versions of a package will not affect the build.
It is true that most package managers will automatically update a lockfile if it's incomplete instead of failing with an error. That's a different behavior from Go where it fails if the go.mod is incomplete. I suspect in practice this UX choice doesn't make much of a difference. If you're running CI with an incomplete lockfile, you've already gotten yourself into a weird state. It implies you have committed a dependency change without actually testing it, or that you tested it locally and then went out of your way to discard the lockfile changes.
Either way, I don't see what this has to do with lockfiles as a concept. Unless I'm missing something, go.mod files are lockfiles.
There are some subtleties here, but go.mod files are not lockfiles, including go.mod files don't follow a traditional constraint/lockfile split, which I think is part of the point in that snippet you quoted.
Another way they differ is when installing a top-level tool by default npm does not use the exact version from a library's lockfile to pick the selected version of another library as far I as understand, whereas Go does by default use the exact version required by a library's go.mod file in that scenario.
In other words, go.mod files play a bigger role for libraries than a traditional lockfile does by default for libraries in most other ecosystems.
Here's a good analysis on the contrast between go.mod and the default behavior of more traditional lockfiles (using the npm 'colors' incident as a motivating example):
That link also includes some comments on 'npm ci' and 'shrinkwrap' that I won't repeat here.
All that said, go.mod files do record precise dependency requirements and provide reproducible builds, so it's possible to draw some analogies between go.mod & lockfiles if you want. I just wouldn't say "go.mod files are lockfiles". ;-)
Wouldn't build size increase a lot if transitive dependencies were pinned to direct dependency lockfiles? Like if library A says "use version 1.0.0 of library X" and library B says "use version 1.0.1 of library X", then you'd likely end up bundling duplicate code in your build.
Not saying the tradeoff isn't worth it, but pinning to dependency lockfiles isn't without downsides.
FWIW, that's not what Go does. In your scenario, a Go binary ends up with a single copy of library X -- the 1.0.1 version. That's because library A is stating "I require at least v1.0.0 of X", and library B is stating "I require at least v1.0.1 of X". The minimal version that satisfies both of those requirements is v1.0.1, and that's what ends up in the binary.
That behavior is Go's "Minimal Version Selection" or "MVS". There are many longer descriptions out there, but a concise graphical description I saw recently and like is:
That's the default behavior, but a human can ask for other versions. For example, a consumer of A and B could do 'go get X@latest', or edit their own go.mod file to require X v1.2.3, or do 'go get -u ./...' to update all their direct and indirect dependencies, which would include X in this case, etc.
Continuing that example -- in Go you end up with v1.0.1 of X by default even if v1.0.2 is the latest version of X.
That is a difference with many other package managers that can default to using the latest v1.0.2 of X (even if v1.0.2 was just published) when doing something like installing a command line tool. That default behavior is part of how people installing the 'aws-sdk' tool on a Saturday started immediately experiencing bad behavior due to the deliberate 'colors' npm package sabotage that happened that same Saturday.
In any event, it's certainly reasonable to debate pros and cons of different approaches. I'm mainly trying to clarify the actual behavior & differences.
What if the requirement was pinned specifically to 1.0.0 in order to avoid a bug introduced in 1.0.1. With a package that also requires a minimum 1.0.1, that should be unresolvable set of requirements and your package manager should fail to make a lockfile out of it.
The practice of dependencies’ dependencies being specified using SemVer version constraints to auto-accept minor or patch changes is the difference compared to Go, and why lockfiles will not always save you in the npm ecosystem. That said, approaches like Yarn zero-install can make very explicit the versions installed because they are distributed with the source. Similarly, the default of using npm install is bad because it will update lockfiles, you have to use npm ci or npm install —ci both of which are less well-known.
So it’s not impossible to fix, just a bad choice of defaults for an ecosystem of packages that has security implications about the same as updating your Go (or JS) dependencies automatically and not checking the changes first as part of code review. Blindly following SemVer to update dependencies is bad, from a security perspective, regardless of why or how you’re doing it.
> The difference is the npm ecosystem actively encourages automatically following SemVer because by default it uses a ^ to prefix the version number.
So does Go. In fact, Go only supports the equivalent of ^, there is no way to specify a dependency as '=1.2.3'. That is, whenever you have two different dependencies which use the same dependency at different (semver compatible) versions, go mod will always download the newer of the two, effectively assuming that the one depending on an older version will also work with the newer.
The only difference in this respect compared to NPM (and perhaps also Cargo or NuGet? I don't know) is that Go will never download a version that is not explicitly specified in some go.mod file - which is indeed a much better policy.
It’s very subtle, but there are some important differences. For example, lockfiles are not recursive in NPM: the NPM package (usually?) does not contain the lockfile and does not adhere to it when installed as a dependency. It will pick the newest version of dependencies that matches the spec in package.json.
Go mod files are used recursively, and rather than try to pick the newest possible version, it will go with the oldest version.
This avoids the node-ipc issue entirely, at least until you update the go.mod.
This really depends on the specific package manager: if you're building an application in Rust, its lockfile will contain the full tree of dependencies, locked to a specific version.
I might be misunderstanding GP, but I think what they're saying is that when package A depends on package B, building package A will use B's lockfile. Assuming that's the case, I think this is generally not how Rust does things, as by default Cargo.lock is explicitly listed in the .gitignore when making a library crate, although there's nothing stopping anyone from just removing that line. I think I remember reading in documentation somewhere that checking in Cargo.lock for libraries is discouraged (hence the policy), but I don't recall exactly where since it's been so long. (That being said, there's a pretty decent chance you were the one who wrote that documentation, so maybe you might remember!)
> I think this is generally not how Rust does things
That is correct.
> as by default Cargo.lock is explicitly listed in the .gitignore when making a library crate
Even if it is included in the contents of the package, Cargo will not use it for the purpose of resolution.
The "don't check it in" thing is related, but not because it will be used if it's included. It's because of the opposite; that way new people who download your package to hack on it will get their own, possibly different Cargo.lock, so you end up testing more versions naturally. Some people dislike this recommendation and include theirs in the package, but that never affects resolution behavior.
The article seemed to go out of its way not to mention any specific package manager or ecosystem. So I think comparing to Rust is completely reasonable.
It didn’t mention one by name, but Rust hasn’t been subject to any widely publicized supply chain attacks. They do, however, mention left-pad by name. I think it can be implied that they really did just mean npm.
They definitely should have said that then. Simply referring to "lockfiles" paints with a very broad brush that includes a number of package managers that don't have the problems that NPM does.
The default way Go handles go.mod is fairly different than the default way Cargo handles Cargo.lock files, including for example with libraries.
Also, when the blog says:
> Moreover, when a dependency is added with go get, its transitive dependencies are added at the version specified in the dependency’s go.mod file, not at their latest versions, thanks to Minimal version selection.
I believe that is significantly different than default Cargo behavior and for example default 'pub' behavior for Flutter (though I know approximately nothing about Flutter package management beyond a cursory search just now ;-)
To my knowledge, both Cargo and Flutter 'pub' prefer the most recent / highest allowed version by default when asked to solve constraints, whereas Go does not.
Cargo: [1]
> When multiple packages specify a dependency for a common package, the resolver attempts to ensure that they use the same version of that common package, as long as they are within a SemVer compatibility range. It also attempts to use the greatest version currently available within that compatibility range.
Flutter 'pub': [2]
> For each package in the graph, pub looks at everything that depends on it. It gathers together all of their version constraints and tries to simultaneously solve them. (Basically, it intersects their ranges.) Then it looks at the actual versions that have been released for that package and selects the best (most recent) one that meets all of those constraints.
In that same section, the blog describes the behavior of 'go install foo@latest' and contrasts it to how the default install "in some ecosystems bypass pinning."
That is also a difference in default behavior between Go and Cargo.
To install a 'foo' binary, 'go install foo@latest' gives you the latest version of foo, but the direct and indirect dependencies used are the versions listed in foo's go.mod or a dependency’s go.mod file (and not whatever the latest versions of those direct and indirect dependencies might be at the moment the install is invoked).
'cargo install foo' supports the optional --locked flag, but its not the default behavior: [1]
> By default, the Cargo.lock file that is included with the package will be ignored. This means that Cargo will recompute which versions of dependencies to use, possibly using newer versions that have been released since the package was published. The --locked flag can be used to force Cargo to use the packaged Cargo.lock file if it is available.
There are definitely pros and cons here, but to my knowledge it is not "just NPM" that is being contrasted in the blog.
Finally, I'm no world-class Rust expert, but I like using Cargo. I think Cargo is a fantastic tool that set the bar for package mangers, and it has done great things for the Rust community. But it is easier for communities to learn from each other with a base understanding of where & why different choices have been made, which is part of what is behind some of my comments around Go's behavior. ;-)
So, I believe that you're confusing two different cases here.
`cargo install` is not how you add a dependency, it's not like `npm install`. `cargo install` downloads the source for and installs a runnable binary program, like `npm install -g`. Cargo does not currently have an "add this dependency to my Cargo.toml" command built-in, but `cargo add` is coming from this purpose.
With `cargo install`, the default behavior is to completely recompute the dependency tree before doing the build. The `--locked` flag modifies that to use the lockfile included in the package to do that build instead (and in fact fail compilation if it does not exist). That lockfile will still be a full graph of all dependencies and transitive dependencies to build the binary, it doesn't like, recurse or use any lockfiles that exist in any of the dependencies' packages.
I might have misunderstood your comment, but in my GP comment I was indeed attempting to contrast 'go install foo@latest' with 'cargo install foo', which both install binaries. (I wasn't talking about 'go get bar@latest', which now is just for updating or adding dependencies to a project).
Also, I'm contrasting what happens by default at the moment either binary install command is run. My understanding is Cargo's (non-default) 'cargo install --locked foo' behavior is similar to the default behavior of 'go install foo@latest'. In other words, the default behavior is fairly different between 'cargo install foo' (without --locked) vs. 'go install foo@latest'.
I edited my GP comment to simplify the example to use 'foo' in both cases. Maybe that helps?
Ah yes, I did miss that, thank you / sorry :) Too many sub-threads around here!
I don't know go install's semantics well enough to know if that comparison is true or not, I'm just trying to make sure that Cargo's semantics are clear :)
It will copy a binary to $GOBIN. If the binary is not built, it will be built from source. If the source is not available on the local system, it will be fetched.
During the build, any dependencies of the build target not available of the local system will be fetched.
I believe Cargo does still have less strictness over dep versions than Go modules, since it will never use a module newer than the one specified in any go.mod file. Lockfiles are generally not honored recursively, and I don’t think Cargo is different here? Hope I’m not spreading misinformation, though I couldn’t find any docs with a cursory glance.
I don’t want to make assertions that I’m less sure of, but I think NPM and Cargo are actually more similar than different here. They both specify exact versions in lock files, for all nested dependencies, but don’t honor the lock files present inside dependencies, instead calculating the nested deps from the constraints.
I always forget the exact semantics, but the parent's description of them as "recursive" is not the same as Cargo; Cargo determines the full tree and writes out its own lockfile, if dependencies happen to have a Cargo.lock inside the package, it's ignored, not used.
As far as I understand it... the import path that you use to import a package acts as its identity, and only one version of any given package will be installed. The way that it will determine this is by choosing the lowest version specified in any package that depends on a given package. Major versions of packages are required to have different import paths with Go modules, so when depending on two different major versions of the same package, they are treated effectively as their own package.
> by choosing the lowest version specified in any package that depends on a given package
It picks the highest version specified in any of the requirements. (That's the minimal version that simultaneously satisfies each individual requirement, where each individual requirement is saying "I require vX.Y.Z or higher". So if A requires Foo v1.2.3 and B requires Foo v1.2.4, v1.2.4 is selected as the minimal version that satisfies both A and B, and that's true even if v1.2.5 exists).
Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5? But what happens if C is on 1.8 because it doesn't work with 1.5 because an API it needs doesn't exist in 1.5?
Are we not talking about the transitively pinned dependencies in the "lock" section, or are we talking about logical constraints?
Logical constraints would make more sense, but if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
> Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5?
No, you end up with the highest explicitly required version. So 1.8 in that scenario, if I followed. (Requiring 1.5 is declaring support for "1.5 or higher". Requiring 1.8 is declaring support for "1.8 or higher". 1.8 satisfies both of those requirements).
> if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
By default, you get 1.8 (for reasons outlined upthread and in the blog post & related links), but you have the option of getting the latest version of C at any time of your choosing (e.g., 'go get C@latest', or 'go get -u ./...' to get latest versions of all dependencies, and so on).
Also, you are using the word "pin". The way it works is that the top-level module in a build has the option to force a particular version of any direct or indirect dependency, but intermediate modules in a build cannot. So as the author of the top-level module, you could force a version of C if you needed to, but for example your dependency B cannot "pin" C in your build.
I'm fairly sure Go Modules does not support what you’re describing. It specifically avoided having a SAT solver (or something similar), unlike most package managers. You specify a minimum version, and that’s it. 1.8 would be selected because it is the highest minimum version out of the options 1.5 and 1.8 that the dependencies require. Unless you edit your go.mod file to require an even higher version, which is an option. Alternatively, you can always replace that transitive dependency with your own fork that fixes the problems, using a “replace” directive in your go.mod file.
If your dependencies are as broken as you’re describing, you’re in for a world of hurt no matter the solution. I also can't remember ever encountering that situation.
Well after 12 years of using ruby and bundler and maintaining a bundler-ish depsolver at work, and playing around a bit with cargo, I can say that it is becoming clearer as to why I don't grok go modules at all.
The lack of a depsolver is a curious choice...
I don't think my example was remotely "broken" at all, that's just another day doing software development.
> I don't think my example was remotely "broken" at all, that's just another day doing software development.
It's not the norm for me. If this is what you consider to be the norm, then this kind of statement doesn't make me feel any better about Ruby.
I will say that Bundler is one of the better package managers, but the existence of the constraint solver doesn't fix this problem -- Bundler doesn't allow you to have multiple versions of a single dependency. The problem is fundamentally the dependency not maintaining its compatibility guarantees, which I would definitely call "broken". Sometimes breakage is unavoidable, like with security fixes that you want to be available even to users of existing SemVer versions, but it should not be a common situation.
If I'm understanding you properly, recursive lockfiles means that if I depend on some chain of dependencies A->B->C->D->E, and E has a security vulnerability that they patch in a new version, I have to wait for A B C D and E to all update their lockfiles before the security vulnerability will be patched on my system?
That's not correct. You can unilaterally decide to update the version of E without waiting for anyone. Alternatively, if only C for example decides to update their required version of E, you would get that version of E if you updated your version of C (directly or indirectly), without needing to directly do anything with E yourself.
But wouldn't that have the same issue then? Developers decide to update their dependencies to patch any security vulnerabilities, and wind up adding installing node-ipc's malicious update
The difference is that it's an explicit choice instead of other package managers who'd happily install latest compromised versions of packages by default.
It is trivial to manually upgrade dependencies in NPM. You just use `npm update <package>` with an optional version number if you want. And upgrading all dependencies of a Go package is also a single command. So honestly it seems like there is very little difference. My main point here is the trade-off. Either you reduce the friction for upgrades, and run the risk of malicious upgrades like node-ipc. Or you increase the friction, and run the risk of security vulnerabilities being unpatched in many projects.
I personally prefer the former. Encourage upgrades, but then NPM should also have a separate repository for community verified / trusted packages to reduce the chance of a random single developer damaging the entire ecosystem (left-pad, node-ipc, etc)
If I set up a new Node project I get the highest 'supported' version of whatever. If I add a new dependency I get the latest version of any transitive dependency I didn't already have. As far as I know that's impossible to disable. That's the automated upgrade I mean.
I see, in that case yes Go does have more tooling for being able to install the minimum vs the latest of all packages, using their `update` command if you want the latest. But it would also be trivial for Node to add a command to grab the minimum of all dependencies when installing new packages. They just haven't felt the need to add such a feature. Because again, it comes down to which side you want to encourage: installing minimum versions to prevent malicious updates, or installing latest to patch security vulnerabilities.
"A module may have a text file named go.sum in its root directory, alongside its go.mod file. The go.sum file contains cryptographic hashes of the module’s direct and indirect dependencies."
And
"If the go.sum file is not present, or if it doesn’t contain a hash for the downloaded file, the go command may verify the hash using the checksum database, a global source of hashes for publicly available modules."
Should be stressed on. If I committed a dependency version (go.mod) and checksum (go.sum) along with the code, either I get a repeatable build everywhere, or build fails if dependency not found or found to be modified.
I am not sure if all other package managers include checksum with dependency version.
I suspect the primary purpose of the word "may" in that sentence is that you can choose to disable checking the hash against the Certificate Transparency style https://sum.golang.org. In other words, you can opt out. If you do, you fall back to your local go.sum file, which is more-or-less a "TOFU" security model:
https://en.wikipedia.org/wiki/Trust_on_first_use
Discussing "what is a lockfile" is a bit of a headache because different languages have different files which do different things. Generally speaking, there's some file which specifies the dependency versions and some file with cryptographic checksums of the all transitive dependencies.
In Go it's go.mod / go.sum. In NPM, it's package.json / package-lock.json. In Rust it's Cargo.toml / Cargo.lock.
Diving into the exact details of what the author is saying is a bit outside my headspace at the moment. I think the author of the article may not actually understand the scenario where Go's package system differs. (I'm not sure I do, either.)
Suppose you have your project, projectA, and its direct dependency, libB. Then libB has a dependency on libC.
If projectA has a lockfile, you get exactly the same versions of libA and libB. This is true for Go, NPM, and Cargo. However, suppose projectA is a new project. You just created it. In Go, the version of libB that makes it into the lockfile will be the minimum version that libA requires, which means that any new, poisoned version of libB will not transitively affect anything that depends on libA, such as projectA. With NPM, you get the latest version of libB which is compatible with libA--this version may be poisoned.
Conversely, you will get any old security-buggy version of libB instead.
Most package managers when adding a new dependency assume newer versions are "better" than older versions. Go's minimum version system assumes older is better than newer.
I don't think there's any clear argument you can make on first principles for which of those is actually the case. You'd probably have to do an empirical analysis of how often mailicious packages get published versus how often security bug fix versions get published. If the former is more common than the latter, then min version is likely a net positive for security. If the latter is more common than the former, then max version is probably better. You'd probably also have to evaluate the relative harm of malicious versions versus unintended security bugs.
Every change that fixes a security issue implies the existence of a change that introduced the security issue in the first place. Why is bumping a version more likely to remove security issues instead of introduce them?
The reason why older is better than newer has more to do with the fact that the author has actually tested their software with that specific version, and so there's more of a chance that it actually works as they intended.
Security issues aren't introduced intentionally, oftentimes they are found much later on in code that was assumed to be secure. Like the SSL heartbleed vulnerability. Once a vulnerability like that is discovered, you _want_ every developer to update their deps to the most secure version
My statement had nothing to do with intent. Conversely, once a vulnerability is introduced (intentionally or not), you don't want every developer to update their deps to the newly insecure version.
Exactly, so it's a trade-off, do you want to encourage updates at the risk of malicious updates (like with node-ipc). Or do you want to add friction to updates and thus risk security vulnerabilities persisting for longer. Node chooses one approach, Go chooses the other.
Again, it's not just malicious updates. Normal updates can also introduce security vulnerabilities. For example, I have a dependency at v1.0 and v1.0.1 introduces a security bug unintentionally. It is eventually fixed in v1.1. If I wait to update until v1.1, then I am not vulnerable to that bug whereas an automatic update to v1.0.1 would be vulnerable. My point is that in expectation, updating your dependency could be just as likely to remove a security vulnerability as it is to add one.
Go just expects you to manually trigger the updates. Thats all. It still is in favor of updating to take security fixes, so i think your argument is wrong.
Let's say my_app uses package foo which uses package bar.
It turns out there is a security bug in bar. The bar maintainers release a patch version that fixes it.
In most package managers, users of my_app can and will get that fix with no work on the part of the author of foo. I'm not very familiar with Go's approach but I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.
> I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.
That is incorrect. The application's go.mod defines all dependencies, even indirect ones. Raise the version there, and you raise it for all dependencies. You cannot have one more than one minor version of a dependency in the dependency graph.
The version constraint is always listed in your top level go.mod file, so you know the dependency exists, no digging into the dependency tree required at all, and it’s not hidden in some lock file no one ever looks at. Plus, there are plenty of tools that help you with this problem, including the language server helping you directly in your editor and Dependabot on GitHub.
I’m not aware of any languages that send you an email when your dependencies are out of date, so yes, you need to check them. Dependabot can do this for you and open a PR automatically, which will result in an email, so this is one way for people to stay on top of this stuff even for projects they deploy but don’t work on every single week.
If you’re suggesting that indirect dependencies should automatically update themselves, then you are quite literally saying those code authors should have a shell into your production environments that you have no control over, compromising all your systems with a single package update that no one but the malicious author got to review. It is possible with tools like Dependabot to be notified proactively when updates are required so you can review and apply those, but it is not possible to go back in time and un-apply a malicious update that went straight to prod.
Repeatedly assuming that the Go core team never thought through the design of Go Modules and how it relates to security updates is such a strange choice. Go is a very widely used language with tons of great tooling.
I'm sorry but I'm not super familiar with the workflow for working with dependencies in Go, I've only read about it. You say:
> Raise the version there.
Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?
But that transitive dependency was first added there by the Go tool itself, right?
How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?
> Repeatedly assuming that the Go core team never thought through the design of Go Modules
I'm certainly not assuming that. But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.
>No, run `go get <package with vuln>@<version that fixes vuln>` and Go will do it for you.
The point GP was making is that it's not a given you'll know that there's a security vuln in a sub-sub-sub-dependency of your app. Is it reasonable to expect developers to manually keep tabs on what could be dozens of libraries that may or may not intersect with the dependencies of any other apps you have on the go?
Maybe for Google scale where you can "just throw more engineers at the problem".
> Is it reasonable to expect developers to manually keep tabs on what could be dozens of libraries that may or may not intersect with the dependencies of any other apps you have on the go?
Well, in the NPM model you need at least one transitive dependency to notice it and upgrade, and you need to notice your transitive dependency upgraded. But also, it might upgrade despite nobody asking for it just because you set up a new dependency.
In the Go model... you need at least one transitive dependency to notice it and upgrade, and you need to notice your transitive dependency upgraded. But at least it won't ever upgrade unless someone asked for it.
Well, the easy thing is just let something like Dependabot update your stuff.
If you are just wanting "update all my stuff to the latest version", just run `go get -u ./...`?
> Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?
"Common" is probably not accurate, but what's wrong with hand editing it? If you mess up, your project won't compile until you fix it. You can update the versions a dozen different ways: editing by hand, `go get -u <dependency>` to update a specific dependency, `go get -u ./...` to update all dependencies, using your editor to view the go.mod file and select dependencies to update with the language server integration or by telling the language server to update all dependencies, by using Dependabot, or however else you like to do it. The options are all there for whatever way you're most comfortable with.
> But that transitive dependency was first added there by the Go tool itself, right?
So what? It's still your dependency now, and you are equally as responsible for watching after it as you are for any direct dependency. Any package management system that hides transitive dependencies is encouraging the user to ignore a significant fraction of the code that makes up their application. Every dependency is important.
> How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?
You already know[0] the answer to the most of that question, so I don't know why you're asking that part again. As a practical example, here is Caddy's go.mod file.[1] You can see the two distinct "require" blocks and the machine-generated comments that inform the reader that the second block is full of indirect dependencies. No one looking at this should be confused about which is which.
But the other part of that question doesn't really make sense. If you don't "preserve" the dependency, your code won't compile because there will be an unsatisfied dependency, regardless of whether you wrote the dependency there by hand or not, and regardless of whether it is a direct or indirect dependency. If you try to compile a program that depends on something not listed in the `go.mod` file, it won't compile. You can issue `go mod tidy` at any time to have Go edit your go.mod file for you to satisfy all constraints, so if you delete a line for whatever reason, `go mod tidy` can add it back, and Go doesn't update dependencies unless you ask it to, so `go mod tidy` won't muck around with other stuff in the process. There are very few ways the user can shoot themselves in the foot here.
Regardless, it is probably uncommon for people to manually edit anything in the go.mod file when there are so many tools that will handle it for you. The typical pattern for adding a dependency that I've observed is to add the import for the dependency from where you're trying to use it, and then tell your editor to update the go.mod file to include that dependency for you. The only time someone is likely to add a dependency to the go.mod file by hand editing it is when they need to do a "replace" operation to substitute one dependency in place of another (which applies throughout the dependency tree), usually only done when you need to patch a bug in third party code before the upstream is ready to merge that fix.
In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.
> But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.
They were extremely late to the party, so they were able to learn from everyone else's mistakes. I really don't think it should be surprising that a latecomer is able to find solutions that other package management tools didn't, because the latecomer has the benefit of hindsight. They went from having some of the worst package management in the industry (effectively none; basically only beating out C and C++... and maybe Python, package management for which has been a nightmare for forever) to having arguably the best. Rust's Cargo comes extremely close, and I've used them both (as well as others) for the past 5+ years in several professional contexts. (Yes, that includes time before Go Modules existed, in the days when there was no real package management built in.) It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.
Some "trade-offs":
- For awhile, the `go` command would constantly be modifying your `go.mod` file for you whenever it didn't like what was in there, and that was definitely something I would have chalked up as a "trade-off", but they fixed it. Go will not touch your `go.mod` file unless you explicitly tell it to, which is a huge improvement in the consistency of the user experience.
- Go Modules requires version numbers to start with a "v", which annoys some people, because those people had been using git tags for years to track versions without a "v", so you could argue that's a trade-off too.
- There has been some debate about the way that major versions are implemented, since it requires you to change the name of the package for each major version increment. (By appending “/v2”, “/v3”, etc to the package name. The justification for this was to allow you to import multiple major versions of the same package into your dependency graph — and even the same file — without conflict.)
- The fact that the packages are still named after URLs is a source of consternation for some people, but it's only annoying in practice when you need to move the package from one organization to another or from one domain name to another. It's simply not an issue the rest of the time. Some people are also understandably confused into thinking that third party Go modules can vanish at any time because they're coming from URLs, but there is a transparent, immutable package proxy enabled by default that keeps copies of all[#] versions of all public dependencies that are fetched, so even if the original repo is deleted, the dependency will generally still continue to work indefinitely, and the lock sums will be retained indefinitely to prevent any malicious updates to existing dependency versions, which means that tampering is prevented both locally by your go.sum file and remotely by the proxy as an extra layer of protection for new projects that don't have a go.sum file yet. It is possible to disable this proxy (in whole or in part) or self-host your own if desired, but... I haven't encountered any use case that would dictate either. ([#]: There are a handful of rare exceptions involving packages without proper licenses which will only be cached for short periods of time plus the usual DMCA takedown notices that affect all "immutable" package registries, from what I understand.)
Beyond that... I don't know of any trade-offs. Seriously. I have taken the time to think through this and list what I could come up with above. A "trade-off" implies that some decision they made has known pros and cons. What are the cons? Maybe they could provide some "nicer" commands like `go mod update` to update all your dependencies instead of the somewhat obtuse `go get -u ./...` command? You have complained in several places about how Go combines indirect dependencies into the `go.mod` file, but... how is that not objectively better? Dependencies are dependencies. They all matter, and hiding some of them doesn't actually help anything except maybe aesthetics? I would love to know how making some of them exist exclusively in the lock file helps at all. Before Go Modules, I was always fine with that because I had never experienced anything better, but now I have.
There are plenty of things I would happily criticize about Go these days, but package management isn't one of them... it is simply stellar these days. It definitely did go through some growing pains, as any existing language adopting a new package manager would, and the drama that resulted from such a decision left a bitter taste in the mouth of some people, but I don't believe that bitterness was technical as much as it was a result of poor transparency from the core team with the community.
> You already know[0] the answer to the most of that question, so I don't know why you're asking that part again.
My point is that once you start editing (either by hand or through a tool) the version numbers in the second section, then that's no longer cached state derived entirely from the go.mod files of your dependencies which can be regenerated from scratch. It contains some human-authored decisions and regenerating that section without care will drop information on the floor.
Imagine you:
1. Add a dependency on foo, which depends on bar 1.1.
2. Decide to update the version of bar to 1.2.
3. Commit.
4. Weeks later, remove the dependency on foo and tweak some other dependencies. Tidy your go.mod file.
5. Change your mind and re-add the dependency on foo.
At this point, if you look at the diff of your go.mod file, you see that the indirect dependency on bar has changed from 1.2 to 1.1. Is that because:
A. You made a mistake and accidentally lost a deliberate upgrade to that version and you should revert that line.
B. It's a correct downgrade because your other dependency changes which have a shared dependency on bar no longer need 1.2 and it is correctly giving you the minimum version 1.1.
Maybe the answer here is that even when you ask the tool to remove unneeded transitive dependencies, it won't roll back to an earlier version of bar? So it will keep it at 1.2?
With other package management tools, this is obvious: Any hand-authored intent to pin versions—including for transitive dependencies—lives in one file, and all the state derived from that is in another.
> In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.
This sounds like you more or less get the same results as you would in other package managers, but with extra steps.
I don't know. I guess I just don't understand the system well enough.
> It contains some human-authored decisions and regenerating that section without care will drop information on the floor.
> Maybe the answer here is that even when you ask the tool to remove unneeded transitive dependencies, it won't roll back to an earlier version of bar? So it will keep it at 1.2?
I'm not aware of any package management system that will remember dependencies you no longer depend on, except by human error when you forget to remove that dependency but keep punishing yourself and others by making them build and install that unused dependency. No matter where the information lived before it was deleted, it's still up to the human to do the right thing in a convoluted scenario like you're describing.
> With other package management tools, this is obvious: Any hand-authored intent to pin versions—including for transitive dependencies—lives in one file, and all the state derived from that is in another.
That doesn't solve anything if you don't look go back to the commit that had that information. If you do go back to that commit, you have all the information you need right there anyways. You can add your own comments to the `go.mod` file, so if something changed for an important reason, you can keep track of that (and the why) just as easily as you can in any other format. Actually, easier than some... does package.json even support comments yet? But it only matters if you go back and look at the previously-deleted dependency information instead of just blindly re-adding it as I imagine most people would do, which is a huge assumption.
>> It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.
> This sounds like you more or less get the same results as you would in other package managers, but with extra steps.
> I don't know. I guess I just don't understand the system well enough.
I've done my best to explain it to you, literally multiple hours of my day writing these comments. Maybe I suck at explaining, or maybe you're not interested in what I have to say, or maybe I'm somehow completely wrong on everything (but no one has bothered to explain how). "More or less the same" is not the same. The nuances make a huge difference, and Go Modules has done things incredibly well on the whole. Package managers aren't a zero sum game where you shuffle a deck of pros and cons, and you end up with the same number of pros and cons at the end.
> I've done my best to explain it to you, literally multiple hours of my day writing these comments.
Not the op, but wanted to express appreciation here. I read all your comments and feel like I gained a lot of clarity and insight from them. Would love to read your blog if you had one.
I appreciate the time you put into these comments and definitely learned a lot from them.
My original point was just that the article reads like an incorrect indictment of all other package managers that use lockfiles. Whether Go does things better is for most part orthogonal to what I was initially trying to get at.
As I understand it, that's true in the simple case. If you have `my_app -> foo -> bar` then there's only one path to bar, and you only get a new bar when you upgrade foo.
It's more complicated in general, with diamond dependencies. There needs to be a chain of module updates between you and foo, with the minimum case being a chain of length one where you specify the version of foo directly.
So, people do need to pay attention to security patch announcements. But popular modules, at least, are likely to be get updated relatively quickly, because only one side of a diamond dependency needs to notice and do a release.
> As I understand it, that's true in the simple case. If you have `my_app -> foo -> bar` then there's only one path to bar, and you only get a new bar when you upgrade foo.
This is not correct. You can update bar independent of foo directly from the top-level go.mod file in your project.
I'm not talking about direct dependencies. Direct and indirect are all listed in the go.mod file. If they aren't listed there, then they aren't in your final binary. If you delete indirect dependencies from the top-level go.mod, your project will fail to compile.
You keep harping on old buggy version where Go has been very clear that it is operator's explicit responsibility to have correct/updated/fixed versions of dependency running.
It specially does not look good in your case considering you work for Google on a different programing language. If you have a clear point to make then compare it with your approach. Instead of making neutral sounding arguments when they are not.
Don't drag in ad-hominem attacks. If you want to defend Go's approach, explain why having it be the "operator's explicit responsibility" is a good policy, likely to make apps (in general) more secure. The obvious implication of the example given is that, on average, it will be a mess.
The code itself isn’t the only risk factor; it’s weighted by others, like discoverability, which are asymmetric in time. If the white hats fix an issue in version n+1, they’re going to make sure people know. If black hats (or normal devs making a mistake) introduce an issue, no one will tell you about it.
I.e. even if both strategies win just as often, min-version pulls ahead by taking less of a hit from losses.
The author is saying that Go provides the same guarantees with just a package list in the go.mod file that other package managers need both a package list and lock file to solve.
go.sum is essentially a distributed / community maintained transparency log of published versions of packages.
Maybe I'm just not familiar with it enough but I don't see how merging a package manifest and lockfile into a single file is a net win.
This means it's no longer clear which dependencies are immediate and which are transitive. It's not clear which versions are user-authored constraints versus system-authored version selections. For dependencies that are transitive, it's not clear why the dependency is in there and which versions of which other dependencies require it.
Other packages separate these into two files because they are very different sets of information. Maybe Go's minimum version selection makes that not the case, but it still seems user-unfriendly to me to lump immediate and transitive dependencies together.
I think I personally lean towards keeping them in separate files entirely because I like a clearer separation between human-authored content and machine-derived state.
In other systems, you do that by creating a direct dependency. Even though your code doesn't directly import the module, you author an explicit dependency because you care about the version of the module that your app ends up using.
This way, there's a clear separation between human-authored intentional dependencies and the automatically-derived solved versions and transitive dependencies based on that.
If you see a diff in your package manifest, you know a human did that. If you see a diff in the lockfile, you know that was derived from your manifest and the manifests you depend on. The only human input is when they choose to regenerate the lockfile and which packages they request to be unlocked and upgraded. That's still important data because regenerating the lockfile at different points in time will produce different results, but it's a different kind of data and I think it's helpful to keep it separated.
Direct means that your code actually imports it directly instead of transitively. It has nothing to do with the version of that dependency being selected in the go.mod file.
Yes. The go.sum file that sits alongside go.mod keeps track of the hashes so that no modification like that can be made, and dependency fetches actually transparently go through a module proxy/mirror that keeps those same hashes as well, and it will prevent you from getting an altered version of a known module even if you’re starting a new project and don’t have a sum file yet. Versions can’t be republished.
Most package managers have lockfiles. Yes, npm's decision to have both "npm install" and "npm ci", just so you can confuse and mislead developers, is a bit silly.
But Ruby's Bundler, for example, has been refusing to run your code if your lockfile is inconsistent for as long as I remember.
Locking dependencies is, generally, a solved problem across most ecosystems (despite node botching its UX). Go doesn't get to claim that it's superior here.
But of course, supply chain attacks are still possible with lock files. Because somebody is going to update your dependencies at some point (often for security reasons). And at that point you might be pulling in a malicious dependency which you haven't carefully vetted (because nobody has time to vet all their dependencies thoroughly nowadays).
That's still an unsolved problem, as far as I know. I don't think that Go has solved it.
> Locking dependencies is, generally, a solved problem across most ecosystems
As someone who is building a package manager for work, and has looked at pretty much every package manager out there (and their ecosystem adoption), I can only say that those don't reflect the current reality of package management (no matter how much I wish it were true).
Bundler was the first mainstream package manager to adopt a lockfile (AFAIK) a mere 12 years ago. Many many language ecosystems predate that and are still lacking lockfiles (or even widespread adoption of a single compatible package manager).
NPM only got lockfiles 5 years ago (after being pressured by yarn). Gradle got them less than 3.5 years ago, and Maven still doesn't have them (though a niche plugin for it exists). The Python ecosystem is still a hot mess, with ~3 competing solutions (Poetry, Pipenv, Conda), of which Conda just got a 1.0 of their decent conda-lock incubation project a month ago, but due to how setuputils works, the cross-platform package management story is broken almost beyond recovery. In Conan lockfiles are still an experimental feature today.
I could go on and on, but I hope that I could paint a picture that while one could argue that with the advent of lockfiles, locking dependencies has become a solved problem _conceptually_, the current status of implementation across ecosystems is still horrible. I'm also constantly amazed about how little love is put into package managers in most language communities, even though they are so crucial for their respective ecosystems.
As far as I can tell nowadays Go does have one of the better package managers, which given their horrible starting point is quite the feat. As a nice side-effect of experiments in the Go package ecosystem, one of the people working on go dep also created one of the best resources around package managers: https://medium.com/@sdboyer/so-you-want-to-write-a-package-m...
You are probably right that, practically, not every language has "solved" dependency locking and that I was probably unduly extrapolating from my experiences with ecosystems where this has been solved, but there are enough package managers (Bundler, Yarn, Poetry, that I know of, and from what people claim, also Composer and Cargo) that have solved it so that go claiming credit for it seems unwarranted. If anything, this should be credited to Bundler (though it's possible that it wasn't the first package manager to do so).
I can give no credit to npm, it would never have had lockfiles if not for yarn, and even its current attempt seems half-assed.
Python has the problem of too many package managers, some which are bad, unfortunately (the list of open bugs for Pipenv is especially alarming; I once had to rip it out of a project because the dependency resolution failed after half an hour with a stack overflow). That said, poetry solves the problem well and correctly, IMHO.
Over in the Java world, you're right that dependency locking is a bit rarer. But I also think the situation is not nearly as dire there. BOMs, where lists of compatible dependencies are curated, are relatively common, so that alleviates some of the pain. Plus, there seems to be less churn than in some other ecosystems. Still, it would probably be technically better to use dependency locking.
To be fair the title of the article is "How Go Mitigates Supply Chain Attacks" not how it solves.
And I think it does a good job at that.
For example if you had any JavaScript package that depended on node-ipc in your project, a simple npm install after cloning the project would download code that tries to corrupt files in your disk if the malicious code determined that your IP was from Russia. (before the malicious package was taken down/fixed)
With Go you would have to explicitly bump dependency versions. Simply cloning the project and installing dependencies would not have downloaded the malicious version. And bumping would at the very least appear as a diff in a Pull Request.
Yes, node's attempt to include lockfiles is botched, unfortunately. Not only are there UX issues (it's completely unintuitive that you should be using "npm ci", for example), but the lock file can also get corrupted e.g. during a merge conflict and npm performs no sanity checking on it.
I once had a case where a build was suddenly failing. The reason turned out to be that (for whatever reason) a dev had managed to corrupt the lock file, probably during a merge conflict, in such a way that the entry for package A actually contained the URL for package B. It turns out that npm didn't realise that this was inconsistent (with the package.json, and with the npm registry) and downloaded package B but exported it as package A, making the error incredibly hard to pin down.
Doesn't node have lockfiles? Cloning a project and running npm install would install the exact dependencies declared in the lockfile right? To quote the docs[1]:
> The goal of package-lock.json file is to keep track of the exact version of every package that is installed so that a product is 100% reproducible in the same way even if packages are updated by their maintainers.
Maven and gradle don't have lockfiles(by default), and have never really had a serious need for them, because dependency declarations generally don't use ranges.
The central repositories don't allow versions to be replaced, and artifacts are all signed with PGP keys of the developers(although most people don't verify these).
I've never really seen the value in dependency ranges, they make builds more complicated, and bring minimal value.
No it’s the same behaviour for transitive dependencies, if two libraries require different versions of the same transitive dependency, the newer one is chosen.
Deterministic no lock file required.
> if two libraries require different versions of the same transitive dependency, the newer one is chosen.
Unfortunately, this is not how maven works. It picks the version required by the dependency nearest in depth to the project root, breaking ties by first listed in the file. It is deterministic, but it's not what anyone ever wants by default.
(This is called "dependency mediation" if you want to Google it.)
OK, but that has a whole host of other issues such as the dependency that relies on the older version potentially breaking with the newer version. I think BOMs get around this issue, hence why they're so common, but then it just means that you trade one issue for the other and that's why you have different solutions. A Ruby project, for example, doesn't need BOMs.
The elephant in the room here is NPM, and I think the obvious problem there is the culture. I have a tiny app I've been playing with using create-react-app. There are over 800 directories in node_modules. That absolutely dwarfs the number of any other language I've used. Even in a medium sized rails app, you likely have some awareness of what every dependency is. It's just impossible with npm.
One thought I've had to "reboot" the npm culture is to somehow curate packages that are proven to have minimal and safe dependencies, and projects can shift to using those. I imagine there has to be some sort of manual review to make that happen.
In the past I've needed to display a timestamp as something like "n weeks ago" (in a mac app). My first instinct was to write a quick function to do the transform. Then I can tweak it and extend it later to fit my app's needs.
However when I asked the web app team at my company to see their code so I could use the same initial set of intervals, it turns out they use a library to do it. The first instinct of a frontend dev seems to be - even for very tiny, single function solutions - download a library.
To be fair, there are problems that look easy at the first sight, but turn out surprisingly difficult -- and working with time is almost always one of those problems.
npm may be an elephant, but why is it in a room talking about go.mod?
Why not drag APT and the Debian unstable repository in as well? i think install-debian-os also has around 800 dependencies. Debian unstable is vulnerable to supply chain attacks by package maintainers, and Debian stable suffers from lack of volunteers backporting and verifying security patches. Yet, if their numbers increase, the small chance increases a cabal of them attacks the supply chain by pushing and vouching for a fake patch that introduces malicious code ...
The elephant in the room is not some package manager unrelated to the topic. The elephant is trust.
GO argues to mitigate attacks by verifying and locking all dependencies yourself. Which, does not scale.
Debian argues with a proof of work, carefully curating what they publish, and who can publish to fast tracks like the security repo. In theory these people could go insane or be hacked in the same way node module devs go insane or get hacked. But curated publishing mitigates some of the risk.
As far as i know there is only one "third party" GO module repository (actually its generic, supporting both go.mod, npm, and many more) that has a multi-stage system of curation. It promises to integrate lots of tools, global cooperation, configurable policies, an AI and a team of specialists to help with curation. It is also proprietary, stupidly expensive and may not fully deliver on those promises.
The creator of NodeJS talks about how one of the things he regrets is hard-coupling Node to the NPM registry[1]. I imagine this makes it hard to have curated or trusted third-party registries (although note that it is possible to configure private or third-party registries in Node). This is also one of the problems the creator tries to solve in his new runtime, Deno.
I think the NPM organization is completely aware just how dangerous this all is, and is eager to hide it. For example, if you look up an NPM package, it'll list its direct dependencies. But, there's no acknowledgement whatsoever of all the stuff that comes along for the ride.
I'd love to have a well-supported ranking of NPM packages in terms of their dependencies (and their dependencies' dependencies, etc). Knowing the breadth of immediate dependencies, PLUS the depth of the total dependency tree, would give you some inkling of just how much you're taking for granted when you start using a package.
I agree that it would be nice for NPM to show the total footprint of a module, especially if that provides some social incentive to reduce the dependency count.
Something this articles glosses over is that some of these approaches, especially the way 'All builds are “locked”' is achieved with minimum version selection, and “A little copying is better than a little dependency” are tradeoffs against an alternative security model, where transitive dependencies are automatically updated to pick up security fixes.
Part of the churn and noise in the Node.js dependency ecosystem actually stems from security-related issues being noted in a low-level module, and the ripple effects caused by that when a bunch of maintainers have to go around bumping lockfiles and versions.
There is a deeper strategy here with go vs. node; having a standard library maintained by professionals.
I would rather build on a common set of libraries secured by people who are paid full-time to maintain them, and maybe have slightly worse ergonomics, than have a community of libraries that come and go and have inconsistent quality.
This standard library approach yields fewer dependencies, fewer changes over time, and better consistency between projects.
The downside of the standard library approach is that things tend to ossify. While I agree that slower change can be a good thing sometimes, putting things like a HTTP server in the standard library means less experimentation around different ways of doing things, and more difficulty getting performance and other improvements into the hands of language users.
Sure, people can make a third-party module that implements a HTTP server, but the incumbent default that's shipped with the language has an inherent (and often unfair) advantage and a lot of inertia behind it.
I don't really care about the whole "professionals" bit. Sure, I don't want to be relying on something mission-critical to me that's maintained by one person doing it in their spare time. But there is a world of possibilities between that and having a dedicated paid team. Consider, also, that the Go team is only funded so long as Go is important to Google's corporate strategy. Once it isn't, funding will start to dry up, and Go will have to look for a new funding and governance model. That's not necessarily a bad thing, and I'm sure Go would still succeed despite that. But that's kinda my point: this whole "maintained by funded professionals" thing doesn't really matter all that much.
I wish we'd stop trying to make broken languages work. This feels like hill-climbing into the strangest local optimum possible. JS is not the best example of an interpreted language. Wouldn't it be better to put Python in the browser than to put JS on the server? Can't wait for WASM to be a first-rate citizen on the web so we don't have to deal with this anymore.
I don't think the comparison is entirely fair since one of the main attractions of TS is that it runs in the browser. Python can unfortunately not fill the same role right now. So I'd keep that in mind while looking at that ranking. But yes, I see many people like it. Maybe I'm missing something, but it's still too JavaScript-y for me.
> Wouldn't it be better to put Python in the browser than to put JS on the server?
I think that's a categorical "no", because Python isn't an objectively better language than JavaScript. I'm saying this as a Python developer since v1.5 (>20 years).
Yes Node.js ships with what is effectively a very thin standard library for some low level things like interacting with the file system, the process model, some security features like TLS.
> tradeoffs against an alternative security model, where transitive dependencies are automatically updated to pick up security fixes.
One thing to keep in mind is that Go doesn't stop you from updating.
For example, its common to do 'go get -u ./...' or 'go get -u=patch ./...' from your project root to update all of your direct and indirect dependencies.
The built-in tooling & language server give you nudges, and if desired it can be automated via things like dependabot or otherwise.
In practice, it means it is often a slightly slower cadence for typical projects in the Go ecosystem compared to say the Node.js ecosystem, but the upgrades still happen. That slightly slower pace I think has worked out so far, and was a conscious choice[1]:
> Many developers recoil at the idea that adding the latest B would not automatically also add the latest C, but if C was just released, there's no guarantee it works in this build. The more conservative position is to avoid using it until the user asks. For comparison, the Go 1.9 go command does not automatically start using Go 1.10 the day Go 1.10 is released. Instead, users are expected to update on their own schedule, so that they can control when they take on the risk of things breaking.
> transitive dependencies are automatically updated to pick up security fixes
Does Node do this? That seems like an awful idea. People should be manually updating dependencies, never automatically. Stuff like dependabot need to die.
I'm annoyed by the false dichotomy that colors most discussions around package management that there are only two solutions to publishing software packages: 1. a carefully curated professionally maintained standard library, 2. the complete wild west where anything goes. It's not really "false" because this is the reality of how package managers are designed today, but it's false in the sense that it doesn't have to be this way.
You can see this tension in virtually every discussion, users resisting using packages that aren't published in the standard library for fear of attacks and poor quality, and maintainers that resist publishing in the standard library for fear of changing requirements and the appearance of better designs. Sure there are admissible entitlement / responsibility arguments against these respective positions, but that's mostly a distraction because both have a valid point.
The problem is that there's no space for intermediate solutions. We need packaging tools to aggregate and publish groups of packages that relate to a particular domain, and organizational tools to ensure quality and continuity of these package groups over time. This mitigates users' fears and reduces their cognitive load by curating the solution space, and it mitigates maintainers fears of ossification and backcompat hell by enabling them to create new package groups.
I'm saying there's an entire dimension of valid tradeoffs in this space, but the current design trend of package managers force us into one extreme or the other.
I'm unclear from reading your comment if you know this or not... but, what the Go team is describing is an intermediate solution. Not the exact one you describe, but it is intermediate. There is no particular requirement to get into the Go package ecosystem, no gatekeepers, it's all namespaced by the URLs you store your source code at, but between what the proxies do and the way the version requirements were specified, you are also not simply naked to every update someone somewhere pushes.
Please note I'm not claiming it's perfect or that you'll like every aspect of it, I'm just saying it is an intermediate solution between the two extremes.
The original solution is the intermediate solution. You can build an entire useful userspace around nothing but libc and an ssl lib (take your pick between OpenSSL or GnuTLS usually). That is effectively what busybox is. Need to do some heavyweight math in Fortran? BLAS and LAPACK probably have everything you need. You can get really far with a C++ application using nothing but Boost.
For whatever reason, newer language ecosystems migrated away from that in the direction of increasingly smaller libraries until npm practically became a parody of it.
There is no reason you can't have lots and lots of useful functionality packed into a few large, well-maintained, well-packaged, well-vetted and trusted libraries, but you need trustworthy organizations willing to that maintaining and vetting. Historically, that seemed to largely be universities and research labs, where the funding and incentives are a lot different from the weekend warriors and solo devs that dominate open source landscapes today. Interestingly, I think library projects that still have large organizations behind them keep with the larger old-school ethos. Look at the world of ML and scientific computing. NumPy and SciPy are still huge libraries. Same with PyTorch and Tensorflow. QuantLib is an interesting example because it actually doesn't have a single large organization behind it. A bunch of Quants just got tired of doing the same things from scratch over and over and decided to aggregate their work for their common good. But it was 22 years ago, so maybe it was still just different back then and the trend toward small libraries hadn't kicked in yet.
I don't think your suggestion works for Web Development.
Imagine an application that talks to a Postgres DB, a Redis cache, several AWS services (e.g. S3, Lex) and also has to be able to parse excel documents. I've worked on applications like that. That's a whole lot of libraries you'll need to include. But I don't think you should include all of them in some "standard library", most people are not gonna need most of these dependencies.
I agree that left-pad is ridiculous (over in the JVM space, there is apache-commons, which also provides e.g. left-pad but of course it's not the only functionality that it provides), but using only 2-3 libraries isn't realistic either.
This is kind of interesting. In the Linux world, you have package maintainers who (among other things) vet and vouch for the quality of the packages they maintain. I think there are similar things in the Docker ecosystem these days (since Docker really did/does seem to be the wild west).
It could be interesting if there was a similar concept for Go (and/or other ecosystems), except that instead of actually packaging the packages into artifacts (especially with the licensing headache that entails), it could be essentially a registry of verified package versions. So the "maintainers" in this sense are just validating the dependencies and maintaining a list of the approved dependencies (including their versions/checksums) and then automated tooling could be used by consumers ("consumer" here may or may not imply payment depending on whether this hypothetical venture is open or closed) to identify unverified dependencies in the consumer's project.
I'm sure someone has thought of this already--link me to relevant projects if you know about any.
> We need packaging tools to aggregate and publish groups of packages that relate to a particular domain, and organizational tools to ensure quality and continuity of these package groups over time
Whats stopping you? golang.org/x is kind of like that. Theres nobody stopping you from aggregating packages under foo.bar domain and build a reputation for high quality.
Good question! The thing that the standard library has that I don't is version aggregation. The problem is not publishing under a single domain, the problem is publishing under a single version. Publishing a bunch of packages that I claim are high quality doesn't help users decide which versions to use, they still have to make this decision on a package-by-package basis. Note that I may be in the middle of a big redesign and some individually-complete packages use the new design but others don't, so you can't just say use the latest of each because they don't all work together yet. In this case I still want to publish each package individually for people who do want fine-grained control, but I wouldn't publish a new version of my "package group" with these versions because they're not integrated yet. My point is that aggregations of verified-interoperable packages is a separate problem domain and existing tools don't suffice to solve it.
You could sort of do that using a Go module that points to all your other modules. Then anyone who depends on that will get the versions you specify (at a minimum).
But a problem is that they would also download all the modules you point at, whether they use them or not.
To fix that, the package system would need a "soft dependency" where, if a module exists, it must be at least the version indicated.
Good idea with "soft dependencies". You'd also need to make the go.mod RequireSpec version argument [1] optional so it can be overridden by the module group.
require example.com/my-module-group 1.2.3
require example.com/some/module // selects minimum version specified by my-module-group
I also noticed Workspaces [2] which I hadn't seen before. They look interesting, but appear to exist for a different purpose. Maybe workspaces with a bunch of replace directives & some cli tooling could emulate a system like what is described here.
The article, and the comments praising this approach, don’t do a great job of explaining how any of this is substantively different from running the likes of yarn install --frozen-lockfile, or cargo build --frozen.
Here’s the thing: You can argue about being secure by default and encouraging better CI practices. I’d fully agree it isn’t great that one has to know a somewhat obscure flag to get a secure CI build in those environments.
But claiming in what I perceive to be in parts a somewhat grandiose tone to have reinvented the wheel, when you’re just describing a standard approach, can make you sound uninformed.
I think at most there's pride in their own solution, which is not something anyone should object to - it's pretty good. It's better than some other systems, but no point in being specific.
Not doing specific comparisons is likely a deliberate strategy, since it means the blog post is less likely to go out of date, and it avoids controversy if they get something wrong.
Comparisons will need to be written by people familiar with both systems, and they're likely to go out of date quickly.
> Unlike most other package managers files, Go modules don’t have a separate list of constraints and a lock file pinning specific versions.
The weird thing about the Go devs is there is always that little bit of elitism under the surface that I detect in their writing (whether it be colors in the playground, the GC, etc). I spent years writing Go and have now moved to Rust. What I find odd is the Rust team has done (IMO) one of the greater achievements in PL history and yet they seem to not have this elitism thing going on (or maybe I just haven't noticed). Go on the other hand, IMO, made some "interesting" language choices (like keeping null) and they seem to want to be celebrated for it and claim their achievements as new and novel.
EDIT: To clarify, I'm talking about the core Go devs - those that work on stdlib and the compiler
I don't see that elitism in this article. Supply chain attacks are a hot topic right now, so it makes sense for them to make a statement about where the language stands with them. They make compelling points, and they're not calling out specific language or package manager as a comparison.
This is a pretty genuinely confounding response, and I mean that with absolutely no offense intended. There is a tremendous amount of fighting between devs who prefer Go and Rust, and a tremendous amount of elitism as well, truly from both perspectives. Rust gained a reputation for elitism long before Go did; “Rust Evangelism Strike Force” was never meant to be pejorative, and “Rewrite it in Rust” was never meant to be a joke, but it became one anyways. It’s not hard to see why; Rust is genuinely novel in a way that few other programming languages are. It feels the most like the “future.”
But I still like Go a lot. I like Go because of how easy and simple it feels. There is definitely elitism over simplicity, but the elitism I’ve seen and even received from Rust and C++ programmers (…despite that I have been coding C++ forever and do have a few Rust projects as well…) has been pretty much the opposite: Go is too stupid and simple; real programmers need absurdly complex metaprogramming to make basic CLI tools or what have you. Now for what it’s worth, that has cooled down in many regards, and also, Rust is amazing and there’s nothing wrong with advanced metaprogramming. (It’s just another set of tradeoffs, after all. Unquestionably has its benefits.)
However, whereas people who have hated on Rust have often come to see it for what it is (an immensely cool, novel programming language,) Go has received the opposite treatment. People soured on it. Now everyone seems sure the GC latency (which of course is just about state of the art) is simply too much for most use cases. It’s seen adoption in all sorts of places and even been competitive with Rust software in performance, but it is commonly discussed as if Go is inherently obsolete because Rust is a better option in every way that matters. Bringing up Go in certain places often subjects you to ridicule, and I’m not joking. The memory ballast is a favorite among detractors to prove that the language is stupid and bad for production environments.
So when people do try to tout the benefits of Go, it’s routinely discredited and downplayed for some reason. It’s a nice language to use with a stellar standard library, nice tooling, and pretty good runtime performance.
This article doesn’t mention Rust (that I noticed) and Go is still being measured up to Rust in the comments. They both trade blows in different categories, but I truly believe that the fact that Go lacks the novelty of Rust with its borrow checker and language design has caused a lot of people to view it very negatively, and I think that is sad. People loved C for a lot of what it didn’t have. Go is a lot different than C, but for me, the sentiment is very much the same.
I think people see what they want to see. I like Go and Rust, but I find myself going back to Go for various reasons and it feels like every year it leads more and more people to ask for justification that they wouldn’t for other languages. It’s a little tiring.
> “Rust Evangelism Strike Force” was never meant to be pejorative, and “Rewrite it in Rust” was never meant to be a joke
Maybe it's just been such a long time, but my recollection was exactly that: both of these terms were invented by outsiders intending to denigrate the Rust community, and became jokes inside the community as a means of recuperating them.
Okay, I will mention one other thing here:
> This article doesn’t mention Rust (that I noticed) and Go is still being measured up to Rust in the comments.
I agree with this in 99.999% of threads, this happens all the time, and probably shouldn't. However, in this thread in particular with the way that the Go package management story developed, including all of the drama there, I don't think it's surprising that Rust/Cargo get mentioned in comparison.
I won’t comment on Rust Evangelism Strike Force too much; it seemed to be unironically used as a term of endearment at some point, but that could’ve been after its use as a pejorative. At this point, I can’t remember, and frankly, the world is probably better off forgetting.
I understand. In the earlier days of Go package management, it was pretty common for folks to compare it to Cargo. In retrospect, this was probably bad, but it did serve to highlight some pretty damning issues with Go at the time. But I feel they addressed the shortcomings significantly with Go modules, and now it has become much more a matter of taste.
I enjoy Go’s idea of trying to make source control the only source of truth, but I don’t think it’s as well-received as the more tried-and-true approach of Cargo and other centralized package repositories. I suppose time will tell.
To be clear, I have no issue with anyone who prefers Go. I was speaking about the core devs in particular. I would agree that the users of the language definitely go back and forth "trading blows".
Interesting and apologies for misunderstanding. I didn’t read the article as being elitist, though I can see how it reads as self-congratulatory to a degree. Maybe the matter-of-fact way that Go’s developers state its advantages comes off poorly compared to, for example, coming from the standpoint of trying to explain how they got to their current design based on the challenges. Personally, I find articles like this easier to read because they tend to be more terse when written this way versus some other approaches that are perhaps more humble.
Lovely comment. I share your sentiment. Go is truly hated on hacker news these days (and, if you're brave enough to venture there, reviled on r/programming).
I’ve read this article and debated it to the end of the planet. It’s not that the article is factually wrong, but it seems to be under the impression that there is an objective definition for “correct code.”
Of course, there’s degrees. To take an example, filenames. What is a filename? Is it a bag of bytes, a set of UNICODE codepoints, something more elaborate? No operating system agrees 100%. Go handles this by just simply not. It converts stuff to and from UTF-8 and if it doesn’t convert, you can’t use it. This is a limitation of Go programs that use Go’s built-in filesystem support (roughly all of them.)
That decision is a clear-cut simplification of reality. It can matter. However, the fact that it comes up so seldom is a reflection of reality: normally, you are totally able to make the concession that you only handle paths that are valid UTF-8 and nothing else. Go makes this concession on your behalf. It makes many such concessions, and it is documented, though not very well-known, because most developers don’t care. What most developers want is a language that makes sensible tradeoffs so that their programs can remain simple but still reasonable. In TYOOL 2022, I absolutely think it is reasonable that a program may impose valid UNICODE filenames as a requirement.
Rust handles it with OsStr, of course. Now OsStr is a good idea, but it pushes the decision for how to handle the problem further down. On one hand, this is great if you absolutely must handle ridiculous filenames that have essentially arbitrary bytes or old encodings in them, which, to be sure, is a real thing.
The file permissions stuff is similar. If Go is going to present a file permissions interface and then just fudge it, what’s the point of including it at all? Well, in my estimation, the point is that on POSIX systems, you can’t really avoid dealing with file permission bits. In most cases, when a Go program specifies mode bits, its doing so so that on POSIX, the program doesn’t write a file with permissions that don’t make sense, are unusable, or potentially even open up security issues. (You would not want user-uploaded data to go into a file with +x!) On Windows, it can usually be ignored, at least from the scope of the Go software. That’s, again, an opinionated choice. If I needed more granular control over permission, I would probably need more OS-specific code and interfaces anyways, something that is totally doable in Go.
So far Go is oversimplifying things in a very opinionated manner. And in fact, this means that some programs are difficult to write correctly in Go.
But, and here’s the kicker, I don’t usually want to write code that breaks the assumptions that Go makes and requires in its standard library. Even if I’m in Rust, if I want to write some utility code that deals with filenames, often times the string transformations I want to perform on the filename I will want to do in the space of valid UNICODE, because it’s simple and predictable. Even if all I want to do is append a suffix before the file extension, I still would prefer to work in valid UNICODE only. If I’m dealing with say, a broken Shift-JIS filename and want to append a suffix, even if I try to do the right thing and treat it as an ASCII compatible bag of bytes, I could wind up with an even more broken filename as a result, because the state of the filename before the suffix could corrupt the suffix.
The key difference is in perspective. “100% correct” is demonstrably unattainable. You can make more exact abstractions, but for sanity we make assumptions left and right that are “good enough.” You really shouldn’t use the user home directory or user profile directory of a given OS for much directly because it’s honestly a different directory with very different semantics per OS, yet plenty of Rust software (and Go software!) still does this anyways.
Meanwhile, while writing perfectly watertight code in Go is basically impossible, it’s also not a fair benchmark. Nothing does that. Rust is especially prone to stack overflows due to its lack of placement new, and it’s absolutely trivial to do this. Go doesn’t really have stack overflows at all, because it has a dynamically expanding stack. Cargo and its ecosystem has a problem with dependency hell; 200 nested dependencies to install a tool that counts lines of source code is a real thing that actually exists. Go’s ecosystem is definitely in better shape here, for reasons mentioned in the article that I wholeheartedly agree with. There are more things I could say that I don't like about Rust. I could possibly even fill an article about it, but I don't think that it's very productive. I think it's better to just acknowledge that most of these pain points are a direct result of the fact that any decision you make about language design and tooling will have knock-on effects down the ecosystem.
This article is also unfair in that it basically pins Go, a language which intentionally limits scope by taking these concessions, with Rust, a language that happily expands scope to try to cover more edge cases. Both choices are valid and offer different tradeoffs. However, there is a world of different programming languages, and Go is not the first one that implies limitations on filenames, for example. Hell, what happens if you write plain C code that tries to access broken Windows filenames? How much Win32 C code exists that doesn’t handle UNICODE or broken filenames? How many times have you tried to compile some code and had to move it to a directory without spaces in the filename because the program didn’t handle that?
Go is an opinionated language. If you don’t like its opinions, you won’t be happy with it. Orienting this as a correctness issue makes it seem like the Go developers made these tradeoffs haphazardly and without thought. I just flatly disagree.
Go is a distillation of many decades of software engineering experience. The people behind Go (e.g., Russ Cox) have learned from history.
The peanut gallery loves to complain about superficial aspects of Go. Typically these are people with little or no actual experience using the language and tools. They fixate on imagined problems that don't matter in practice.
But anyone who has used Go full-time for a few years is likely to deeply respect and appreciate it.
I've been using Go for a couple of years now and it's funny: I do not love Go, I just work effectively in it. I am not passionate about it, but I recommend it for absolutely all appropriate use-cases.
It doesnt go for intellectual satisfaction, it goes for getting shit done. You have to respect it for being so radically bland.
I haven't used Go full-time, but I have used it on and off for more than a few years. There are certainly things I respect and appreciate about it, but there are also a lot of things that annoy me about it. Some that you might consider "superficial", I consider important. If I'm going to be spending all day in a language, I want the ergonomics of the language itself to work with me, not against me, and Go often does not fit the bill there (for me -- others' opinions are free to differ).
I don't find most of this article to be all that persuasive. Rust, for example, has a separate lock file, which the article derides. That doesn't really make sense, as lock files are also checked into source control, so you get the same benefit that Go touts of go.mod. My threat model doesn't consider having a separate module/package repository to be much of a risk, so I don't care about that point all that much. Admittedly, having source control be the source of truth is just simpler, which is good, but it also means that module publishers can pull versions (or the entire module) for arbitrary, selfish reasons, and then the community is left with a lot of difficulty (there's also a big problem if someone wants to move their code from GitHub to GitLab or something like that). Centralized module repositories can remove this problem if they choose to. The Go Module Mirror appears to be a hack that tacitly admits this problem.
I did find the "a little copying..." bit to be interesting, and I agree with it. With Rust, pulling in a single dependency tends to pull in many tens of transitive dependencies, which I don't like.
"but it also means that module publishers can pull versions (or the entire module) for arbitrary, selfish reasons, and then the community is left with a lot of difficulty"
By default go get will download the source code into the pgk/mod folder. So if a module is pulled by the author, you can just use your copy of the source to fork it.
That's fine if you've already downloaded it, but doesn't help for people trying to pull it for the first time, either as a direct or transitive dependency.
that's what the proxy infrastructure is for: retracted versions won't be selected by default if you're adding it as a new direct dependency, but if you request a version directly (because you need it as an indirect dependency, you have a fresh machine, etc), it will always return the cached code.
For a decade+, people complained that go lacks generics. Would you say those were all people with no experience fixated on an imaginary problem that doesn't matter, or were the complaints valid?
I would say that it is legitimately annoying to have to copy and paste data structures, but mostly doesn't matter, and the intensity of the complaining does not match the intensity of the problem.
People mostly just dont like the expertise of the go devs.
Some of those "superficial aspects" are really annoying.
For example, Go forced version tags to have a `v` prefix in git repos for their dependency system, which broke a whole host of CI tools that expected plain numeric values for release versions. There's a outsized amount of Go-specific special casing for this one seemingly arbitrary decision in multi-language CI systems.
> Go forced version tags to have a `v` prefix in git repos for their dependency system
"Forced" is a bit strong - you can pin any ref, the vX tags just also have some default semver-ish treatment.
> a whole host of CI tools that expected plain numeric values for release versions
Like what? Pure numeric tags are also ambiguous with git refs; this can be worked around by careful arguments, but it means most tooling was already broken when dealing with such things.
If you were one of the people using "release-X.Y.Z" or "rXYZ" I feel for you though.
I've used golang at my employer for several years now, and I came out respecting and appreciating the design decision that have and are going into Java/C#/Kotlin much more given the atrocities I've seen written in golang.
A few years of Golang under my belt and I still hate it. Russ Cox and co. seem incredibly arrogant to me in that they can ignore decades of PL research only to reinvent a bizarre way of achieving what other languages do in a more standard way (package management, error handling), or just adopt that standard super late (generics).
Go has some great qualities and you can make great software with it no doubt. But I find the development of the language frustrating to witness, not inspiring.
C# probably. Though I enjoy Typescript and Python as well.
C# has generics, exceptions, is the birthplace of async/await, has LINQ, an unmatched standard library, a great build system and package management system, is open source, cross platform, fast..
It doesn't compile to a single native binary unlike golang which is a bummer. But 95% of my software ships as a container so this isn't too big of deal for me. The MS-provided base images are really good too.
Agreed. There are some folks very closely associated with other language ecosystems but they spend more time in endlessly critiquing every little thing about Go.
Generally I find if you're writing code like that, your code is too high level. Okay, you've propagated errors, but what is the calling code going to do with those errors? I find large functions like that usually have enough context to handle the errors themselves. Errors, after all, are just conditions with a special name. It's not common to propagate conditions up several layers of code, so why do the same with errors?
Go doesn't have (automatic) backtraces. If you don't wrap errors with a trace or with a custom message you often have no idea where the error came from.
if err != nil {
return mherr.WrapErr(err, "optional context")
}
I just wrote a custom function that adds the stack trace to the error. I also have it setup as a code snippet in VS Code so all I have to do is type "if err" and hit tab. Yes it looks a bit verbose but it adds approximately zero extra work and makes error handling extremely easy by default.
Also, turn on log flags to add line numbers.
log.SetFlags(log.Llongfile | log.LstdFlags)
Well you can actually put many exception yielding statements in the try, unlike with go where your code is full of "if err:=nil{}" after every potential error.
Most of the time when an exception occurs in sequential operations that can fail, you don't really care where it failed exactly, only that the failure was caught.
The irony is that Go does have half baked exceptions (panic,recover) on top of that error as value convention.
Far easier to work with and understand when you dont need to perform a massive disruptive ceremony to handle exceptions. I've been working full time in java for the past 4 years and basically no one handles exceptions because its so cumbersome and bad to read. Not so in Go. Go does it better.
> Far easier to work with and understand when you dont need to perform a massive disruptive ceremony to handle exceptions. I've been working full time in java for the past 4 years and basically no one handles exceptions because its so cumbersome and bad to read. Not so in Go. Go does it better.
No Go doesn't do it better, Go just doesn't give you the choice, from a convention perspective. You have a discipline issue with Java, it is not an issue in the language itself, it's with you and your team.
Ironically I've seen a few Go libs and project trying to re-invent optional types due to the verbosity of Go errors, just like people were trying to roll their own generics with interface {} or code generation before they were added to the core. It's a demonstration that some developers don't like the status quo.
The encoding/json package in the Go standard library uses exceptions for error handling. You absolutely have a choice, but it's a choice you're not likely to want to make because exceptions are named exceptions for good reason: It's for exceptions, not errors. Those are very different things.
But sometimes the trade-offs of overloading something beyond its intent is worth it. In those cases do it. Balancing trade-offs is what engineering is all about.
> No Go doesn't do it better, Go just doesn't give you the choice, from a convention perspective. You have a discipline issue with Java, it is not an issue in the language itself, it's with you and your team.
Its my entire company, and its with the open source ecosystem. Its naive to think that language constructs dont influence dev's decisions. You need to make it easy to do the right thing, not hard, and go makes it easy, which java fails at.
The right thing is to automatically maintain error context (e.g. stack traces) which golang fails at. Furthermore, the vast majority of the time you will have a top level handler that will log the error anyway. golang is just introducing boilerplate for the sake of it, with none of the upsides. Not to mention it also makes it easy to accidentally ignore or overwrite errors, I've seen several instance of that happening over several years of working in golang. Something that would never have happened in an exception based language.
It's quite trivial to write a generic function that handles a particular exception else returns a default value, which addresses the above scenario.
final var x = getOrDefault(C::foo, Something.class, quux());
return x.bar();
And now with pattern matching in Java, it's trivial to write something similar to Rust's/Scala's `Result<T, E>`/`Try<T>` types and be explicit about all exceptions.
I haven't compiled the following, but it's along the lines of:
final var x = switch (foo()) {
case Ok(var r) -> r;
case Error e -> quux();
}
return x.bar();
Neither of those is equivalent to what I wrote (in different ways!), and the first one doesn't even work because of type erasure - you can't write generic functions over exception types. You're also focused way too much on the specific lines I wrote and not the general pattern that leads to exception over-handling.
> trivial... trivial
Rethink your use of this word.
Anyway, alternative proposals don't really matter - I could also offer patterns for Go which make accidentally ignoring error values more difficult. We're talking about actually extant code, and nobody writes Java (or Go) like that outside of forum arguments.
That you've marked the function `throws E` is an admission it doesn't really work; that's the one exception type it definitely doesn't throw. (Default handling is also still different than the starting point.)
Please just stop trying to show off how much you know when I'm trying to discuss how programs in general are actually implemented. You're very clever, but part of why you keep getting it slightly wrong is because probably you don't even write error handling like this day to day.
> That you've marked the function `throws E` is an admission it doesn't really work
Correct, I didn't spend a lot of time refining it. I suppose the only way to make it work is to remove `throws E` from the signature and make it `throw new RuntimeException(e)`, and be similar to C#/Kotlin/Scala.
My last JVM job used Scala, and we wrote things that are similar to
val x = foo() match {
case Success(res) => res
case Error(e) => return quux() // Returns from function scope
}
> Its my entire company, and its with the open source ecosystem. Its naive to think that language constructs dont influence dev's decisions. You need to make it easy to do the right thing, not hard, and go makes it easy, which java fails at.
Go makes it pretty easy to ignore errors, it's even worse than Java. An uncaught managed exception will not compile, and an uncaught unmanaged exception might terminate your program.
_ := YieldsAnError()
will will never terminate your program. Go makes it pretty easier to do the wrong thing.
> An uncaught managed exception will not compile, and an uncaught unmanaged exception might terminate your program.
While it's nice that uncaught exceptions stop the program from continuing with invalid state, it'd be strictly better to not have them in the first place. That's what languages with explicit error returns provide - can't have uncaught errors if you're forced to explicitly deal with every possible error. The resulting software is way more robust that way.
> _ := YieldsAnError()
This is a deliberate decision to ignore the error - the programmer needed to explicitly type this out. As such, it's not a problem at all. Even if it was, writing a linter to detect it would be trivial.
The real issue with Go's error handling is that you are allowed to implicitly ignore errors in certain cases. Most of them are caught by linters, but some edge cases remain and not compiling is better than needing a linter in the first place.
> This is a deliberate decision to ignore the error - the programmer needed to explicitly type this out. As such, it's not a problem at all. Even if it was, writing a linter to detect it would be trivial.
So is not handling Java exceptions is deliberate, and again, Java forces you to handle managed exceptions.
> While it's nice that uncaught exceptions stop the program from continuing with invalid state, it'd be strictly better to not have them in the first place. That's what languages with explicit error returns provide - can't have uncaught errors if you're forced to explicitly deal with every possible error. The resulting software is way more robust that way.
Errors as value in Go are purely a convention, nothing precludes you from doing the exact same thing in Java, or JavaScript or any other possible language that has exceptions. Furthermore, Go has (stupid) exceptions, it's called panics.
> So is not handling Java exceptions is deliberate
That's the thing, there's no visible difference between a deliberately ignored exception and a purposefully ignored one. The code looks exactly the same:
foo()
Did I purposefully ignore the exception or not?
Can `foo()` even throw an exception?
There's no way to know.
Once you've programmed with explicit error returns, coming back to exceptions is unnerving. Anything may or may not throw at any time and there's no way to know.
> Java forces you to handle managed exceptions.
I'm referring to the far more common unchecked exceptions. Java's checked exceptions are a whole another bag of demons I don't want to get into right now.
> Errors as value in Go are purely a convention, nothing precludes you from doing the exact same thing in Java, or JavaScript or any other possible language that has exceptions.
Thing is, conventions matter. A lot. The proof is in the pudding:
- what's the ratio of exception-based Java code to error-based Java code?
- what's the ratio of error-based Go code to exception-based Go code?
Somehow both ratios end up close to 100%, despite neither language forcing their users to use a specific error handling method.
That's because there's space between forcing something and completely ignoring it. I often call it "nudging". Languages often do, and should, nudge their programmers into certain ways of programming that fit their philosophy.
> Furthermore, Go has (stupid) exceptions, it's called panics.
I assume by "stupid" you mean that panics are underfeatured compared to "real" exceptions. That's intended, you're not supposed to catch panics in the first place. They're only there to address runtime issues that are almost always unrecoverable: running out of memory, out-of-bounds array access, regex compilation failure, etc.
You're arguing a technicality, which is a waste of time. We should focus on reality, not imaginary theoretical issues. In reality, that never happens. And people even develop linters (frankly unnecessary) to guarantee it, if you're really paranoid.
This is a social problem, influenced by language design. You need to be thinking about the way humans actually behave in practice, not how they theoretically might behave.
In practice people do not accidentally skip errors in Go. In practice people do not handle errors at all in Java. This is the responsibility of the respective cultures and language designs.
> You're arguing a technicality, which is a waste of time. We should focus on reality, not imaginary theoretical issues. In reality, that never happens. And people even develop linters (frankly unnecessary) to guarantee it, if you're really paranoid.
Because you aren't? you're projecting.
> This is a social problem, influenced by language design. You need to be thinking about the way humans actually behave in practice, not how they theoretically might behave.
Sure, your problem with Java is also a social issue that has nothing to do with the language.
> In practice people do not accidentally skip errors in Go. In practice people do not handle errors at all in Java. This is the responsibility of the respective cultures and language designs.
This isn't Java's "culture" to not handle errors at all. Go's conventions imposed in its std libs are certainly not a superior model by any measure as I previously showed.
Edit: Answer to your follow up since HN ratelimits flamewars
> This is java's culture, and I assert it from experience. It is a result of the language design (and communiques from language authorities).
Your appeal to authority is a logical fallacy, you have never demonstrated that it is Java culture. I say it isn't Java culture the same way you establish an unfounded claim.
> You only showed that it was possible to misbehave in Go, which is totally uninteresting. You have failed to show that your theoretical problem is a practical problem. This is a perfectionist argument.
You don't get to say what argument is deemed interest and which isn't in a discussion.
Ignoring Go errors is as much as a practical problem that not handling Java exceptions. The difference being that Java does force the developer to catch managed exceptions, Go doesn't care, it's purely a convention. A language construct to deal with errors baked into a language is in theory and practically superior to random conventions established by a vendor. PERIOD.
Whatever experience you had with Java isn't representative of Java's culture at all.
>Sure, your problem with Java is also a social issue that has nothing to do with the language.
Quoting myself: "This is a social problem, influenced by language design."
>This isn't Java's "culture" to not handle errors at all. Go's conventions imposed in its std libs are certainly not a superior model by any measure as I previously showed.
This is java's culture, and I assert it from experience. It is a result of the language design (and communiques from language authorities).
You only showed that it was possible to misbehave in Go, which is totally uninteresting. You have failed to show that your theoretical problem is a practical problem. This is a perfectionist argument.
Having worked on golang code bases for several years now, I assure you it's not a theoretical problem. It's not fun to have ignored or overwritten errors that keep the program going as if nothing is happening.
If you overwrite or ignore your variables you're going to have a bad time. Period. That's not unique to errors. I'm not sure you've said anything to convince us that making mistakes around errors alone is anything more than theoretical.
Beyond defining an interface named error (early versions didn't even offer that), Go does not have a concept of errors, so I'm not sure it is even possible for there to be a problem specific to errors. Certainly Go could do more to help you with correctness in general.
Because errors are just return values in golang, and it's not possible to write a linter to guarantee that they're handled, there's always going to be the possibility of them being overwritten by accident. Not to mention due to the lack of composability, you end up with even more hacks like what gorm does, making it even easier to mishandle errors.
> Go does not have a concept of errors
Exactly the problem. There needs to be a notion of error handling in the language.
> Because errors are just return values in golang, and it's not possible to write a linter to guarantee that they're handled
That's true of all types, though. Nothing specific to errors applies there. Spend enough time in Go and that's going to bite you, even when the error interface is nowhere to be found. You've still not made clear what's special about errors that makes the concern about errors, not all types, more than theoretical.
> There needs to be a notion of error handling in the language.
Why? The computer doesn't have a concept of errors either. All the computer offers is different states. What Go could offer is more guarantees around ensuring that you've handled different states correctly, but that has absolutely nothing to do with errors specifically.
Now, humans have a concept of error. Of course. We classify specific computer states as being errors. But since the language and computer treat errors and other states as being the exact same thing, why would that human classification of error magically lead you to start making mistakes that you wouldn't make with other states? That doesn't make any sense.
Here's a thought: Stop classifying those states as errors. Call them "happy fun times". Then you won't have whatever mental hangups around errors that are causing you problems specific to errors. Because, from a technical perspective, errors don't exist. They are entirely a human construct. There is nothing about that human construct that should be tripping you up in the tech.
No, there should not be any special concept for errors. Errors are not special. As far as the tech is concerned, errors don't even exist. What Go could do is improve on state management generally. Applicable to not only states you've decided to call errors, but all other states as well. Anything that tries to improve state management for only what you've decided to call errors still leaves all of the exact same problems dangling for every other type. That is poor design and doesn't actually solve the problem.
> You've still not made clear what's special about errors that makes the concern about errors, not all types, more than theoretical.
The way errors are handled in golang, its possible to ignore them accidentally. This doesn't happen with other types because you don't keep constantly overwriting the same variable over and over (immutability is generally good, another thing that golang lacks), increasing the likelihood of ignoring an error.
E.g.
a, err := foo()
if err != nil { ... }
b, err := bar(a)
c, err := baz(b)
A linter will complain if b or c are unnused, but it cannot complain that err is unnused, because it is used and has been declared before. Whats worse something like this
a, err := foo()
if err != nil { return err }
b, errBar := bar(a)
if errBar != nil { return err } // oops
> The computer doesn't have a concept of errors either.
This is exactly the same reasoning that golang authors gave about why it doesn't have a notion of optional or non nullable pointers. To the computer, a pointer is a pointer. This isn't the way to think if we want to make reliable and readable programs. Computers don't have notions of methods or inheritance or even functions either, everything is a jmp of some sort.
The entire field of programming is to make it easier for humans to reason about code, OOP, functional programming, etc. Otherwise, we'd all be writing assembly or machine code.
Errors are special because they need to carry certain state about the program when an error happened (e.g. stack trace). golang errors are basically strings, which is why people invented frameworks to capture stack traces in golang errors. On the code bases I worked on, this building of the call stack is either done manually (by wrapping errors) or by concatenating strings, or by logging errors everywhere and capturing the stack at the log site. Quite horrible experience overall and just keeps polluting the code with boilerplate the makes it even less clear what's going on.
I've experienced the issues that come out of golang's overly simplistic design. The overly verbose code that is hard to see what its doing at first glance, the non-composability of errors, the mishandling of errors, etc.
Other than exceptions, languages like Rust have it much better than golang. You still get to have explicit error handling, but with much superior ergonomics that make them much more difficult to mishandle.
> you don't keep constantly overwriting the same variable over and over
Except when you do...
> Whats worse something like this
Yes, this could be catastrophic.
dir, file := path.Split(fileToKeep)
dir2, file2 := path.Split(fileToDelete)
if file2 == "x" {
removeFile(file)
}
What does it have to do with errors?
> golang errors are basically strings
That's not true at all. The error interface asks that you provide a string representation of your error when called upon (i.e. the Error() method), but you don't want it to be a string underneath. Further, you don't even need to use the error interface. Not all Go functions, even in the standard library, return errors declared using the error interface.
> Errors are special because they need to carry certain state about the program when an error happened (e.g. stack trace).
A stack trace is useful when you have an exceptional circumstance, but Go's exceptions (panic/recover) already provides that for exceptions. We are, however, talking about errors, not exceptions. Why would you ever need a stack trace when the network is down, for example? There is nothing unusual about the network being down. It's just another happy state along your happy path and logically dealing with it is no different than branching on the user inputting 1 vs. 2. Should languages also have special constructs for dealing with that? Of course not.
> I've experienced the issues that come out of golang's overly simplistic design.
No doubt. I have made clear that Go could improve here. None of those improvements are specific to errors, though. The idea that errors are special just doesn't jive. Every problem you encounter with errors will also be encountered with other types sooner or later.
> languages like Rust have it much better than golang.
Rust is a good example of how errors aren't special, though. The constructs you may use for errors in Rust are built upon lower level features that help with state management generally. Mentioning Rust here is a is a bit strange seeing as how it contradicts the entire premise you're trying to push. The features of Rust that can help you with errors can also help you with problems of other types. Rust is a good example of how good design can work to solve the problem generally, not just for errors like you want to do.
To really get to the meat of this, your entire comment sums up to say that Go could do more to help you with state management. We already established exactly that in previous comments. Where do you think errors specifically begin to come into play?
The fact that the former scenario is much more likelier to happen in golang due to its error handling. The former can be caught using unit tests, and should stick out more in languages like Java with scoped try-with-resources blocks that limit the scope of lifetimed variables (another bad design in golang where defers are function scope, not local scope limited, introducing unnecessary runtime costs, while also being less useful in practice).
At the same time, you don't want to be writing unit tests for every simple scenario. Some people who use dynamic languages argue that you just type assert everything in unit tests and you should be good. Obviously, that's not how things work out in practice. Similarly here, because of the way errors are handled in golang, the same pattern is everywhere in the program, and it is extremely tedious to write the same unit tests over and over everywhere, while making sure to reach a certain level of coverage.
> Not all Go functions, even in the standard library, return errors declared using the error interface.
Do they just return integers then like C?
> Why would you ever need a stack trace when the network is down, for example?
Because it's still important to know where you are in the program when the network broke, in order to make sure that recovery happened correctly for example. Seeing a "network is down" error in the logs is useful, but more so is knowing exactly what I was doing (what if I wasn't expecting to access the network in a particular path? etc.)
> Every problem you encounter with errors will also be encountered with other types sooner or later.
Practicality matters, otherwise C is all we'd ever need, and we wouldn't be having constant CVE's etc.
> Mentioning Rust here is a is a bit strange seeing as how it contradicts the entire premise you're trying to push.
Not really. Rust errors compose nicely, and you're explicitly forced to handle them (unlike golang's), and are much harder to accidentally swallow or ignore compared to golang. This scenario is repeated many times when you compare golang to other better designed languages. The language constantly takes the easy way out so to speak, making the compiler implementation simpler, while pushing complexity onto the user. Furthermore, as I pointed out, there's nothing preventing us from using the same approach in Java, now that it has sealed types and pattern matching.
That is not possible in golang due to its lack of support of all those constructs. Now with generics, some people are starting to use some similar approaches, but they fall flat on their face because (1) golang doesn't have those constructs as I mentioned, but also (2) the way generics have been implemented in golang (similar to the rest of the language) are simplistic (you can't have types on member functions), leading to further abominations and verbose code.
Java's checked exceptions are not much different actually. They still require you to handle the error either by try/catch or by declaring that the method throws the same exception, or a superclass of it.
I also remembered another shortcoming of golang error handling that I've seen several times in real code bases, being forced to check both the error and the other return value to make sure that things are working. Yes, a properly written program shouldn't need to do that, but reality doesn't care. And what's ironic is that golang was supposed to be designed to support "programming in the large" (another unverified claim that is contradicted by reality). The fact that it opens these doors is indicative of the mentality that went into designing it.
> At the same time, you don't want to be writing unit tests for every simple scenario.
Yup, exactly. In a perfect world you would, but the world ain't perfect. Sooner or later you're accidentally deleting the wrong thing because you didn't test it/didn't test it correctly.
And for what reason? A language can guard against that kind of mistake quite well.
So, that still leaves us wondering why you don't find it advantageous for a programming language to be able to deal with these kinds of problems unless the problem is related to an error? If you have a solid foundation the problem with errors you are trying to show would disappear. Go only has problems with errors because it also has problems generally.
> Practicality matters, otherwise C is all we'd ever need, and we wouldn't be having constant CVE's etc.
Agreed. Which leaves it to be insane to only want to fix the problems with errors when everything wrong with errors is also wrong with every type. If you fix the error problem properly you've automatically fixed it for all types. Why on earth are you suggesting that you'd purposefully make the fix harder just to ensure that it only fixes errors?
> Rust errors compose nicely, and you're explicitly forced to handle them (unlike golang's), and are much harder to accidentally swallow or ignore compared to golang.
This isn't a feature of specialized error handling, but Rust's overall design towards helping with general state management. Rust alleviates the same problems for all types. Rust yet again shows us that errors aren't special and don't matter. Provide a good foundation for state management and managing error state becomes easy by virtue of just being yet another state. You are able to cleanly deal with errors in Rust because it took care to get the problem right generally, not because it took care to worry about errors at the expense of everything else.
Go could, like Rust, do more to help with general state management. It would be insanity to try and only do that for errors, though.
> You have a discipline issue with Java, it is not an issue in the language itself, it's with you and your team.
Only if you consider the "language" to be only the language itself, i.e. syntax + semantics. When people refer to a "programming language", they almost always mean the whole package, i.e. the language itself, the community, package ecosystem, tooling, conventions, etc.
If there is widespread usage of an incorrect pattern, then it is the failure of the language for not addressing it. For example, C's lack of package/build management.
The experiences with exception-based languages others in this thread brought up mirror my own. No thinking about error handling whatsoever, empty `catch` blocks, dumping stacktraces in lieu of error messages, etc.
Needing discipline to prevent those is exactly the problem. The language is what should guide you towards a better way, in this case explicitly returning errors.
> Only if you consider the "language" to be only the language itself, i.e. syntax + semantics. When people refer to a "programming language", they almost always mean the whole package, i.e. the language itself, the community, package ecosystem, tooling, conventions, etc.
You're making a lot of assumptions based on absolutely no factual data, only your alleged experience. I can dismiss all your arguments easily as anecdotal.
As I said earlier Go suffers from having a bad exception system itself (panics/recover) and on the top of that go makes it a breeze to ignore errors:
_ := YieldError()
You're basically claiming that teams are magically more disciplined just because they use Go. That's of course complete bullshit. An undisciplined team will ignore errors and exceptions the same way. There is nothing in Go that compels anybody to handle errors, just like there is nothing in Java that compels anybody to ignore exceptions.
Go errors are purely a convention, Go doesn't have anything in its syntax that forces error as value, it's just what the std library uses. So when talking about Go errors, they are purely talking about a convention, with absolutely no syntax to compel people to follow or enforce that convention. It requires thus even more discipline. Do you really believe your undisciplined programmers that ignored Java exceptions because it's verbose to deal with them are going to magically adopt an even more verbose paradigm? Of course not.
At least one can respect Go for being consistent, even if requires a little extra manual labour. The languages that require you to manually bubble up some types, but not others, are just bizarre. Like, they're onto something neat, but why stop halfway? I don't want to put in the manual work for any type. If I've decided the manual effort is worthwhile, I don't want weird exceptions to worry about. Pick a lane, programming languages.
Dependencies being immutable and identified by hash was such an obvious thing 20 years ago. The problem was fitting that into the flow of these crappy UN*X based build systems where to do any mundane task you need to fiddle with files and encodings and semi documented folder heirarchies and use a CLI tool to change other stuff that is too cumbersome to encode directly into files / text. The most obvious concrete instance of this being that it's cumbersome to import a dependency by hash (in Java for example, you really want hashes everywhere instead of reverse TLDs) but obviously better tools (such as a structural code editor) solves this perfectly. It's also annoying every time a company takes a nano step asymptotically towards these proper solutions and spams their crap and everyone buys it, but that's to be expected when your field is broken industrially and academically and even in hobbiest communities.
I feel like much of the current software complexity is mostly caused by the pain that is compiling C dependencies and the outdated Unix build and configuration tools.
Having dependencies not change is literally trivial. Just link to them by hash. Async/await is a pragmatist garbage hack, and does not fit in the archetype you are trying to name.
Off topic but: Since I assumed this was about physical supply chain attacks (where someone nefarious will either intercept your package to install custom firmware etc. or even change the physical device in some way) - does anyone know where I could find a good guide on mitigating such attacks?
Worth noting is that, while go build doesn't run arbitrary code, go generate does:
package evil
import "fmt"
// the echo is intentional, in case someone actually tries this for some reason
//go:generate echo rm -rf /
func PretendGood() {
fmt.Println("I am good")
}
When they say fetching and building code doesn't execute it, that's specific to go get and go build. There's no guarantee that every go subcommand is safe. This is pretty obvious if you know how go generate works and it isn't a flaw of the language, but if I were new to go, this is the kind of article I'd read but still not understand exactly what was safe and what wasn't.
One downside of such pinning of versions is that if one of your transitive dependencies has a security vulnerability affecting your package, your package will remain affected until you update.
As I understand it dependencies-of-dependencies are fetched from the config provided by the library that includes the dependency, in the case of a transitive dependency, everyone between you and the vulnerable package needs to update, and they need to do so in order (i.e. if you depend on A, A depends on B, B depends on C, and C fixes a vulnerability, then first B, then A have to update in order for you to pull in the change).
I feel like everyone is beating around the bush here: NPM is a garbage fire, from the interface to the tooling implementation to the theory to the governance. We can talk at each other back and forth about theoretical benefits, and "friction" vs. "usability" or whatever, but NPM has been and continues to be an unmitigated security disaster. The module proxy could have a package takeover a month for the next three years and still not even come close to the ridiculous shit that has happened on NPM.
One problem: unless you have an endless amount of time, money, patience, and...did I mention patience? that just isn't a viable approach for anything but the smallest of hobby projects
So we just don't build GUI applications unless we understand all of the nuances of layout, text rendering, graphics programming, etc sufficient to implement it ourselves (and without the bugs that even domain experts have introduced but which have been found and fixed over time in libraries)? Or maybe we just say "fuck users who speak languages that aren't easily expressed in ASCII"?
There's a reason libraries exist. It's not like they were the default state of computing and no one has tried to write applications without them. On the contrary, we tried to build applications by writing everything ourselves, but that doesn't survive encounters with the real world. And "well if you can't do it yourself, you don't need it" (which may or may not be your argument, I genuinely can't tell) is just technologically regressive ideology.
Software programmer usually need to solve specific task, not worlds problems. If you go too generalized you will need 10x, 100x or 1000x more amount of time and code.
Look at Big tech, this is what they are doing, they employ thousands workers that write millions of lines of code every day, only to make it work for every case in a world. And still can't compete with specialized solution.
> Software programmer usually need to solve specific task, not worlds problems. If you go too generalized you will need 10x, 100x or 1000x more amount of time and code.
Yes, I accept this is true, but your earlier claim was much more specific: that using dependencies at all makes things worse. I gave you a specific example (GUI libraries) but you completely ignored it. How does your 0-dependencies theory survive an encounter with that basic example?
No, but if your GUI involves displaying text in multiple languages (i.e., virtually everything), you now need to become an expert in text rendering (which implies a significant breadth of knowledge in linguistics, graphics programming, constraint solving systems [for things like word wrapping], etc).
Except KISS is subjective, as are most things. Please don't take this the wrong way, but anyone who touts any one, true "religion" in sw development I have learned to take with 6 grains of salt. There is no one magic way or solution. Everything is trade offs and I've learned this over the last 30 years through trying every new magic elixir that was going to "save us all".
That said, most sw projects are determined by business need. I can't think of a single one I've ever written that hasn't needed dependencies or would have been viable had I not used some. Going to a theory level, there is no reason to think the stdlib is magically immune to the issues of a very popular dependency either. Shun absolutes and make the correct trade offs based on your business goals.
It may not be quite a URL, but it contains the URL. It definitely more than just "sort-of resembles a URL". See how the module path to URL lookup is done here: https://go.dev/ref/mod#vcs-find
I understand your meaning, but that link supports my claims. The package path helps the Go tool infer the actual URL, but it doesn't contain the URL itself (e.g., the actual URL has a scheme/protocol component and potentially a `go-get=1` query string argument which don't exist as part of the path). This is what I meant when I said it "sort-of resembles a URL", but I understand from this conversation how that wording wasn't clear.
Fair. Strictly speaking it doesn't even "contain" a URL. But I think in the context of this conversation it acts like a URL -- it allows the Go tooling to fetch code from an arbitrary domain and path on the internet.
I guess that's fine so long as you're covered by the standard library and/or are willing to reimplement a lot of stuff yourself, but that's a significant trade-off you're asking.
It sounds defensive but Go stdlib is all you need. I believe I'm qualified to say that as last year I challenged myself to only use stdlib, out of several languages I used over the course of the year on big projects, Go was painless and that was a unique experience. So far this year I haven't seen much need to add libraries to my work because everything is already within grasp.
>> The only commands that will change the go.mod (and therefore the build) are go get and go mod tidy. These commands are not expected to be run automatically or in CI, so changes to dependency trees must be made deliberately and have the opportunity to go through code review.
GO doesn't do jack shit to mitigate supply chain attacks. Version pinning with checksum and that is it. But what could go do? Solve supply chain attacks as a language feature? That doesn't even make sense.
Application developers using Go must prevent supply chain attacks against their applications. So go get some SAST for your pipeline.
Sure there is truth in saying: always verify your dependencies (and their dependencies) yourself with a code review on every update. But you should not do that alone, so let's talk about collaborative vulnerability management. (there is more to sast than vulnerability assessment, but we have to start somewhere)
Let's say repositories that publishe go modules should also publish a curated list of known vulnerabilities (including known supply chain attacks) for the modules they publish. This curation is work: reports must be verified before being included in the list and they must be verified quickly. This work scales with the number of packages published. And worse, modules could be published in more than one repository, module publishing repositories can be different from the modules source code repository, and vulnerability lists can exist independent from these repositories - so reports should be synced between different list providers. Different implementations and lack of common standards make this a hard problem. And implicit trust for bulk imports could open the door for takedown attacks.
There is an argument that vulnerability listing should be split from source and module publishing: each focusing on their core responsibility. For supply chain attacks this split in responsibilities also makes it harder for an attacker to both attack suppliers and suppress reports. But for all other issues it increase distance as reports must travel upstream. And it creates perverse incentives, like trying to keep reports exclusive to paying customers.
To pile on the insanity: reports can be wrong. And there are unfixed CVEs that are many years old (well ok maybe not for go... yet). Downstream there are "mitigated" and "wont-fix" classifications for reports about dependencies and many SAST tooling can't parse that for transitive dependencies.
Really, supply chain attacks are the easy case in vulnerability management, because they are so obviously a "must-fix" when detected. (and to please the never update crowd: for a downstream project "fix" can mean not updating a dependency into an attacked version. you are welcome)
Long story short: go get some SAST in your pipelines to defend against supply chain attacks. Don't pretend pinning the version and half-assing a code review when you update them is actually solving supply chain attacks. Don't tell me everyone who uses go can find a sophisticated data bomb or intentional rce in some transitive dependency of some lib they update to a new feature release. And don't give me some "well if its transitive then the lib dev should have." Should have doesn't solve shit.
Tools like these help you sort out supply chain attacks that other people have already found, before you update into them and push them downstream. Collaboration is useful. Sure you are still left with reading the source changes of all dependency update, because who knows, you may be the first one to spot one, but hey, good for you.
node, python, ruby...etc including go modules can lock their dependencies, Go really wins when it can be wrapped in a reasonably-sized one binary, nothing beats that in deployment, for all the others you have to pull in lots of packages into the target system
I don't like the authoritative tone of the article. Especially
given the fact that author "conveniently forgets" about
go mod edit
and
go work
both of which are deliberately designed counter-mitigations, i.e. they exist
to poke small holes in the pin-everything wall.
I agree with the spirit of the message though, the surface is much smaller
with Go and it shows much planning went into that.
> Unlike most other package managers files, Go modules don’t have a separate list of constraints and a lock file pinning specific versions. The version of every dependency contributing to any Go build is fully determined by the go.mod file of the main module.
I don't know if this was intentional on the author's part, but this reads to me like it's implying that with package managers that do use lockfiles a new version of a dependency can automatically affect a build.
The purpose of a lockfile is to make that false. If you have a valid lockfile, then fetching dependencies is 100% deterministic across machines and the existence of new versions of a package will not affect the build.
It is true that most package managers will automatically update a lockfile if it's incomplete instead of failing with an error. That's a different behavior from Go where it fails if the go.mod is incomplete. I suspect in practice this UX choice doesn't make much of a difference. If you're running CI with an incomplete lockfile, you've already gotten yourself into a weird state. It implies you have committed a dependency change without actually testing it, or that you tested it locally and then went out of your way to discard the lockfile changes.
Either way, I don't see what this has to do with lockfiles as a concept. Unless I'm missing something, go.mod files are lockfiles.