First of all:
Thank you, yarn, for helping the community see the naked emperor. Deterministic builds by default are such an obvious (in retrospect) core requirement.
Couple questions:
Question 1:
Does anyone else who's been around more than a couple years share my view that Yarn : NPM :: IO.JS : Node?
IOW: healthy competition, catalyst for necessary change, ultimately a bridge or stopgap.
Question 2:
Any good comprehensive writeups on best practices?Committing lockfiles is a no-brainer. But what about globals? Per-node-version seems logical, but there are also semantic problems with "global" packages vs concept of executable binaries. I'd love to see a strong writeup outlining and defending a standard approach.
When io.js merged back into node.js the following happened:
1. Node.js was abandoned and replaced with io.js.
2. Io.js was relabelled as node.js.
3. The node.js project was put under open governance via the creation of the Node Foundation.
This doesn't really compare to npm:
* The npm-cli's name is using npm Inc's trademark: giving it away would leave the company with no name or create unwanted ambiguity between npm Inc the company and npm-cli the (then) community-owned open source project unaffiliated with the company.
* Yarn does not share any source code directly with npm-cli, it's a complete reimplementation of a similar feature set. It's currently backed by a mirror of the npm registry but that's the extent of the overlap.
* The Node Foundation already exists and Yarn is already owned by the community. I could see Yarn joining the Node Foundation and the Node Foundation replacing its bundled dependency on the commercially owned npm-cli but even then npm Inc has nothing to contribute to the process.
Personally I would much rather see Yarn join the JavaScript Foundation because it speaks to the broader appeal of Yarn outside the traditional "Node community" and the politics involved in the latter and its close ties to npm Inc.
FWIW I consider the close ties between the Node project and npm Inc nothing more than a historical accident that has resulted in a conflict of interest that is overdue to be resolved by dropping npm-cli from the official releases.
Now that yarn is no longer distributed using npm (although it's still possible to install it that way) that seems more realistic than ever.
Back when the npm registry was really struggling with reliability (I believe because it was secondary to profit for Joyent) I was saying we needed a decentralized registry solution.
But right when that got some upvotes on r/node, within a few days isaacs had npm inc. going. And within a few weeks or months, the npm registry was pretty rock solid. And I have not had a single registry issue in forever.
And npm 4 works well, and I'm sure npm@5 works even better.
I still think it makes sense to have a decentralized repository and I guess its nice to have a non-commercial alternative to npm. But for me its not important anymore because npm is working really great now.
Reliability sure was tough in those days. npm wasn't secondary or tertiary or any-ary to Joyent. It was my nights and weekends project the whole time I was there, and IrisCouch generously donated infrastructure, which we pushed to its very limit once Nodejitsu acquired them.
Something like IPFS could perhaps be used for this. I think the biggest issue is probably access control. I.e. just releasing packages addressed by content isn't enough, because people generally depend on @whatever/left-pad and not 5b055a0b42f1c or whatever the hash may be. Real estate in a global name space means someone gets there first, so you'll need to consider how to deal with name squatters and the likes. With a centralized source like npm it's as easy as getting in touch with support (they're very good) but when no one knows the name space everyone owns the name space – makes it hard to make things "nice".
So I guess what I'm saying is that just storage isn't the tricky bit to distribute really, but having a global namespace is I guess.
well, you could depend on left-pad@5b055a0b42f1c, and the name doesn't really matter. Normally you would include that as require('left-pad') but if you have multiple versions, I don't see why require('left-pad@5b055a0b42f1c') couldn't be used. There is a tool for doing this in Golang already[0] and I've also done some experiments to get this to work in JavaScript[1]
Go does something like this, by referencing packages through Git url. Of course, 99% of packages you install are through Github, but that's not Go's fault. It has downsides, but its really a great and extensible system.
Yarn does not run a mirror of the registry. registry.yarnpkg.com is a pass-through domain to the npm registry. It allows them to collect stats about yarn usage but is not a mirror.
NPM currently doesn't provide a standalone distribution as far as I can tell, but presumably they would offer one if they no longer had the luxury of being bundled with Node.
Since changes generally don't happen overnight I would expect a transitional period where Node still bundles npm-cli but npm Inc has the time to prepare a standalone distribution before yarn replaces npm-cli. Additionally downstream channels could decide to provide legacy packages containing both.
Answer 1: Yes but they're not exactly analogous and I don't see them merging. Yarn is a complete rewrite, and the compatibility "surface area" is much lower than Node itself so there's not much pressure for them to merge. Yarn has been working essentially flawlessly for me, so npm-cli is really going to have to surpass yarn in some way for me to switch back.
Answer 2: I don't use global packages often. For binaries I generally add a "scripts" entry to package.json for common tasks, which automatically adds "./node_modules/.bin" to the PATH. For one-off tasks I just run "./node_modules/.bin/executable"
Git dependencies with semver support seems to pretty much leapfrog yarn. One of the few reasons for a private npm package is removed with this feature.
It's more accurate to say that npm v5 : npm :: io.js : node.
Same team, performing a big refactoring and significant improvement and modernization of some bits that were outdated and difficult to improve without a controlled demolition and rebuild.
The analogy breaks down, of course, because the governance didn't change, the project's relationship to its corporate backer wasn't standing in the way of progress, etc. (Except, I guess, that it might have gone faster if we'd had more money to hire more devs? But some of this was just a slog through a very old codebase with a lot of scar tissue, and it's hard to speed that up by putting more hands on it.)
Everything in npm 5 was literally planned years in advance. When we have this many people depending on a thing, we have to be careful about how we make drastic changes. Yarn was a strong signal from the community that we were on the right track, but it only seems like a "catalyst" when seen from the outside. Correlation is not causation, even when it lands first.
"Deterministic builds by default are such an obvious (in retrospect) core requirement."
I agree... I like the theoretical concept that our dependencies, could be automatically managed/upgraded behind the scenes via a declarative versioning system, like semver. That, in my view, was the paradigm behind not having defaulted to dependency graph snapshot (lock/shrinkwrap) by default. In practice, people don't follow semver correctly and autoupdating patch+minor versions can cause you a lot of pain. It would be cool to make versioning more tied to actual API signature, so that changes to the signature could not actually be published as minor/patch changes... but would be forced to use a major version bump in publish. I don't know how this would work particularly in a language like javacscript w/out a typed interface... but I'm just theorizing...
Yeah, I think that's a good comparison, except the upsides of yarn seemed much more obvious to the community than was the case with IOJS. There seemed a lot less controversy than with IOJS.
What a time those days were, feels like a million years ago already.
> Thank you, yarn, for helping the community see the naked emperor. Deterministic builds by default are such an obvious (in retrospect) core requirement.
Even with shrinkwrap, npm install is not deterministic, install order still matters. To get a deterministic install, you need to use shrinkwrap and do an `rm -rf node_modules` before every install. And it's still not completely deterministic if any dependency or sub-dependency has optionalDependencies.
That, and the fact that it's not the default behavior in npm gives yarn a pretty big advantage in my opinion. Especially for devs who are newer to Node.js.
I personally had invalid dependencies generated with shrinkwrap which did not work with a "npm install" after and had to edit the file manually so I would say it's not yet there.
Shrinkwrap is a partially successful attempt to solve the problem, but still has obvious flaws. For example, instead of the relatively neat version specifications normally used in package.json, with a shrinkwrapped project you wind up with https://registry.npmjs.org/... paths instead, but only sometimes. This gets messy if you want to both install stable versions of all your packages and use a local registry to supply them for reliability, which is not an unlikely combination.
We dropped npm in favour of yarn a while back, but I imagine we'll switch back once the stable release is out. Our needs are perhaps simpler than most, but the key issues we were having with npm at the time were directly solved by yarn:
1. Dependency install was taking too long.
2. Inconsistent builds between devs because of no lock file.
3. Could not search the registry from the command line in a timely manner.
Having just done a quick test on a random project, I'm pleased to see that all of those concerns are now taken care of, plus the install (for this project at least) is ~30% faster than it is with yarn.
Just tried on a couple of projects with a lot of dependencies, we moved to yarn a while ago due to performance issues and it seems to be resolved.
On cold cache:
Yarn: 20.94 seconds
NPM5: 21.11 seconds
With cache:
Yarn: 10.35 seconds
NPM5: 15.20 seconds
For some reason, when node_modules folder is still there, yarn exits in a couple hundres milliseconds but npm5 does something for around 5 seconds.
Haven't checked lock file / installation consistency stuff. Yarn has been great on that too so we have no intention to go back but this is a decent release.
So performance is now comparable, which is awesome, but I'd still stick to yarn because we have been burned too many times by npm v2/v3 with call stack issues and other errors. I don't have the energy (or the time/faith) to go through that again.
Competition is wonderful though. I'll check npm again in 6-12 months and see what the community thinks of it before switching again.
So happy with the --save by default. Someone at work kept installing new dependencies without save (they didn't knew about it, somehow). We then had an unusable package.json. I had to manually find directories in node_modules and install them on production -_-.
You should implement basic ci with travis et al. That commit should never be able to make it into master. If you are using Github you can force green tests before a pull request is mergeable.
And I think this is a complete bs. I have to try the module and then make a conscious decision to use (save) it, not kind of save it first and then hope that it actually does what I want.
Like wtf, who thought that saving something you download maybe for the first time as a dependency is a good idea?
It is perfectly reasonable and avoids mistakes. It's better to have a module that's unused than to break your whole build.
If you want to try something install it and then remove it. That's the more conscious decision. I don't want to have to remember to save the module if I end up liking it, but I will remember to remove it when tearing it down from the codebase, and if I don't, nothing bad happens.
Does uninstall default to --save too? Because if so, then your workflow doesn't even have to change gear. You just install it to check it out, you like it then do nothing, you don't like it then you can just `npm uninstall`, something you'd hopefully do anyway.
First time I hear about '--save' for uninstall, but it doesn't make a difference in my case because I always install many things, test them all at once, then '--save' what I need and 'rm -rf' the 'node_modules'. After that, 'npm i' gives me everything I need.
When you commit your changes, it's similarly easy to inspect package.json changes, and only commit the lines you want [1].
This change increases the chance that what you have working locally is reproducible from "source". IMHO that's a very important goal, while debate over convenience of omitting --save vs --no-save is superficial. Obviously there are 2 groups and one will be unhappy either way, but does that justify calling it "beyond stupid"? (I could maybe see that if you had actual data that 90% people want --no-save...)
Also, seems you can config old default by `npm config set save false` (with the risk one day you'll work on another machine and be surprised by uncustomized defaults).
https://twitter.com/maybekatz/status/859193277894991872
[1] If your commit flow doesn't make this a pleasant experience — e.g. command-line git IMHO doesn't — you deserve better tools. I can recommend `git citool` for start — old and ugly UI, but fast, portable, has keyboard shortcuts, also very convenient for amending.
It bothers me somewhat that they say this is standardized, but not by whom. Is the idea that Yarn should fall in line or is it a case of https://xkcd.com/927/ ?
I want to be able to search by tag, constrained by license and with packages that use dependencies I already have installed floated to the top. There are so many ways this could be nicer.
Yarn doesn't support all of npm's features. At least not as of a couple months ago when it 1) didn't support one of our needs at all, and 2) broke a couple NPM packages on install. This may or may not be part of why it's faster: it does less.
A look through their issues on Github will reveal quite a few "npm supports this feature, yarn doesn't, plz add" entries. There are even more "this works in npm but not yarn, and I'm not sure why", many of which are also effectively a result of features/cases that npm supports and yarn doesn't. I don't have an exhaustive list or anything, but it's right there to see if you're interested enough to dig for a minute or two. It's pretty clearly not a drop-in replacement for npm—at least not yet.
My experience a couple months ago was that almost anything slightly off the path of the most basic use case rapidly entered "here be dragons" territory, such that every problem became "are we doing something wrong, or is yarn broken again?", which wasn't worth any benefits it provided. If you're just installing packages from NPM, and those packages don't do anything even slightly weird, it's probably fine.
[EDIT] a running theme of the issues is that a lot of correctness-checking and edge-case handling is missing or incomplete, in addition to some "strange" npm features not being supported. Anyway, here are a couple examples of what I'm talking about:
(last one has a lovely "why are you doing this?" as its first response, from the repo owner for bonus LULZ)
FAR from the only ones. IIRC one of our issues had something to do with Yarn totally rewriting the package.json of a package installed from a local directory, removing the "main" entry entirely, which broke it (obviously). Maybe it'll be something I use later, but right now I don't really want to have to wonder whether any given problem is due to some obscure breakage in my package manager when I can... just use npm and not worry about it.
I also had a weird issue where for certain packages yarn would pick up a version which was older than what npm would. Not sure if this had anything to do with the fact that we have an internal npm mirror (that being said, yarn claims it picks up the .npmrc information, so even if it did, that would be a yarn bug).
In addition, yarn seems to increasingly be prioritizing its own default configs over what may be saved in my ~/.npmrc. Had several proxy issues after upgrading to a newer version of yarn that I didn't have earlier.
The reason I switched to yarn wholesale was because my existing npm based infrastructure worked seamlessly (at least to the extent that I was using npm features). But I've noticed that newer versions of yarn seem to have regressed in terms of npmrc support, which is a huge problem.
This seems to be very similar to Gradle vs Buck (or Bazel).
Generally speaking the main tool is slow, or otherwise suboptimal but does a lot. A different tool is built with an emphasis on speed. Eventually, the main stream tools picks up the speed features of the other tool and then "wins" because it does more. (Side note: I have no idea when Gradle came out in comparison to Buck/Bazel) For context, Gradle in the 4.0 version is starting to really support build cache support.
Edit: this comment is a reaction to all the other comments I see here about yarn vs npm.
It installs two node.js projects (react & ghost) and shows how long it takes to do under multiple scenarios (cold cache, installed and lockfile). It is automatically run each day. It also creates an average if one version is run multiple times.
I never used yarn, but I just did some research about the changes in comparison to yarn.
Well, actually they are more similar now, obviously (saving by default for example). In terms of speed, npm has improved as well (https://twitter.com/passcod/status/860437637764571140). Even though npm still seems to be slower than yarn (https://docs.google.com/presentation/d/1re-AGrTOf4zmnXXDMTG1...). At the end, I'd say it's about personal preference, they both get the job done.
>since npm@3, npm will automatically update npm-shrinkwrap.json when you save
All installs update shrinkwrap then? doesn't that start to make it redundant to package.json in the first place now. Does this make git tracking package and shrinkwrap mandatory in-case you want to install, test but then roll back the version, as shrinkwrap will already have changed.
edit: Oh I'm supposed to --no-save now for that described flow?
Shrinkwrap did not use hashes and did not guarantee deterministic installs. Subdependencies could still end up as different versions.
Even the shrinkwrap file itself contained a lot of trash and generated massive diff noise. I had a file full of scripts to make the shrinkwrap file usable and even then we had (production!!!) issues due to changing subdeps. We reworked our build processes to zip & deploy the exact code that passed the tests to work around that, but it was still a massive pain that installs at different points in time would yield different trees.
When yarn came out, I deleted a folder full of hacky scripts, cut my install time by 60% and finally got deterministic installs. Needless to say, I was ecstatic.
npm used to have issues that if you had a shrinkwrap and a pre-existing node_modules directory and ran `npm install`, then npm would often report success but silently fail to make the node_modules directory actually match the shrinkwrap. ... After our build system ran into this issue once and built and deployed code to production with fatally mismatched dependencies, I wrote a hacky wrapper script which would double-check that node_modules really did match the shrinkwrap, and if not it would remove the directory entirely and re-run `npm install`... Thankfully this was fixed in npm v4.
OK that's a bug, but it's fixed for more than a year. Nobody in this entire thread has given a reason why a captured versions of the entire tree wouldn't produce deterministic output.
Yeah, it should only be bugs that cause subdependency versions to not match the shrinkwrap. Your question seemed valid. I can only guess others have run into similar issues as me and hadn't known if they were fixed.
This is probably a naive question, but the only use case I've come across of needing a flat-install option was to use Polymer.js web components (which is why so much of the polymer project relies on bower).
Aside from that, what is the use case that flat installs solve?
> A new npm cache verify command that will garbage collect your cache, reducing disk usage for things you don’t need (-handwave-), and will do full integrity verification on both the index and the content
> Running npm while offline will no longer insist on retrying network requests. npm will now immediately fall back to cache if possible, or fail. (#15666)
"package-lock.json" can someone please define what it is and does, I've heard of shrink wrap, google doesn't reveal anything about this file, I have a vauge idea that it has something to do with shrinkwrap but don't know why it exists or if it's supposed to replace shrinkwrap or whatever. splainit!
How is the package manager for a language still iterating through major versions and making breaking changes nearly a decade after launch? Node is bizarre in that you have to follow its package manager so closely.
npm has to deal with scale: it's now the largest programming language package manager, and most project dependency trees have exploded in size as a result. Optimizing that led to yarn and to npm@5.
I could be incorrect, but I don't believe Bundler was the first to lock and manage dependencies? I seem to recall using such tools before 2010. This isn't to take away from your main point by any stretch; only recently has JavaScript gone from "downright terrible" to "tolerable" and most of it is through doing what others already did, better, many years before. It still feels like a lot of the community doesn't have sufficient exposure to stuff outside their closure and people who work with that community's stuff are the worse for it.
This statement is false and has been since the very first work done on the npm cli. npm has, in every version of it that has ever existed, since before there was even a package registry, installed preferentially from its local cache of packages where possible.
Couple questions:
Question 1: Does anyone else who's been around more than a couple years share my view that Yarn : NPM :: IO.JS : Node?
IOW: healthy competition, catalyst for necessary change, ultimately a bridge or stopgap.
Question 2: Any good comprehensive writeups on best practices?Committing lockfiles is a no-brainer. But what about globals? Per-node-version seems logical, but there are also semantic problems with "global" packages vs concept of executable binaries. I'd love to see a strong writeup outlining and defending a standard approach.