This is a huge leap forward for the JavaScript community—probably more than many people will realize right away.
I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once. npm solved this problem elegantly, but still struggles with non-determinism. I had resigned myself to thinking that maybe these were just fundamental tradeoffs in package management that could never be resolved.
Then I had the opportunity to use Cargo, the package manager for Rust. It synthesized what, in my mind, are the best features of npm and Bundler into a single package manager, with terrific speed to boot.
Yarn looks like it will be Cargo for JavaScript. After Cargo improved on many of the great ideas of npm, those ideas are returning to help improve the JavaScript ecosystem. Cross-pollination at its best.
This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.
I'm also happy to see that a significant amount of work has already gone into the governance model. Despite the largest contributors being Facebook employees, it looks like they've really outdone themselves making sure this is a community-run project. A BSD license, no PATENTS file, and an Ember/Rust RFC process[1]; this has all the hallmarks of a community open source project. That's critical for open infrastructure projects like this, and they've nailed it.
I'm very much looking forward to using Yarn in my own projects because it looks like it solves a lot of real problems I encounter every day. Thanks for all the hard work!
> This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.
And a shrewd move by FB: to not announce their new registry on the same day.
> Yarn pulls packages from registry.yarnpkg.com, which allows them to run experiments with the Yarn client. This is a proxy that pulls packages from the official npm registry, much like npmjs.cf.[0]
Time will tell whether they only want to be proxying NPM or will allow direct pushing to their own registry. If they do, JS ecosystem might see another big shift.
That big shift will have to happen first. I don't see them ever making their own registry unless 99.99% of people are using yarn and are having a lot of problems with the current npm registry. While I see a lot of people using yarn, I'm not sure about 99.99% and I think npm's registry itself is pretty good.
So I don't think interests will ever align to create a new registry. Nobody wants to do that. That would have serious consquences for the JS community and would take years to recover, in my opinion.
Why would it be a bad thing to support additional repositories? Personally, I don't like how centralized the JS ecosystem is.
For example, if I refer to 'left-pad', it would default to 'npmjs.org/left-pad'. If the author goes rougue, I think it would be great to enable people to publish and consume 'thirdparty.com/left-pad'
Disclosure: I'm a FB employee with no knowledge of our plans in this regard
1. any left-pad issue has been "eliminated" with the new rules npm (the company) has enforced.
2. you can already have your own version of thirdparty/left-pad by maintaining your own npm repository. i think what your parent post is referring to is facebook saying we're going to make our own public node/javascript package registry and you should publish to our registry.
doing this at the moment does nothing for the community other causing a lot of pain points. ex) now npm authors will have to publish to both registries so developers don't have to dig to find where it was published to, then they also have to hope that someone else didn't register the module name in one of the registries..
there is just too much splintering if facebook decided to become a competing registry rather than just using npm's registry and building on top of it.
I used to find such comments funny until I began working for Big Name Corps myself and I realized how much one's personal philosophy could be inconsistent or sometimes even contradictory to the employer's philosophy. I make money by selling my skills to an employer despite inconsistent philosophies. I think it's like a chef that can cook meat for his/her guests although the chef has decided to refrain from consuming meat himself/herself.
> So I don't think interests will ever align to create a new registry.
Possibly not soon, at least from a technical perspective, but I could definitely see a PR fiasco (security, bias, etc.) causing a loss of confidence in its stewardship.
And I don't think it'd be that big of a disruption if it were to happen; for the hypothetical case of Facebook+Yarn, they're already proxying to NPM, so they could easily continue to do so while also accepting packages that are Yarn-only.
Even if they don't get 99% of users (or any major percentage) I think it would still benefit the community to have an alternative to npm.
Also Facebook has the advantage of not needing to make a business out of it (they already got a pretty good one) so in theory it could be entirely open source and free.
Just to expand on the stricter versioning rules that I mentioned, some things that in my opinion could improve the reliability of a package repository:
- strictly defined version update time intervals, e.g. you can't update your package more than once a week (or have to take some special actions for critical updates, e.g. contact support)
- "delayed" publishing, e.g. when you submit your package it will only be published in 24 hours, until then you can re-submit updates, etc.
- similar to above, but your package wont be published until it was tagged on github (or elsewhere) for a certain amount of time
- published packages can not be removed, but you can disassociate them with your account by marking them as "not maintained" and possibly assign new maintainers for it
- maybe introduce some way for developers to mark new versions as "backwards incompatible" if they do break backwards compatibility
I think there is definitely a "market" for some stricter node package repo.
Well, it seems like it could be according to the article. What would you still need to use NPM (the cli) for, other than package hosting?
According to http://blog.npmjs.org/post/151660845210/hello-yarn, it seems it doesn't work with private packages yet, which may or may not be an issue for your project. But it seems this is a complete CLI replacement for NPM.
"Could have been implemented as part of the existing client" isn't the same as "Should have been implemented as part of the existing client".
I personally don't know much about either tool (don't do a ton of JS), but it's possible that fixing the existing client without either breaking backward compatibility or making it too complicated (multiple modes of operation) was too difficult or not worth it.
Also, I'm having a really hard time understanding the complaint about a new client. The value is in repository of reusable code, not the client. That you can use different clients with the same repository is a feature, not a bug.
Yeah - the npm client codebase would not be fun to do a big refactor on. And you would need to do a big refactor to make the v3 flat install process deterministic and/or performant.
These issues have been raised a few times, along with the shrinkwrap/reproducability stuff, and it didnt seem like it was a big priority for the core team. Understandably I guess they seem more focussed on the enterprise/private repo side of things and just keeping things running on the client side.
The whole version range stuff got me many times. I went to use fixed versions on my own package.json files, but the deps of my deps could still be dynamic, which is even worse, since they sit deeper in my dependency graph AND there are more indirect deps than direct deps. (~50 direct, >200 indirect)
Also, npm isn't deterministic and it got even worse with v3. Sometimes you get a flat list of libs, if a lib is used with multiple versions, the first usage will get installed flat, the rest in the directory of the parent lib, etc.
The fun of kicking off a CI build after the weekend with no commits and see stuff randomly break because some dependency of a dependency got updated and broke things in a minor version is something I've only experienced in JS - beautiful.
In fairness to the language and tools, this seems to be more of a cultural problem than anything.
You can do the same kind of version range tricks in typical Java builds, for example (Maven), but most people hardcode the values to keep builds as deterministic as possible.
For some reason, the JS community seems to prefer just trusting that new versions won't break anything. Its either very brave of them really (or maybe just foolish).
> the JS community seems to prefer just trusting that new versions won't break anything. Its either very brave of them really (or maybe just foolish).
Let's not pretend that we aren't all blindly tossing in random libs of dubious quality and origin we find on github into our package.json and hoping for the best anyway. My company talks a mean talk about "best practices", but, my god, the dependencies we use are a true horror show.
I hate non-reproducible builds and semver-relaxed dep-of-the-dep issues, but, while a broken dep fails the build for lots of people (downside), the upside of this is that very quickly (within hours of a new dep being published) there will be lots angry people complaining about it on GitHub, and a faulty dep will be typically quickly rolled back / superseded with a patch. Otherwise, some bugs might be sitting hidden for a long time.
Say that I use yarn to depend upon some module X, specifying a fixed version number like the good boy scout that I am. Module X resides on npmjs and in turn depends upon the bleeding edge version of module Y. And then one day module Y goes all Galaxy Note and bursts into flames.
It's not a cultural problem. People make mistakes. People don't know what a non-breaking change is, especially those not well versed in refactor work.
I don't think Yarn solves any of these problems, tbh. It seems like what we really need is a package manager that tests the api of each package update to make sure nothing in it has broken expectation in accordance with Semver.
You shouldn't be kicking off CI builds based on unpinned deps (unless you're deliberately doing 'canary testing' etc), because of course that will break. The npm solution for this is to use 'npm shrinkwrap' and you should always have been using this at your site/project level otherwise there was no hope it could work.
It's not that npm devs were naive enough to believe that unpinned deps would be safe for reproducible builds.
However I've heard several people allude over the years that 'npm shrinkwrap' is buggy, and isn't fully reproducible (though never experienced any problems personally). This is the aspect yarn claims to address, along with a supposedly more succinct lockfile format.
Obviously without until we broke and looked in to it :D
JS dev has been a minefield like that, the entire ecosystem reminds me of C++ except lower barrier to entry means a lot of noise from newbies on all channels (eg. even popular packages can be utter crap so you can't use popularity as a meaningful metric)
I agree 100%, but the default upgrade strategy for npm --install does not help matters: it's much saner to wildcard patch versions only and lock major.minor to specific versions.
this obviously doesn't fix anything and I think the points in this discussion stand, but I've never understood why the defaults are not more conservative in this regard.
What we do currently is we lock everything to an explicit version - even libraries.
At least it's possible to get deterministic builds if you are willing to do a bit of work carefully / manually updating all of your dependencies at once.
You shouldn't have that happen with Cargo, given that we have a lockfile. Even when you specify version ranges, you're locked to a single, specific version.
If you follow the best practices for using Cargo you don't have a Cargo.lock file for libraries.
This means your library tests will not be deterministic.
Using the Cargo.lock file for libraries does solve this, but then every binary package you build that references your library will have to be specifically tied to the versions in the library.
We do this internally because it's the only way to provide deterministic builds over time.
Over time I suspect it will get harder and harder to keep this rigid web of dependencies working.
One thing we might try is to enforce the rule 'no library will contain any tests'. At first glance this kinda makes my skin crawl, but maybe if we could find a way where every project that used a library could somehow share the tests for that library it could actually work.
git submodules might actually be able to provide these tests for each binary package. If only git submodules would stop corrupting our git repos... :-(
NPM is deterministic when there using the same package.json and there is no existing node_modules folder.
And if you want to lock versions for your entire dependency tree, npm shrinkwrap is what you're looking for (It's essentially the same as lockfiles in other development package managers). Though for security reasons I prefer to keep things locked to a major version only (e.g. "^2.0.0"). Shrinkwrapping is useful in this instance too if you need to have predictable builds (and installs as it'll use the shrinkwrap when installing dependencies too if it's a library rather than the main app) but want to ensure your dependencies stay up to date.
It's not perfect by any measure, but there are ways to make it work the way you want.
Mhm that sounds reasonable.
I haven't gotten into problems with versions yet but I don't really trust npm update, because I am not always sure how it behaves.
From a build tool perspective (we use npm scripts for basically everything [and some webpack]) I am also not missing something particular.
Looking at other comparable solutions (from other languages) I'd say npm does a pretty good job.
As a practical matter, the npm ecosystem today relies on duplication, and no new client that made the "highlander rule" (there can be only one) mandatory could succeed.
Yarn does offer a `--flat` option that enforces the highlander rule, and I'm hopeful that the existence of this option will nudge the ecosystem towards an appreciation for fewer semver-major bumps and more ecosystem-wide effort to enable whole apps to work with `--flat` mode.
Just imagine two packages you depend on (a and b) that both have a shared dependency (x). Both start off depending on x version 1.0 but then later a is updated to 2.0 while b isn't. Now you have two packages depending on different versions of the same package and hence the need for duplication. You have a that needs x@2.0 and b that needs x@1.0, so both copies are kept.
Don't upgrade a when it wants a half-baked x. Choose versions of a and b that agree on a known-good version of x. If there aren't any, it's not sane to use a and b together unless x is written very carefully to accommodate data from past and future versions of itself.
It's not as simple as that. Lodash is a great example of why the highlander rule doesn't work within the npm ecosystem: older versions are depended on by many widely-used packages which are now "complete" or abandoned. Refusing to use any packages which depend on the latest version of Lodash is just not practical.
I'm arguing for choosing dependency versions that don't require you to break the highlander rule. "a is updated to 2.0" doesn't mean you should start using that version of a right now.
Right but what I'm saying is that two versions of the same library will live quite happily together, because node.js absolutely abhors global scope. The two versions have their own module-level scope to store functions and data.
So go crazy and install every version of lodash! Nothing will break.
That will work for lodash, because it's just a bag of functions. But anything that creates objects that are passed outside of a single module could have problems.
That seems to be a slightly different situation? As the issue poster described, a single page had invoked two copies of React, both of which assumed ownership of a particular "global" variable. (Not really global, but a particular member of a global.)
I'm not a React expert, but I don't see why that situation would only affect multiple React copies with different versions?
I'm not a large JS developer but with my other experiences managing dependencies, it doesn't always allow this as a possibility. A bug in your system is tracked down to the library b using x@1.0 but the fix is to upgrade to b with x@2.0 however a using x@1.0 doesn't have an upgrade path. Waiting for another company or organization to release with an update is not an option. Our projects have several cases of requiring different versions of the same library -- we try to avoid this using the same logic you suggest but it's not a possibility in all cases so we have to work with it.
It's placing your own release cycle at the whims of your dependencies' release cycles. In the corporate world that would not be a viable solution.
It's almost certain that a's version of x and b's version of x have distinct types whose names collide, and I don't know of any Typescript or Flow notation to prevent passing incompatible instances between them (e.g., a wants to call a method that didn't exist in b's version of x), so if anything works it's only by luck.
Node's module system doesn't import dependencies into a global namespace. So the `x` required by `a` and the `x` required by `b` will have separate namespaces and won't conflict, or even know of each others' existence. There are pros and cons to this approach, but it definitely does work. Flow, at least, (and presumably Typescript) understands the module resolution logic and handles this very common situation without issue.
It's not possible to import two different versions of a module in the same module.
If a depends on x v1 and exports this dependency then your application also imports x v1.
If b depends on x v2 and exports this dependency too that means your application is transitively importing both x v1 and x v2 which is not possible.
If a or b only use instances of x internally and do not accept instances of x as parameter or return an instance of x they don't have to export their dependency on x.
If either a or b do not export their dependency on x then there is no problem. Your application will only depend only on x v1 or x v2 directly.
Would it be possible to create hardlinks or symlinks to a particular package/version pair shared as a dependency between other packages? I know this only works on unix-like OSes but otherwise it could revert to the old behaviour of duplicating the dependency.
I think they're just saying that any new client that tried to not support duplication at all would likely quickly run into a large amount of npm packages/package combinations that just don't work. So within the context of using the npm registry duplication is mandatory.
Cargo definitely allows two dependencies to rely on different versions of a transitive dependency. If those deps expose a type from that dependency in their API, you can very occasionally get weird type errors because the "same" type comes from two different libraries. But otherwise it Just Works(tm).
Cargo does allow multiple versions of transitive dependencies. It tries to unify them as much as possible, but if it can't, it will bring in two different versions.
What it _won't_ let you do is pass a v1.0 type to something that requires a v2.0 type. This will result in a compilation error.
There's been some discussion around increasing Cargo's capacity in this regard (being able to say that a dependency is purely internal vs not), we'll see.
With tiny modular utilities this is very much necessity - and not a bad idea if the different versions are being used in different contexts.
For instance, when using gemified javascript libraries with bundler it is painful to upgrade to a different version of a javascript library for the user facing site while letting the backend admin interface continue using an older one for compatibility with other dependencies.
You've got to take the ecosystem into account. There are a lot of very depended-upon modules that are 1 KB of code and bump their major version as a morning habit. Forcing people to reconcile all 8 ways they indirectly depend on the module would drive them nuts, but including 4 copies of the module doesn't hurt much.
That's exactly what Cargo does, but it also takes a further step of making `^` the default operator and strongly discouraging version ranges other than "semver compatible" post-1.0.
Personally I'm not really sure I like it. If I specify an exact revision of something, chances are I really do mean to install that exact revision. I don't see why I need an extra flag for that.
That can cause some serious problems in at least some portion of times. I've dealt with the subtle errors that have been caused by this problem in c++, and don't really know javascript libraries that well so I can't give a more concrete example. But imagine that there are the following libraries:
* LA: handles linear algebra and defines a matrix object.
* A: reads in a csv file and generates a matrix object using LA
* B: takes in a matrix object from LA, and does some operations on it
In this case, if B depends on version 5 of LA and the new version of A depends on version 6 of LA, then there's going to be a problem passing an object that A generated from version 6 and passing it to B which depends on version 5.
The problem does happen in JavaScript. But since its unityped, there is a strategy to deal with it
* Figure out early on (before 1.0) what your base interface will be.
For example, for a promise library, that would be `then` as specified by Promises/A+
* Check if the argument is an instance of the exact same version.
This works well enough if you use `instanceof`, since classes defined in a different copy of the module will have their own class value - a different unique object.
* If instanceof returns true, use the fast path code (no conversion)
* Otherwise, perform a conversion (e.g. "thenable" assimilation) that
only relies on the base interface
Its not easy, but its not always necessary either. Most JS libraries don't need to interoperate with objects from previous versions of themselves.
Would this even work in what I describe? For instance, if mat.normalize() was added in LA-6, and B provides an LA-5 mat, and then A (which has been updated to use the new method) calls mat.normalize() on the LA-5 mat expecting an LA-6 mat but because of duck-typing that method doesn't exist.
However, since A exposes outside methods that take a matrix as an argument, it should not assume anything beyond the core interface and should use LA-6's cast() to convert the matrix.
The problem is partially alleviated when using TypeScript. In that case the inferred type for A demands a structure containing the `normalize` method, which TypeScript will report as incompatible with the passed LA-5 matrix (at compile time). That makes it clearer that `cast` would need to be used.
Define the matrix class in a seperate library for compatibility purposes if it's so widely used and doesn't change. Maybe some people don't need both the linear algebra and instead only the definition of the matrix object?
Another solution is to provide version overrides and make B depend on version 6.
However if there are differences in the matrix class between different versions of the library then you're forced to write a compatibility layer in any case.
If an API expects the outside world to hand it an instance of a specific library, all bets are off. Maybe it gets `null` or the `window` object, who knows? But a library can at least declare what dependencies it wants. If you take that away, it ratchets up the uncertainty factor that much more.
This is especially true when you're going to be serving your code over the web. It's very easy when using npm to accidentally end up shipping a lot of duplicate code.
That alone has me super excited about Yarn, before even getting into the performance, security, and 'no invisible dependency' wins.
I feel like this is especially problematic when using NPM for your frontend. Now you have to run all your code through deduplication and slow down your build times or end up with huge assets. I wonder if it's really worth the trouble.
> I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once
This is due to the way that Ruby includes code, where it's available globally, vs Node where code is scoped to the including module. I'm not sure how Ruby could support multiple versions with changes to the language and/or RubyGems
Why do you think that everybody is inspired from javascript tech? It is probably the other way around. For example there were very good build tools long before npm which are still used today and unmeasurably better (like Maven or Gradle).
This may come off as a troll, but it's an honest question. I'm not a javascript guy. It's not a language I deal with at all. Why on God's green earth does it need as much tooling as it seems to have? Are people really making projects with dozens (hundreds? more?) of dependent libraries? Are there aspects of the language or runtime that reward multiple layers of configuration management? In short, what the hell is up with this ecosystem?
1. In the browser we have less control over the environment the code runs in, requiring tooling to achieve a stable, normalised environment which we can improve at our own pace, rather than the rate at which browsers are updated.
2. JS has become very widely used very quickly, which means that there have been lots of hastily built things, which have been replaced by better things (and for every 'better thing' there are many alternatives which fell by the wayside), and a larger pool of people to build them.
3. It's easier than ever to build new tools (or any kind of app, for that matter), because the npm ecosystem makes it trivial to combine existing smaller pieces of code into new combinations.
Your number 3 is the one that really drives it home for me. It's the unix philosophy taken to the extreme, and (at least in my experience) it's working very well.
It does take some getting used to, but when you stop looking for one big tool that can solve 5 semi-related problems, and start looking for 5 little tools that each solve one of the problems, things get much more clear.
And yes, I know that this kind of reliance on dependencies can cause issues, but from my experience, those issues don't seem to bite that often, and many of them can be resolved with even more tooling.
browser incompatibility is the biggest issue (if not the only issue). this is almost a non-issue if you're just writing node related programs. but if you need to support in the browser, now you have edge, chrome, firefox, and safari at minimum. then some have to continue supporting ie10/ie11, or even worse, ie7+.
don't forget about mobile devices, with ios having such a difficult-to-write-anything-new version of mobile safari.
i'm writing a new app for myself, so i don't have to worry about any backwards compatibility issues, except it needs to work on my iphone, which uses mobile safari webviews, which means i have to either use shims or restrict myself back to es5, non-html5 features (or work around their html5 limitations).
example: you cannot cache audio files to play using <audio> tags in mobile safari. safari just refetches it from your server (also has to recreate the audio object, which causes a lag in the audio being played). technically you _can_ cache if you convert all your audio files to base64 and use the webaudio api. i'm probably going to have to go this route because i'd like the app to be offline usable.
so rather than spend time on my app, i now have to write out a build step for converting audio to base64. then test that it works fine in ios and desktop browsers (at least chrome, firefox). it all just keeps adding up.
I came to the Javascript world late, starting with React. I think its tooling has gotten out of hand, but there are a couple of reasons it's not a _total_ disaster in my mind:
* Partly, there are a larger number of Javascript developers that haven't been exposed to other ecosystems, and end up reinventing some wheels
* More importantly though, I think the pace of innovation in the javascript ecosystem is actually good in a way, because other more mature languages (I'm not talking about age of the language, just maturity of the tooling and ecosystem) have gone through the same growing pains, it just took a lot longer. It's easy to forget, coming from the Python world for instance, the pains of easy_install and distutils vs seuptools, etc etc. I still find Java dependency management a bit of a pain, because I don't know it very well.
We're just watching a sped up version of a language maturing, and it's painful as individual developers trying to keep up, but I don't think it's as negative as HN often makes it.
> Are people really making projects with dozens (hundreds? more?) of dependent libraries?
Yes.
> Are there aspects of the language or runtime that reward multiple layers of configuration management?
A significant fraction of the node community likes to split their project into really, really small packages and treat them as if they're independent even if they come from the same source repo and tend to get used together. As an example, the popular collection processing library Lodash publishes a package for each method and people actually do depend on individual methods.
There are two major rationales for this. The first is that JS is a dynamic language and in dynamic languages, smaller chunks of code are easier to reason about than larger ones. The second is that the JS community as a whole cares more about artifact size than pretty much any other community. Having smaller packages allows you to produce a smaller payload without having to rely on techniques like tree shaking.
I find micro packages maddening but people give me their code for free and it mostly works so I don't complain about it that much.
What do you find maddening about micro packages? I hate depending on a giant stack of frameworks (slowing installs, builds, adding behavior surface area) when I just need a function or two, so micro packages are a joy for me.
Reading the article it seems to me that this micro packages approach is what slows down everything.
I never, ever have seen all these problems in even not-so-state-of-the-art dependency managers like maven or nuget.
Seriously up to now it was impossible to have a build server isolated from the internet if you didn't want to check-in all the dependencies???
Simply crazy. I really can't understand how people can even think to use a dependency manager system that doesn't satisfy the essential requirement of having your CI server sandboxed.
> this micro packages approach is what slows down everything
Not really. The npm client is just flaky, very slow, buggy, has some major design flaws in its v2 incarnation, and has a completely different set of major design flaws in its v3 incarnation.
The core architecture works fine for small packages.
> Seriously up to now it was impossible to have a build server isolated from the internet if you didn't want to check-in all the dependencies
Of course not, although I admit the linked article implied it was if you didn't read closely. There's a wide number of solutions, including running a private registry, or a copy of the public registry, or a local cacheing proxy of the public repository, or a custom npm client that hits a local cache, etc. Some of the solutions work quite well, and in fact yarn itself is just a re-implementation of some existing solutions.
> I really can't understand how people can even think to use a dependency manager system that doesn't satisfy the essential requirement of having your CI server sandboxed.
As the article noted, Facebook was sandboxing their CI server and using NPM; they just didn't like the existing solutions. (Nor should they; they were a bit naff.) But that doesn't mean (nor is anyone claiming) that there were no existing solutions. Yarn looks great, but it's an extremely incremental change.
Javascript is very easy to work with if you just stick to micro packages. What slows it down is that the language has been forked to an incompatible spec (ES6) so there is essentially a hard fork in pool of libraries. To deal with this—and calm anxieties around lack of inheritance, trauma over having to type function over and over, etc—layers of pre-processing have been added: JSX, SASS, asset compilation, compatibility layers, framework booting. This pipeline distances developers from the DOM and other low-level protocols, increases the surface area of the build process, and frequently disables debugging and introspection tools. In addition megaframeworks like Angular, Ember, and React add mandatory layers of indirection and runtime dependencies between each snippet of code, introducing behavioral complexity and confusion from non-deterministic builds.
All of this is why people are excited about Yarn, but to me it's a band-aid on several architectural mistakes.
Just say no to frameworks. Just say no to ES6 and transcompilation. Just say no to asset pipelines. Viva la JavaScript. Viva la Netscape. Viva la Node.
Where would a CI server retrieve dependencies from, if not either over the network, or from within the repository? Do you keep a module / library cache on the CI server itself? In other words, what do supply as input to your CI process, besides a Git URL?
In a local nexus/nuget repository in the intranet, for sure I'm not allowing internet access directly from a CI server.
And apart from the obvious and scary security concerns it is also much faster.
Got it. You can also set up an NPM repository like that, but I don't think that that is commonly done by small dev teams. This does lead to hilarity like the left-pad clusterfork.
I have no idea if you're familiar with the java ecosystem, but I am, so I'll use that as an example:
Node libraries are typically several orders of magnitude smaller and more single purpose than java libraries. Java libraries like Guava and Spring are typically quite large. A practical example is from unit testing: In Java, you might use JUnit+Mockito for your test dependencies. In Node, to get similar functionality, I have the following deps: Mocha, Sinon, Sinon-As-Promised, Chai, Chai-Moment, Chai, Chai-Things, Chai-Shallow-Deep-Equal, Chai-Datetime, Sinon-Express-Mock, Supertest.
Many Javascript applications both client and server functionality, and there is different tooling for both. Webpack is currently popular for building stuff that will run in the browser, whereas server side stuff is different.
It's a much newer technology than java and there is quite a lot of churn around tooling in general as the nature of the way applications are built change.
Strangely enough both Java and JavaScript were first released in May 1995. Server side JavaScript was released the same year just after the browser version.
Though I get that you are referring to the js eco systems (commonjs, npm) compared to the much more mature Java ecosystem.
Why is that?
In the Java ecosystem there is Spock framework that gives you everything required for testing, you don't even need junit+mockito.
In which scenario would be better to use 11 different libraries for doing the same thing as a well written framework?
In this case, I would argue it is not better. Many JS devs have moved onto simpler, greener pastures. https://github.com/substack/tape is what I prefer. Small little library. Its API has everything I need and nothing I don't. Those other test libraries listed are pretty much overkill.
Similar with http://mithril.js.org/ over React, Angular and Ember. A single drop-in library with 13 functions in it's API. The library itself is a single, understandable source code file. Mithril is pretty much the exact opposite of Angular and Ember.
A lot of front end devs are eschewing frameworks completely. With modern DOM and web APIs, you can do a lot with "just" vanillajs.
But, this is exactly what I'm saying. you start with a small, focused library that contains everything you need, which is usually fairly minimal, so that library is small. but then, along the way, you find yourself wanting, so you add a library that adds support for that specific thing.
I didn't add sinon-as-promised for fun, I added it for in-memory sequelize integration tests. same for all the other libraries I mentioned.
My point is that by keeping libraries small, you will end up with either more dependencies, or a lot more hand-rolled code. Neither of these things are inherently bad or good, and I'm not defending them. What I am trying to do is explain to the commenter why the node ecosystem seems to have such an emphasis on package management and the tooling around that.
From the readme I can't really see how to write extensive unit tests with that thing..
If I need to test something I want only one tool that works perfectly and does all the Stubbing, Mocking, verification, data driven test and behaviour driven test.
I still haven't found in any language/ecosystem anything comparable to Spock in completeness, readability and ease of use.
Honestly I can't really see how to do BDD with tape.
its not that hard. That being said, the current all in one turnkey testing framework for javascript is Jest, and that will pretty much do it all in one tool.
The benefit of what's happening in JS is that a lot of libraries are reasonably compartmentalized for a specific purpose.
As an example: you _could_ have one SpockJS framework that gives you React + Redux + ReactRedux, but instead each of these libraries are separate so you can use them without each other. This allows you to, say, move from React to Angular without changing the state of your application.
With everything moving so quickly in JS, this separation turns out to be valuable enough to tolerate a lot of packages and dependencies. It also makes it easier for communities to exist for solutions to specific problems, which at least in theory results in better solutions.
It's not necessarily better, just a different approach.
For testing it doesn't make any sense. I want a coherent and expressive framework that does everything. If I'm doing BDD I know that I will need everything from the start. And from the little that I have seen about jest..well it seems really a jest. I can't really think to do any serious BDD with that.
I guess you could call it "diversification" of dependencies. If one dependency stops being maintained or is not the best approach you can jump to another one.
A very limited standard library with no native (well, just approved but not ratified) support for importing/libraries. Coupled with inconsistent implementations and you end up need more layers of abstraction.
Let's consider Microsoft Office. It is a massive software and such as massive software requires a lot of (internal) dependencies and toolings.
Now imagine Office 365, the same software with the same functionalities, except running in the browsers. The task is to essentially write Microsoft Office (front-end logic) entirely in JavaScript (or TypeScript), with the additional problems of dealing with browser compatibilities issues, which are just as hard as OS compatibilities issues.
Then, imagine all the software going to the "cloud". JavaScript now has to have the capability of re-writing all these software originally written in C++, C#, Java... all sorts of languages. It is quickly becoming a superset of all ecosystems where everyone is trying to migrate their software to. So it is not surprising that people are inventing more and more tooling to make it more powerful to suit their needs.
You say tooling, but then talk about libraries. I think both are just as numerous for nearly all popular programming languages. It’s just that JavaScript is currently “hot shit”, so all this stuff is a lot more visible.
There are tons of libraries for C, many of which aim to do the same, only faster/smaller/scalable/whatever. There are many compilers, both open and closed source. And there are documentation tools, syntax checkers, unit testing frameworks and whatnot.
Of course there are also languages like C# where there’s a de-facto standard for almost everything: IDE, compiler, base library, GUI framework(s).
Because, from this outsider's perspective, it seems like most of the tooling is library management (rather than, say, profiling or linting -- does javascript even have a linter? -- or debugging).
JSLint, JSHint, ESLint. Each of which built on the good aspects of its predecessor and evolved as new requirements came up. There are a _ton_ of tooling libraries for JavaScript. Just google "X JavaScript" for any tooling you'd expect, and you'll usually have several options. Of course, on the flip side, people will complain that there are too many options. Blah blah blah. No matter what the JS community does, people will criticize.
There are tons of tools, of course also including jslint, debuggers (e.g. IntelliJ, dunno about others). There are editors and even IDEs. Then there’s the problem of bringing (NPM) modules to the browser using tools like webpack or SystemJS or whatever. There are general purpose build/automation systems like gulp. There are transpilers that “compile” ES6 to ES5. For any given task, you can probably find at least five programs that do it.
And if you look at all the topics, you’ll find that library management makes up only a small fraction of everything.
This is a valid question coming from an outsider looking in.
There are various packages that are extremely small (see leftpad and associated controversy). They often wind up in a slightly larger package, which itself winds up in a slightly larger package, which recurs until you finally wind up with one actual package of consequence that you're targeting. For example, Express (a very common Node.js abstraction) has 26 direct dependencies, yet 41 total dependencies.
A lot of this results from early Node.js' mantra of having small, focused packages. This could potentially be a good thing because instead of having three major dependencies that have their own way of leftpadding they can all rely on one library and thus code is efficiently reused. This can be bad, however, when they each need their own version of the dependency - or worse - the dependency is removed from the registry (see Leftpad and associated controversy).
One of the legitimate problems that I've seen is that there are various libraries that are just thin wrappers over functionality that is native to Node.js - but written to be "easier to grok". Thus, these thin abstractions become practical code duplication out of some mixture of Node's developers lack of effective verbosity/documentation and application developer laziness. But then they creep higher up the dependency chain because someone at a lower level used it.
On one hand it can be quite simple (and rewarding) to write code while minimizing dependencies. On the other hand, abstractions such as Express make it very easy to write code and it feels like you only have "one dependency" until you look under the hood.
Babel, the most popular transpiler, is 335 total dependencies, and it doesn't even do anything out-of-the-box. You need to add babel-preset-latest for 385 total dependencies if you want to actually transpile anything.
A lot of these answers are pointing out the x-browser complexity which is true but that isn't what drives JS package mgmt. It's mostly Node stuff. The mature common JS libs for browser work are pretty robust when it comes to x-browser stuff now days.
For browser work, you can do almost everything you need with some jQuery, handlebars and moment if there are dates. A few other small libs here and there and you've got almost everything covered. Some edge cases may drive you to use some library that is heavily dependent on 32 different sub-libs but it's really not that often.
Server-side Node.js is the issue, not browser compatibility.
Handling dependencies is hard. Handling dependencies for a platform you don't control is harder. Handling dependencies for a platform you don't control, where every single kilobytes count, is insane.
Add those together, and you get the need for real complex tooling.
With that said, tooling is nowhere as complex as it is for languages like C++ or Java. People are just used to it on those platforms, and the tooling is more mature/understood because it already went through the countless iterations, 15-20 years ago or more.
The tooling for building C++ consists of make or something that generates make files. There's also no "1 function" libraries.
Adding a dependency is done by (compiling it if needed) and referencing it from the make file.
It's super easy to understand, but involves manual work because C++ doesn't have a package manager.
Java isn't too complicated either. I've used Gradle and Maven but doing things manually like in C++ can also work.
Crucially, they don't reinvent the wheel every couple of months so a C++ or Java dev can focus on building things instead of relearning how to build things.
Javascript in my opinion is still very young. You can't compile it to web assembly (yet) and the fact that there are different browsers that interpret it in so many different ways make for libraries to exist to mitigate this. Also none of these tools are 100% necessary, they just make your life easier and tools always have room for improvement. I'm sure others can add to this. I don't think any of this is a bad thing, it's just overwhelming for newcomers that haven't been following this whole thing.
Others have touched on the subject but one of the biggest issues is browser compatibility. Building a responsive website that works on all major browsers, across multiple versions, as well as all the mobile quirks is not an easy task in the slightest ... and that's just to make a site look pretty. Then we have to start talking about building massive web apps that are front-end heavy which requires even more tooling to scale, maintain properly.
Because JS is open source and it's not controlled by an Oracle / Microsoft / Apple / XXX foundation. Everybody is free to do stuff - and does. In the end there aren't actually as many tools as people think, a lot of hobbyists projects who die quite quickly
Is npm the bazooka or C++? As someone who's worked with C/C++ (and its hilarious explosion of tooling -- half-a-dozen at Google alone), maven, bundler and Cargo, I think it's just not accurate to say that the npm ecosystem has a particularly high level of complexity.
Interested to see the emphasis on security, it's definitely something that could use more focus in this area. On that note, are there any plans to implement TUF (https://theupdateframework.github.io/) or similar into Yarn?
npm chooses the first version of a package it encounters to install at the top level. Every other version is installed nested. If you have N uses of version A and one use of version B, but version B is installed first, then you get N copies of the package at version A.
What's the solution for that case though (one which requires no nesting)? I don't think a solution with no nesting exists given the current module resolution algorithm of Node.js, which allows for only a single version of a particular package to exist at a given level.
nesting can't be prevented in that case (at least with the current node require() design). It gets really bad if you have 10 packages that depend on c@1.0.0 and 10 packages that depend on c@2.0.0 -- one of them will install in the root directory and the other 10 will be duplicated. ied stores a single copy of each package@version and uses symlinks which is an improvement. Apparently yarn tried something similar but it breaks some packages (that check where they're located for example).
I could be prevented. Another project does this by using a hash of the project to all all dependencies at one level then symlinks the dependencies from hash to name so it references the correct version.
Symlink that all you want, how is it in any way whatsoever thoughtful and elegant? It's not a real solution by any stretch of the imagination. Myriads of pre-existing systems that correctly deal with this scenario existed at the time that NPM was concoted. Ignored. Considering that node is touted as a cross-platform solution, how well does version@3 deal with filesystems that don't support symlinks? I've run into this: not at all.
NPM version@3 could not be further from the ethos of "engineering." It's an outright hack, and it quickly betrays its limitations. It could have been far simpler, far more considerate of pre-existing wisdom.
no, it just installs everything it can into the top level node_modules. npm@2 would do this to a degree as well, it was just less sophisticated about it.
You can use `npm run env bin-cmd` for that. It's not as convenient as `npm exec bin-cmd` would be, but it might be easier to type than `$(npm bin)/bin-cmd`.
You say "decent performance" and "predictability". What is the basis of this claim? I've heard these all before, but unless you've actually shipped a product using this tool I don't know how you can back this up.
As for performance, Yarn is about 3-5x faster installing all of the dependencies and devDependencies of react-native. The benchmarks are here: https://yarnpkg.com/en/compare. It's much faster in most scenarios, especially the ones that used to take minutes.
The benchmarks test both the hot cache and cold cache scenarios for npm and Yarn. Except in the scenario when node_modules is already populated and there is no yarn.lock file, which is uncommon if you're using Yarn, Yarn is faster than npm in all of the benchmarked scenarios where there is a cold cache.
We're not using Yarn at Tilde yet, but I've been using it on Yarn itself and when working on Ember packages, and it's worked well. As far as performance, we have good benchmarks tracking yarn's performance in a whole bunch of different scenarios (https://yarnpkg.com/en/compare)
Ok, didn't realize he worked on it. Bundler is great so that gives me some hope for Yarn. But in general I think its more stable in the long-term for projects to use open standards instead of vendor solutions for browser package management.
I can understand this sentiment in general, but don't think it applies here. npm is also a private company, and this does work with their registry. There currently isn't a vendor-neutral standard to follow.
There's a lot to like here - deterministic builds are great. However, yarn doesn't currently seem to support installing straight from github or private packages [0], [1].
I'm sure this will be added in the future, but it is currently a dealbreaker for me - I'm using a single package that's just straight hosted in a private git repo, as the package itself isn't ready to be published yet. I'm sure other people are using private packages for more normal reasons (closed source, pre-production, etc).
If yarn actually made it simpler to refer to a local package during development, I'd be on board with that. (I am developing the dependency, but want to also work on the thing that depends on it, so I'd rather just save locally and refresh npm on the outer project, but that's hard to get right - file urls don't always update, the easiest workflow seems to be delete the package from node_modules, and npm install, with the package having a git url rather than a version in package.json).
You are right, we aren't 100% compatible right now. Please track the issues on the Yarn GitHub repo that you linked or better yet: help us fix it. This is a community project and we are excited for the community to jump in and help us out and complete the few missing pieces.
I agree local development with many packages can be hard. For Jest we adopted `lerna`
This. I came here to say this also.
In ruby if you want to patch a repo for yourself you just fork it and make changes and then in bundler point to the git url (often on github). Then run bundle install and you're set.
With npm I've been frustrated a few times whereby you can fork the repo and point package.json to the git repo but after running npm I usually get a bunch of compilation errors.
It would be wonderful to make this as simple as with bundler within the ruby ecosystem, point to forked url and running yarn handles the rest.
I'm very excited about this. Private package support is a must-have for it to get any traction in my company. We are more than ready to leave shrinkwrap behind.
It is quite a congenial response and no disrespect to npm inc, but yarn will be technically far superior (if it's not already), and I can't help but interpret this as the beginning of the end for npm....It was only a matter of time though since there are a number of systemic problems with npm and (in my opinion) it is at the center of the dysfunctional world of modern JavaScript tooling. It did move the ecosystem forward significantly for a number of years, but now something better is required. Sorry to be blunt, just my opinion.
NPM can evolve; Gradle and SBT also didn't turn out to be the end of Maven, which evolved to being more of a repository and dependency management tool with multiple clients.
This would make a lot of sense. I would /never/ make a build using maven again, but having maven central around is a huge plus for the jvm ecosystem and I'm happy it survives. (plus lots of people continue to make new builds with maven and legacy maven builds will be around basically forever)
I'm glad they're okay with it but then again what else would they post? npm has some major, major issues with it and it has barely moved in years. This Yarn looks like it solves quite a few issues with npm and it took another company to build it.
That's insane. It wouldn't surprise me if yarn becomes popular and its proxy to npm slowly turns off and boom, everyone would be migrated and npm would be left with little to no users. Yeah that's probably unlikely for the immediate future but Yarn is positioned perfectly to siphon off npm's users very painlessly.
Well, but remember that time they shipped an SSL cert that broke all updated clients? I'm not sure they move slowly because they don't want to break things, because that would have been a pretty easy bug to catch...
It's more of a failure to double check that npm runs when you change the SSL cert you bundled with it.
I think it's a great example of moving a little too quickly, honestly. Especially since it bricked plenty of npm installs instead of being "just another bug".
Do they necessarily have to break them to move faster? It's a package installer. Yarn is handling locking / shrinkwrapping different but I would argue shrinkwrap needs to be abandoned and re-done anyway so I'm not seeing the risk. It took them, what, two years just to flatten dependencies and move to npm 3? That was actively breaking Windows usage of node!
I love the service they offer. Not a fan of the tools or pace. Npm comes in the node installer so why do they even have to care about improving?
Because they focused on hiring SJW profiles that write codes of conducts and readme's all day long instead of actually software engineers that could improve the tool.
Please resist the temptation to glom ideological snark onto unrelated threads. That's not what this site is for and it ruins threads for everyone by being simultaneously distracting and boring.
Looks like they are trying to write off yarn as just another alternative install tool. I think this is short cited and hope they are taking this threat more seriously internally.
That's pretty well written, though you can see slight hints of bitterness in the tone (though its subtle). Or maybe I'm a little biased, because I've seen some of the non-public reactions, and they were more than a little bitter...
Interesting how they point out how Yarn still depends on them since it pulls packages from NPM's package repo. Kind of like a reminder of who the dominant player still is.
I'm really liking yarn. It solves some of my main issues with npm, it's fast and it provides some great output in the CLI. It's also architected absolutely ingeniously. Seriously they have positioned themselves to seamlessly take over npm's entire business just like that. What do I mean?
So today by default yarn goes through its servers which then proxy to npm's. As time goes Facebook can add more features to its own repo (custom submissions instead of to npm or maybe it does a federated publish to both at first, nice looking pages with community engagement tools built in, better metrics (npm provides me with almost worthless metrics), etc). Then when enough people are using Yarn Facebook could simply...flip the switch. Then boom, zero npm usage.
I would be terrified if I were NPM right now. They've sat on their hands for years not improving npm. I mean npm3 had a silly CLI animation that made all downloads take orders of magnitude longer simply because of the animation!
P.S. if you're trying to use yarn with a custom npm repository just drop a .yarnrc file into your project's directory then add the following in that file:
npm should not be terrified, we've been working with them and they've been very encouraging.
From their perspective it's very difficult to make breaking changes to the client because of the sheer number of people depending on them. (We've faced similar problems on Babel)
Yarn still makes use of the npm registry and all the high quality infrastructure they've built and support they provide for the ecosystem. They are a critical piece of this.
I mean I understand the intent but it turns npm into basically a dumb pipe / just infrastructure. Every business that gets reduced to that struggles to get out of that space and expand.
Granted it's far too early to write npm off. But with how slow they've moved over the years I'm unconvinced that yarn won't take over the space unless it runs into some bad problems. Npm 3 and its launch was an absolute mess and, ultimately, it's almost just a package managing tool. I am unconvinced that breaking changes is much of an issue if at all for them. They could abandon shrinkwrap into a better system and I think everyone would be happy for it; no need to keep propping up that awful system.
As Yehuda pointed out, such a thing won't happen because Yarn was launched as a community project from the beginning, and a lot of members contribute to both, so even if Facebook decided something sketchy, the community would either stop it from happening or continue with a forked version.
> the community would either stop it from happening or continue with a forked version
Would they, though? If Yarn turns out to be the better tool that doesn't change the way they work (beyond solving some pain points) then would they even care? Facebook could even offload the hosting to some open source group like Apache.
It's purely anecdotal but a lot of people I have worked with would love to see an alternative to npm. It's just slow and the features it gives to authors is pretty limited (I want better metrics, please!). I'm not even sure I would call it sketchy, it would just simply be phasing out something deprecated (I'm not talking about a swift take over more like a slowly-boiling-the-frog-in-the-water takeover).
It's only my opinion and if Facebook really doesn't want that to happen then I guess it won't but it's pretty easy to imagine it, IMO.
This is not a problem with the package manager. This is a problem with complexity.
When did it start becoming reasonable for a front-end only part of the MVCC pattern to have 68 dependencies?
Or for a transpiler like Babel to add 100k+ files? I'm sorry I just find it ridiculous that instead of taking a look at the disease (unbounded complexity), we are looking to engineer our way out of the problem by creating a package/dependency manager that "scales". Are the frontend problems at Facebook of showing HTML forms and buttons on a page really that complicated to warrant such a behemoth of a system?
This harkons back to the days at LinkedIn where I spent my time in a misguided effort to create a distributed build system because our codebase had grown so massive that it literally took a distributed system to build it and still release within the day. I feel bad for people working on these systems because while it is fun and "technically interesting" to solve "problems at scale", you're really just digging the ditch deeper for your company. You are putting lipstick on the 500 lbs pig; making room in the dump to pour in more technical debt.
I don't think anyone ever imagined we'd be in a situation where a javascript framework is approaching the complexity of an operating system in terms of individual source files and SLOC.
The Javascript ecosystem has to deal with unrelenting, multi-decade backwards compatibility requirements; supporting numerous, inconsistent implementations; and an incredibly wide set of stakeholders and participants with diverse motivations when advancing the spec.
Do any other software platforms out there face the same magnitude of complications?
To speak to one of your primary examples: Babel is freaking huge and somewhat absurd, but is there any other way to advance the Javascript language? Fortunately, this is only a development dependency, and the client doesn't have to see it.
On a happier note, these are good problems to have. This is another indicator of how successful the web platform has become. You can build some absolutely incredible projects these days, and they are instantly available to billions of people who are operating thousands of different hardware devices (with different specs and dimensions) running dozens of operating systems. No gatekeeper, bureaucrat, or corporate curator must approve your work. Just simply deploy it, and anyone with the URL can access it. It is a mind-blowing human accomplishment. I look forward to seeing how the web continues to progress over the coming decades.
Have you seen some of the things we are building for the web these days? Full applications, with multiple "pages", server-side rendering above the fold content, minified builds that load not only content--but code as well--on demand from the server, etc.
Absolutely: this is overkill for your blog or portfolio website, but part of the rising complexity with the tooling and dependency system is due to the rising complexity of the applications themselves.
I use JS+Node+NPM for my day job and many side projects.
Initial thoughts:
- Why didn't Facebook contribute the updates to NPM directly?
- They are coupling a package manager and registry proxy; the latter has many existing implementations already
- The differenced between Yarn and NPM+Shrinkwrap do not seem substantive; NPM made the design decision to use a sometimes non-deterministic install algo in NPM3 to speed up install times - when network is involved, there is a serious trade off between idempotency and speed
In general, FB seems to love building their own versions of existing tools:
- Flow, introduced 3 months after TypeScript
- Nuclide instead of Atom/Sublime/VSCode
- Jest instead of Jasmine/Mocha
- DraftJS instead of (insert of of the existing 100 text editors here)
- ...
I get that these are useful internally at FB, but I don't think they help the community much. It would be better to work with existing tools and contribute back to them than to reinvent the wheel every time something doesn't work perfectly for their use case.
I get that FB uses these as recruiting tools, it's useful for them to have rights for these projects, and it gives their engineers something to do and be excited about, but I do not want a whole dev tool ecosystem controlled by big FB.
Do you think Flow was built in 3 months as a reaction to Typescript? Not that it was a big enough problem that two different groups of people independently decided to try to solve it?
EDIT: disregard this, Flow was introduced much longer than 3 months after TypeScript. The blog post introducing Flow mentions TypeScript, and is probably worth reading for background [1]
> Nuclide instead of Atom/Sublime/VSCode
Nuclide is an Atom plugin.
> Jest instead of Jasmine/Mocha
Jest provides a superset of Jasmine functionality, and uses Jasmine as its test runner.
> Do you think Flow was built in 3 months as a reaction to Typescript?
They probably knew that TS was in progress when they started, but I don't have a window into that. I did ask some Flow devs a few weeks ago if they are interested in merging with TS, and they answered with a categorical no.
I can't speak for either project, but personally I would say that I don't think static typing in JS is a solved problem, and at this stage to me it seems a monoculture would be much less fruitful than a competition of ideas.
I am not a computer scientist, but I have heard people repeatedly argue that Flow sets out to create a "sound type system" which TS intentionally does not.
What this means in practice I don't fully understand but it seems to result in enough architectural differences that it apparently makes sense to have both despite the superficial similarities in syntax and behaviour.
I think the issue is that TS is non-modular and dictates javascripts reach. It behaves like no other tool and blocks others from working. Take something like React/Redux for instance, which are built around modern day javascript, but if you transpile any example on the official Redux page right now you will see TS bail out because it can't understand some of the syntax.
Flow on the other hand solves one problem and solves it well. It behaves as any other tool would.
You can only do as much as TS allows you to, unless you route through babel in a second pass. TS is a tool that controls to which extend you can use language features that are drafted. They do pick drafts here and there, but others they omitt, object spread for instance which is very common for Redux.
Still not sure I understand. All the code in that link is valid TS - TS is just a superset of JS.
Object spread is not part of the ES7 spec. It is a proposal which has not made it to stage 3 yet, and might still be dropped. That said, there is a TS issue tracking adding support for it: https://github.com/Microsoft/TypeScript/issues/2103.
Hey, I work on Yarn and Jest. Let me respond to why we built Jest: It solves a ton of problems no other test runner solves and it works well at Facebook's scale. We open sourced it more than two years ago but only recently started to promote the project in open source more actively because it is working so well at FB.
Finally, test runners tend to get stale. Any large organization will run into trouble with any kind of tooling as companies scale up engineers or codebases. Testing is no different and we aimed to solve those problems for Facebook; getting to open source our solutions is an added bonus and we are excited to work with the JS community to build the best tooling we can possibly build.
I for one am thankful for Jest and Yarn. I recently switched over to Jest and I love most everything about it and the built in watch with caching reduces friction immensely (now if we can just allow custom preprocessors to return promises that would make life easier for integration of things like rollup). And yesterday I flipped over to using yarn and it definitely solves most of my pain points with npm.
So yeah, I'm definitely on the "keep reinventing" side of this debate.
IMO it's the "javascript way" to invent new tools instead of trying to improve others (when it makes sense to do so).
Let me explain that a little bit... The javascript "ecosystem" takes the unix philosophy to the extreme in many ways. And one of the points of the unix philosophy is to try to avoid "bloating" tools with tons of options, and instead trying to create new tools where appropriate.
Many of those are perfect examples of where "improving" a current tool with the changes they wanted would end up with a more complicated and difficult to use tool.
- NPM makes tradeoffs that the majority of developers in the JS world want right now. Like it or not, deterministic installs isn't something that most people want or need (at least, they don't know that they might want or need it). Making a new tool that anyone can use when that is needed is a great solution, and it avoids bloating NPM with a hundred little flags and tons of code to cover edge cases. It's the perfect example because anyone can switch to using yarn when it's needed without any extra work.
- Nuclide is actually just a layer on top of Atom. They wanted a bunch of things to all work one way, so rather than try to get Atom to change to include what they wanted, or require developers install a ton of extensions, they packaged them up on top of Atom as one system.
- Jest had different goals at the start. Testing react components was ugly and difficult, and Jest wanted to solve that first (IIRC, i'm not very familiar with jest unfortunately). Again, it's something that can be used if needed, and can even be combined with other testing tools if you want (we almost did that at my work, used Jest to test the front-end components, and our current mocha+friends setup to test the "business logic").
> The javascript "ecosystem" takes the unix philosophy to the extreme in many ways.
Bullshit, nix systems rely on stable global dependencies, not 50 versions of the same library installed locally. I don't buy the "unix philosophy" excuse. The nodejs community just doesn't care about API stability which is why there is that inflated number of modules with 90% never maintained more than a year.
Come on now, I said that it's like the unix philosophy in many ways, not that it's unix...
- Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".
Which is what this thread is about.
- Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
JS has wonderful stream support, and it's used everywhere in many tools. This is probably where JS is the weakest out of these.
- Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.
I mean, if JS isn't this taken to the extreme, i don't know what is...
- Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them.
This also describes JS to a T. Tons and tons of tools, each often only usable by a small number of people in niche situations.
Now i'm not saying that JS is as well designed as unix, i'm not saying that it's exactly the same as unix, i'm not saying that Unix doesn't do things better, or that JS doesn't have it's warts. I'm just saying that it's the "unix philosophy" taken to the extreme. It might not be good, it might not last, but it's working out pretty well for many people right now.
+1 for contribution to npm. Especially given npm isn't completely deterministic there is no reason for not pushing for these changes into npm itself.
Given npms years of crap I'm ready to give it a shot anyway. This might be one of FB's projects that actually gets some traction, especially given its developed in collaboration with other big names.
> Yarn, a collaboration with Exponent, Google, and Tilde.
They should mention this at the very beginning. Multiple big players investing in this package manager means that we should maybe inspect a little bit more before chanting xkcd.com/927.
927 was also my first thought. But what made me to reconsider was not more than one big name behind it (but it helped), but the fact that they rely on npm backend and did not reinvent everything from scratch.
Basically, yarn is npm client done right reusing the same npm package repo.
Exponent has been testing out Yarn for about a month, and it's been an incredibly pleasant experience, both in working with the Yarn team and using the tool.
Exponent uses a monolithic code repository (rather than many small code repos) to manage our many interdependent projects. Our repo is actually setup very similarly to Facebook's mono-repo, though obviously considerably smaller in size. We were encountering _many_ of the same problems as Facebook with our NPM setup -- long CI/CD build times, indeterminate node_modules directories that caused our projects and tools to work for some people on our team but not for others, and the inability to do "offline-only" npm installs in CI.
We actually talked with our friends at Facebook about these problems, and tried many of the same approaches they did -- shrinkwrap everything, checking in node_modules, uploading the node_modules folder to separate place (we used another completely separate Git repo), etc. All these approaches either didn't work well or were difficult to maintain, especially on a small team.
Yarn has fixed all these issues for us.
One not-yet-super-publicized feature of Yarn (though I'm told there is a blog post coming) that has been super useful at Exponent is the ability to build a distributable offline cache of dependencies. We have a "node_modules-tarballs" folder in our mono-repo that contains tarballs of all the dependencies for every single project that we maintain at Exponent. Yarn will pull from this cache first before fetching from the NPM registry. This offers HYPER-predictability...it's a level of confidence over and above the "yarn.lock" file, and let's me sleep well at night knowing that my team can get up and running quickly no matter where they're working, how reliable their internet connection is, etc. No more worrying about "did I forget to update the shrinkwrap" -- things just work.
As JS developers, we all, beginners and experts alike, can experience this "JavaScript fatigue" that we always hear about. Fighting with tools is never fun. We just want to write code!
As we've started to use Yarn at Exponent, both on our engineer's machines as well as in our continuous integration and deployment environments, we've at least had one less tool to fight with. Yarn works, works fast, uses the great parts of the NPM ecosystem that we know and love, and most importantly, is _predictable_.
Definitely looking forward to that blog post from the Yarn team, but I'd definitely be interested in your own comments on how to use that "checked-in tarballs" approach.
> [...] the ability to build a distributable offline cache of dependencies. We have a "node_modules-tarballs" folder in our mono-repo that contains tarballs of all the dependencies for every single project that we maintain at Exponent.
That's exactly what we need! How do you tell Yarn to build and use this offline cache?
As someone who just started with Spark lately, that was my first thought after being confused for a minute. It's not like people at facebook and google don't know about Apache projects. I wonder what is their motive to not avoid such name collision? I see these clashes a lot with small projects on github -ironically most of the are javascript- but rarely on a big scale like this.
generate name
compare affinity in concept space
if affinity > .5
repeat
yarn is framework for doing X in web programming, and this a framework for doing Y in web programming. not sure if dependency management and cluster system management are far apart enough.
Search "go" on google, and the first link is the programming language. Search on any other search engine, and as one might expect, the definition or the game of go is ahead of the language.
wow already one of the features i'm loving in yarn is that it tells you which package is firing warnings about package incompatability.
warning electron-prebuilt-compile > electron-compilers > jade@1.11.0: Jade has been renamed to pug, please install the latest version of pug instead of jade
in npm, that would have just said the part after "jade@1.11.0" which was really vague and didn't really make you want to "fix" it because which npm module do you have to go into? who knows because npm (the package manager) didn't tell you.
i seriously want npm to stop warning me of outdated deps 3 levels down my hierarchy. so what if some shitty old unmaintained lib we started relying on a few years back use lodash v3.x? i'm not going to worry about it as long as it works.
Just experienced the same. Loved it! npm was giving vague errors about dependency's dependency. Now it's so much clearer that which package needs to be updated. Also the speed of installation has reduced a lot !
It looks like this addresses the biggest issues people have with npm's CLI, and it's coming from such huge names: Facebook, Google, and Tilde. Reproducible builds are a _huge_ issue, and this gives you that. Looks great!
One interesting little tidbit I found from diving into the source:
Oh hi Steve! https://registry.yarnpkg.com is just a proxy to the normal npm registry provided by Cloudflare so we can add additional caching and work on network performance (lots of stuff we could layer on top). It should hopefully help those in China as well which would be awesome. But everything you do still lives on the normal npm registry.
hi! they are running a mirror of the npm registry which contains all the package data but none of the permissions/user mgmt. this means that they are still dependent on npm infrastructure.
npm, Inc strongly encourages people to build and use mirrors. for example, cloudflare (where the mirror lives, and where seb used to work) has been running https://npmjs.cf/ for a long time.
That's not 100 % accurate. Arguably symlinks are kinda weird under Windows, but that doesn't mean symlinking node_modules doesn't work (you just need admin privileges).
This is a really big deal. We have a very large/complex web application & API with a large number of dependencies, and not a single npm alternative (pnpm, ied, npm-install) could figure out the tree properly.
It has been working well with our npm proxy and is basically a faster, better-deduped drop-in replacement that removes our need for npm-cache, npm-check-updates, and all of our shrinkwrap hacks.
This is finally the npm rewrite we've been waiting for.
In looking at the yarn repository, it's nice to see a clean, modern build, ES6 + async/await, Flow typing, and so on. This is what a well-maintained JS project looks like in 2016 that's built for production use.
npm, for all its flaws, was a revolution when it was first written. But since then, it has been large, unruly, difficult to maintain, and has largely unaddressed many important production use cases. It is nice to see a group of companies using Node in production take the lead on addressing problems that really matter to production JS apps.
My initial tests with this are very positive (I tried this use case here: https://github.com/npm/npm/issues/10999 ). It handled it quite well and we'll definitely be considering this for our package.
I'm curious about the "flattening" approach though -- and why something like ied ( https://github.com/alexanderGugel/ied ) wasn't done instead. ied only ever installs one copy of a module, but allows each module to think they have their own copy of it through clever symlinkery (essentially making it FEEL like npm 2 while getting the benefits of npm3/yarn). I'm not a fan of (explicit) flattening for a number of reasons:
1. Mimicking recursive dependencies while still only having on physical copy on disk has the benefit (or downside I suppose) of isolating and protecting different module's dependencies from each other. AKA, if you require("x").mutate = true, this won't be seen by other modules. Again, depends whether you like this feature or not, but with node's treat-symlinks-as-different-files configurable switch, you can actually control it yourself in ied I believe.
2. You don't run into tricky situations where require("x") can accidentally pick something up you didn't mean to (since it is now in the parent path of ALL your dependencies). So, if A depends on x, and you install B, it will be able to require("x") (maliciously?) and modify it, affecting A's behavior. Perhaps this is considered a strange possibility, but it gets particularly weird with peer dependencies, where you expect the user to choose their own top level dependency, but this can now be accidentally "chosen for you" by a subdependency of another package.
3. I just don't like clicking on my node modules folder and seeing 100 top level directories when conceptually I typed "pkg-manager install one-thing". This can lead to you yourself making the above mistake, by require("x") and having it "just" work, then having it "just break" when you remove the dependency that included x.
I'm willing to be convinced though (on RunKit we had module-fs simulate npm2 install behavior, super simple recursive installation, conceptually similar to what ied does).
Hey! I'm Sebastian McKenzie (@kittens on GitHub) and I'm the lead develop on Yarn at Facebook. We initially used a method similar to ied when we first started experimenting with internal usage at Facebook.
We ran into a lot of issues with OS compatibility around symlinks and existing tools not supporting it. Windows lacks support for symlinks on non-Admin accounts, NTFS junctions work but they're slightly quirky in their behaviour and don't behave exactly the same way.
A lot of tooling relies on the existing node_modules structure. ESLint for example will load tools relative to itself which means that they need to be reachable in the tree, you can solve this with hardlinks but there's caveats with that too. Existing tools aren't very well suited for handling cycles either which in a system like npm are extremely common as a transitive module could depend on one that's higher in the graph.
We care a lot about existing ecosystem compatibility and because of these limitations we went with the current status quo. We'll continue to explore others options though and if there's a chance we can do it without any of these caveats then we're more than happy to explore it.
As Sebastian said we are more than excited to explore what we can improve in the future. If you are interested, please start a discussion on the GitHub issue tracker: https://github.com/yarnpkg/yarn :)
"The React Native package.json currently lists just 68 dependencies, but after running npm install the node_modules directory contains 121,358 files."
That, to me, is what is wrong with npm. The problem stems from node.js not coming with "batteries included" so there is a proliferation of tiny libraries that do the most trivial things.
> That, to me, is what is wrong with npm. The problem stems from node.js not coming with "batteries included" so there is a proliferation of tiny libraries that do the most trivial things.
This is in no way a fault of node.js but of the whole JS standard. People these days use npm modules to run in a browser and ship stuff bundled together with webpack (or other bundlers), so even if nodejs had a proper stdlib, you'd still need to depend on a polyfill so that your stuff works in a browser environment, too.
And due to the fact that even if a sane stdlib would ever be standardized, it would take YEARS of time until it reaches significant market share (looking at you, Android, Safari and IE), so you'd always have to ship a polyfill.
That's kind of a red herring when it comes to NodeJS. Webpack and the 50 other tools you need to run it are in a completely controlled environment. Most of the build tooling is working around issues with NodeJS, and NodeJS doesn't need to wait for anyone to come up with improved APIs around filesystem interactions and build tooling. This is purely build-time stuff.
To make matters worse, each package gets its own node_modules directory. Do a grep for 'async' in your node_modules. Mine contains ~20 instances of it. Some of those, probably, are the exact same version.
Going further down shit creek, each project I work on has its own node_modules. Which duplicates all of this crap, ad nauseam.
Do I really need React and Babel and jQuery and Webpack installed for every goddamn repo? You're not a package manager if you're not managing packages!! You're a glorified wget.
> To make matters worse, each package gets its own node_modules directory. Do a grep for 'async' in your node_modules. Mine contains ~20 instances of it. Some of those, probably, are the exact same version.
As I understand it, the related problem is the poor support in build tools for dead code elimination which encourages people to publish libraries that contain a single function like left-pad.
Yarn is particularly great for front-end web apps because of its flat installation mode.
ES6 module imports and HTML Imports both require that dependencies are imported by URL. This means that the only reliable way to import another module is by relative URL, like:
import * as $ from '../jquery/jquery.js';
This requires that packages are installed flat, as siblings. Yarn is going to enable native JS modules and projects like Polymer to use npm instead of Bower. :)
You can use webpack or JSPM to resolve directories at the top level. That way you can just import "web/X" even if the web/ directory isn't a sibling to your current file.
It's not very helpful if you are distributing an NPM package, but for people who aren't authoring libraries, its a godsend.
That was basically what I was suggesting. I'm not sure what the other option really is. If you're working within the browser (implied by this being a frontend), then you have to use a build tool since javascript modules/imports haven't landed in any browser yet.
> If you're working within the browser (implied by this being a frontend), then you have to use a build tool since javascript modules/imports haven't landed in any browser yet.
For now, but native JS module loading is right around the corner, and at that point I don't want to have a required tool to load my modules.
Currently I use HTML Imports, which don't require tools because they use relative URLs. It works really well, so I hope to use that pattern with JS modules.
--flat is the cli option, but there's also "flat": true support in package.json. Used at the top level this forces a flat install, but used in a dependency it means that that package requires a flat install and it throws an error if it's not.
This means that HTML imports and ES6 modules can force flat installs and start moving the front-end package ecosystem in that direction.
Yarn is currently being powered by a service running at https://registry.yarnpkg.com - at least that's what's referenced in the yarn.lock file that's created.
I see no documentation nor code for the service powering this, nor is there a way to tell the command line tool to use a registry at a different address.
Aren't we just replacing the dependency on npm, Inc.'s registry with a new dependency on Facebook Inc's registry with this?
Yes. The client has a few advantages, but the registry is still closed and while with npm there's an option of running a private registry if you pay them, here, there's no option at all.
that registry is a mirror of npm's registry. the mirror contains all package data but does not include any of the permissions or user management, so this tool still depends on npm infra for both that, and as a replication source.
registry.yarnpkg.com is a proxy, it doesn't contain any package data except for what gets cached by Cloudflare. Just a Cloudflare'd "CNAME registry.yarnpkg.com. registry.npmjs.org."
Is it just me, or does Facebook seem to create new competing tools and technologies more often than the other majors?
My inner-cynic makes me feel like everyone at Facebook is really young, as I've noticed a trait of inexperienced developers is to try to create new things over finding and improving existing tools.
In reality it could be something as simple as Facebook releasing more FOSS software than others, or maybe their releases having more visibility.
Either way, feels like Facebook like to pave their own way, for better or worse.
We don't think this way. We needed to solve this problem for Facebook, explored alternatives and made the case to build Yarn. We work at Facebook to solve Facebook's engineering challenges; when we can collaborate with other companies and the open source community, that's an added bonus.
Brilliant. I've been trying to tackle this problem for the last year or so with shrinkpack (https://github.com/JamieMason/shrinkpack) with _some_ degree of success and a nagging feeling that much more could be done.
It's great to see attention is being spent on what is a really important area, one which has been largely under-appreciated in Node.js until now (aside from for a short period after the left-pad incident).
Kudos to Facebook, I look forward to giving it a try.
The justification for Yarn starts by ignoring the existence NPM Shrinkwrap. It then proceeds to mention Shrinkwrap and refer to its supposed short-comings. Whilst there absolutely were issues with Shrinkwrap in the past, they've already been addressed by NPM.
NPM is slow, perhaps Yarn is faster and has some extra features. But why these couldn't be contributions to NPM seems like a serious case of NIH...
No. NPM is awful in ways that could not be fixed without braking changes. They have a need, so they satisfied it. A need which, by the way, every one of the dozen-or-so Node-using projects I've worked on have shared.
> Whilst there absolutely were issues with Shrinkwrap in the past, they've already been addressed by NPM.
This isn't even remotely true. Shrinkwrap is still a usability nightmare and doesn't begin to address the non-deterministic install behavior that crops up when you add new dependencies.
> why these couldn't be contributions to NPM seems like a serious case of NIH
There's no way that they could have got these changes in to NPM in anywhere near the time it will take for Yarn to become popular; if at all! Politics (or the avoidance thereof) are practically _the_ perfectly valid reason to justify NIH.
I'll wait a while for Yarn to bake, but personally, I hope NPM dies a swift and total death.
> No. NPM is awful in ways that could not be fixed without braking changes. They have a need, so they satisfied it. A need which, by the way, every one of the dozen-or-so Node-using projects I've worked on have shared.
Examples, please.
> This isn't even remotely true. Shrinkwrap is still a usability nightmare and doesn't begin to address the non-deterministic install behavior that crops up when you add new dependencies.
Again, examples, please.
> There's no way that they could have got these changes in to NPM in anywhere near the time it will take for Yarn to become popular; if at all! Politics (or the avoidance thereof) are practically _the_ perfectly valid reason to justify NIH.
Did the Yarn developers make any attempt to contribute to NPM at all? If they did and were turned back, then sure, I understand. However, everything I've read about Yarn thus-far is just vague finger-pointing at NPM.
I'm by no means suggesting NPM is perfect, for one I find it incredibly slow. However, surely that can be optimised.
However, regarding dependency version locking, which is the problem Yarn is trumpeting, what are the actual issues with Shrinkwrap? I'd like something specific. Since Shrinkwrap was updated to keep a flat sorted structure (a while ago) I've been using it no fuss at all. What are the specific issues people are running into now?
I've seen comparisons with Bundler, and how Yarn will be more like Bundler's Gemfile.lock files. However, the thing is the current version of Shrinkwrap works almost identically to Bundler's lockfile e.g. When I update a dependency the shrinkwrap file automatically updates (at least that's the default behaviour).
> > ways that could not be fixed without braking changes
> Examples, please.
The first and foremost: Version pinning should be the only way it works. Obviously _requiring_ shrink-wrap would be a breaking change.
> > Shrinkwrap is still a usability nightmare
> Again, examples, please.
Having to explicitly call shrink-wrap is just silly. Devs on every time I've used shrink-wrap with frequently forget to do it and it causes all sorts of "works for me" problems later. My team has also experienced all kinds of brokenness related to private NPM repositories and shrink-wrap non-determinism.
> Did the Yarn developers make any attempt to contribute to NPM at all
I don't care about this at all. Just because you got there first doesn't mean you should have some kind of special claim to solving the problem.
> The first and foremost: Version pinning should be the only way it works. Obviously _requiring_ shrink-wrap would be a breaking change.
> Having to explicitly call shrink-wrap is just silly. Devs on every time I've used shrink-wrap with frequently forget to do it and it causes all sorts of "works for me" problems later.
Really? An entirely new tool because you didn't like typing "npm shrinkwrap" once, just once per project? I... I don't even know how to respond to that...
> My team has also experienced all kinds of brokenness related to private NPM repositories and shrink-wrap non-determinism.
I'm unclear on what you mean? Do you mean in the past? Cause if so, sure... but hasn't this already been addressed?
> I don't care about this at all. Just because you got there first doesn't mean you should have some kind of special claim to solving the problem.
This is an all too common attitude in the JavaScript community. Hence the plethora of poorly maintained, duplicated NPM packages.
I'd rather humanity not waste the man-hours duplicating effort. Do something productive with the time you saved.
As mentioned, if for some reason the NPM guys weren't co-operative, then I understand, however that doesn't at all seem to be the case.
> An entirely new tool because you didn't like typing "npm shrinkwrap" once, just once per project?
I don't know what kind of projects you work on, but new dependencies get added with some regularity on any large node project, requiring a shrink-wrap every time.
If you've never experienced the forgetting-to-shrinkwrap problem and its interplay with team dynamics, then I envy you. But I also must give up trying to explain to you, since this drives to the heart of the problem and I must conclude that we simply do not have enough common experiences for you to ever understand. The developers of Yarn et al clearly have lived this nightmare.
> Do you mean in the past
Where the past was < 4 months ago. If it has been fixed since that, I don't care: too little, too late.
> This is an all too common attitude in the JavaScript community.
An equally big problem in the JavaScript community is that core pieces of infrastructure have been "claimed" by people who have absolutely no idea what they are doing, but are happy to community-coordinate their way to some kind of idealistic open source victory.
> weren't co-operative
Again, I don't care if they were willing to co-operate. I have zero confidence in them.
> I'd rather humanity not waste the man-hours duplicating effort.
Give up on this dream. People are going to re-invent everything from now until the end-of-time, and rightly so.
For what it's worth, the name is really cool. I just looked up the etymology of Node and it comes from Latin `nodus` meaning knot. So if a Node program is the knot, then the stuff that makes it up and that it comes from must be the yarn! Language is fun.
I just tried to install yarn and ran it to against a relatively large React application. What's interesting is that it found an incompatible module and could not proceed, whilst npm works fine.
The article states that running `yarn` is the same as `npm install`, which doesn't look like the case. At that point, I'm curious what are the differences.
Sorry about that, there are very likely still some bugs out there we weren't able to catch in all our testing which will be ironed out by the community in the next few days. Can you open an issue with details about which package failed to install? https://github.com/yarnpkg/yarn/issues/new
Recently I got annoyed how hard it is to use shrinkwrap in npm and started working on a npm wrapper that would make npm as easy to work with as Ruby Bundler by copying its workflow as closely as possible (https://github.com/szimek/bundlerjs). Thankfully, I don't have to develop it anymore ;) Big thanks to all Yarn developers!
Overall looks good, but it's a shame that flat mode is opt-in instead of the default - this is the sane way of doing package management. Would have been nice to flip the options over so you have to explicitly allow multiple versions of the same package - which you could automatically enable if you're converting from npm.
I don't know JS and I've never used these js package managers, but are these issues such that they couldn't have been fixed in the original package manager? Smells like NIH that is just going to create head ache and misery for everyone in the JS community in the long term.
I hope this becomes the default way to build npm projects... and then npm install becomes yarn :-) Exactly reproducible builds is a godsend to the js community.
This looks awesome. But I have to wonder why create a whole new project rather than fork or upstream these changes to NPM? It doesn't seem like it's doing anything fundamentally different or outside of NPM's scope of responsibility.
The official npm cli is quite old and evolved along with all the different coding styles and architecture choices of the node and JS ecosystems.
It's also full of dependencies on originally purpose-built modules that suffer from the same problems. This compounds various issues, resulting in long-standing bugs like `npm publish` sometimes not actually including all files in the tar bundle it generates[0].
NPM 3 was a partial rewrite but it's still built on the same mountain of code as NPM 2 and 1. npm Inc developers have in the past complained about not being able to address issues for a lack of resources despite having a ton of developers on the team[1].
The official npm client also doubles as an account management CLI to the npm registry. Which apparently npm Inc now considers a design flaw as they recently started creating a standalone tool for managing other aspects of their commercial services[2].
And finally it's important to understand that having a new package manager that is not tied into npm Inc means it's easier to replace the npm registry as the official node module registry in the future. Which would allow the community to get rid of the privileged position npm Inc currently holds in the node distribution (which is otherwise governed by the Node Foundation).
Rewriting the client from scratch actually turned out to be the sanest approach in this situation, I think.
fwiw, only 2 of those humans work on the CLI, (3 if you count their manager, who spends a lot of time managing). other people work in sales, marketing, engineering on the website, engineering the registry, community + docs, etc.
wombat is not a tool for managing our other commercial services! it's a way to manage our webhooks product, one of many products we have, though very certainly the smallest.
yarn depends on the npm registry, which depends on npm, Inc making money to keep it running.
Thanks for clarifying. But that does show that having a company fund two full-time developers to work on the CLI wouldn't have been sufficient. Yarn was mostly built by Seb (403 commits) starting in late January and Konstantin Raev (121 commits) with contributions from several others (<30 commits each). The commit graph[0] actually shows development mostly paused in April and May, putting the total development time closer to half a year.
This most likely includes collaboration with the other people supporting the project (both Seb and Konstantin work for FB) and a lot of architectural planning -- aspects, which likely would have required an even greater investment of time if building on the legacy codebase of the official npm client (not to mention having to negotiate all changes with npm Inc directly).
> wombat is not a tool for managing our other commercial services!
Sorry about the misunderstanding. When wombat came out a lot of people wondered why wombat was a standalone tool when practically every other aspect is part of the npm cli and from what I gathered from npm devs at the time the decision was more of a technical one (different people working on different tools for different things) than a managerial one (different products deserving different tools).
> yarn depends on the npm registry, which depends on npm, Inc making money to keep it running.
Yarn currently uses a downstream mirror of npm as the direct source of truth, with support for private registries being actively worked on to support the private parts of the npm registry and products like npm enterprise or free alternatives like sinopia.
However that is obviously an implementation detail. Decoupling yarn from the npm registry by going through an intermediary allows experimentation (as the npm blog post points out) and ultimately allows avoiding a hard dependency on npm Inc as a company.
If npm Inc were to shut down right now, yarn would still have a read-only mirror. I'm sure Facebook and Google have enough experience solving problems at scale that they could build a public registry alternative if necessary (such as if npm Inc went out of business tomorrow).
I'm not saying yarn will kill npm Inc. But yarn can kill the npm client and replace it with a community-supported alternative (under an open governance model similar to Ember), which in turn makes it a lot easier to fade out the reliance on the public npm registry maintained by npm Inc -- benefiting both the community (by eliminating a single point of commercial failure in the node ecosystem -- exactly like joyent was to the node project) as well as npm Inc (by eliminating a major drag on development resources and cost center).
The only drawback from such a move to npm Inc would be the loss of its privileged position in the ecosystem, the "first mover" advantage of being the place to go if you want private packages (because it's more frictionless than any of the alternatives -- exactly like GitHub in the git hosting space).
The first release of Node.js was in 2009. "Old" is relative.
Nearly all modern JavaScript build tools are built on Node.js and NPM pre-dates most of them.
The only other thing I can think of that has seen as many shifts in JS development practices as NPM is Express (*2009) and that library has far smaller complexity and has arrived at a drastic reduction in features and a far tighter scope.
The NPM client on the other hand was originally cobbled together by Isaac and then grew organically as the NPM registry massively grew over the months and years of node's success.
The NPM client and registry are the epitome of technical debt. Calling it "old" is the most charitable way to phrase this.
Doesn't npm effectively replicate bundler's functionality (poorly) via shrinkwrap? Why not fix that feature rather than replacing it with a separate utility?
Its disheartening to see so many duplicate efforts to solve the same problems everyone faces. Instead of supporting an existing open-source project that attempts to solve the problem in pretty much the same way (jspm), a conglomerate once again builds their own from scratch. I do not think the underlying motivations for doing so are questioned enough. Sure there is control, but it is of course anyway-you-slice-it an aggrandizement of their brand. We should be ashamed as a community to support the idea of yet ANOTHER client side dependency management system. How oh how did we ever solve this problem before Facebook came along and made Yarn in late 2016?
I don't think it's fair to compare jspm and Yarn. jspm is an interesting project, but is tightly coupled with SystemJS. Telling everyone to rewrite their apps to use SystemJS and jspm is a huge lift.
Yarn, on the other hand, has focused on npm compatibility from the get-go. The entire point is to be a drop in replacement, with no changes (or maybe very very minimal changes) required to gain a lot of benefit.
In this slice of the industry where everyone is told they have to rewrite to the latest thing every year or two, I find it very refreshing to see Yarn's focus on backwards compatibility, improving existing infrastructure, and having a clear, painless adoption path.
I don't agree with any of your statements, and I think they are inaccurate.
jspm aims to be type-agnostic with respect to modules (supports all major types, hence why it is called a universal module loader). jspm certainly doesn't advocate rewriting your modules (you can use exports, AMD, ES6, whatever). In fact, when the browsers natively support module loading, systemJS will go away (which is really just a polyfill for this functionality).
yarn global add typescript
yarn link typescript
error No registered module found called "typescript".
info Visit http://yarnpkg.com/en/docs/cli/link for documentation about this command.
what?
edit: also,
$ yarn add typescript
yarn add v0.15.1
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
error ENOENT: no such file or directory, open 'C:\Users\******REDACTED******\@types\react\index.d.ts' at Error (native)
info Visit http://yarnpkg.com/en/docs/cli/add for documentation about this command.
The node_modules folder is my least favorite part of the JS ecosystem. I realize now that's mostly because of npm-cli.
npm-cli's job was simple. Take in a `package.json` as input and output the same `node_modules` folder as fast as possible. But somehow it couldn't do that even after getting 6 years old. On top of that, every now and then with when you ran `npm update` or `npm dedupe`, it messed up the `node_modules` folder with extraneous and invalid packages.
Just installed it and ran it on a largish project ( https://www.bugreplay.com ), the parallel downloads really sped things up and outputted some great diagnostic info about problematic dependencies. BugReplay is a webapp for recording webapp bugs, so having some confidence that our dependency chain isn't introducing our own bugs would be awesome :)
Well, this is a problematic case of "throwing out the baby with the bathwater". Semantic versioning isn't the problem here (the lack of deterministic install trees is!), yet it's thrown out pretty much entirely, without even so much as considering why it exists in the first place.
I wonder whether Facebook realizes just how much damage they are going to be doing to the JS ecosystem with this approach.
As I understand it, it automatically locks in the (exact) version of everything you install, rather than having this as a separate, optional feature (like shrinkwrap does, despite it not working well - but that's an implementation problem).
The big deal about semver and how it's implemented in NPM is that there is a choice of how to deal with it - either you automatically allow compatible updates and risk an occasional maintainer fuckup, or you explicitly pin everything and take on the burden of constantly updating your versions manually and tracking security releases.
The latter might work for Facebook and other large organizations, but for many smaller developers, the consequences of an accidental break are much, much less serious than those of constantly having to keep pinned dependencies up to date. End result: people don't bother at all, and insecure versions never get upgraded for the majority of the userbase.
This, in turn, removes the incentive for package authors to do anything with semver at all, once Yarn's approach becomes commonplace. Why bother with semantic versioning if everybody is expected to manually check and pin their dependencies anyway?
Facebook also needs to take into account their influence over the ecosystem. The reality is that whatever project Facebook puts out is immediately seen as gospel and "naturally, a well-engineered thing and a good choice to use!" - this is literally a daily argument in #Node.js when somebody is trying to argue why a poorly-fitting tool is somehow appropriate for their usecase. "Well, Facebook uses it!"
In reality, the Yarn workflow is designed to work for Facebook and organizations of similar size, and in all likelihood it will break down in other situations. People will not realize this until they've already bought into it, and by then it's too late.
"Facebook also needs to take into account their influence over the ecosystem. The reality is that whatever project Facebook puts out is immediately seen as gospel and "naturally, a well-engineered thing and a good choice to use!" - this is literally a daily argument in #Node.js when somebody is trying to argue why a poorly-fitting tool is somehow appropriate for their usecase. "Well, Facebook uses it!""
Absolutely agree with this statement, it drives me insane listening to the hype before anyone has really even had chance to form an opinion on it.
> Why bother with semantic versioning if everybody is expected to manually
> check and pin their dependencies anyway?
I think this is the disconnect here: the lockfile does protect me from updates, but ranges are still useful: they allow a tool to help me upgrade my dependencies. With the lockfile-based approach, you usually _don't_ pin to specific versions: with Cargo, we even defaulted to the ^ operator, so 'foo = "1.2.3"' is the same as 'foo = "^1.2.3"'. So this is what happens:
1. foo v1.2.3 comes out and I put that in my Cargo.toml.
2. foo v1.3.0 comes out. My build doesn't change.
3. I decide "hey, there's some new versions of my dependencies, let's upgrade."
4. I type "cargo update". Now, my lockfile is updated to have 1.3.0 instead.
That is, ranges allow you to use semver as a tool. The big issue here is transitive dependencies; yes, for my own deps, I could have specified an exact version and done it every time. But what about the dependencies of my dependencies? Or the dependencies of the dependencies of my dependencies?
I totally understand your concerns about cargo-culting tooling, and that Facebook-sized organizations have different needs than non-Facebook sized ones. But repeatable builds are a big deal, and managing dependency ranges so you can still use semver while getting repeatable builds is a big deal. Even for smaller organizations.
(And that's beyond the speed and security aspects of yarn.)
> That is, ranges allow you to use semver as a tool.
I understand that, but you're looking at it from a technical perspective, while it's partially a psychological issue - namely, the question of how you drive certain desirable behaviour.
When you make upgrades explicit, rather than automatically installing newer versions, you reduce the deployment of (security) bugfixes, because many people simply will not run `upgrade`.
The reason is that you've created an expectation of manually checking versions before updating them - after all, why would it be a manual action otherwise? - and people do not want to commit to doing that. That is why this will harm the ecosystem - it undermines the mutual trust that a maintainer will be using semver, and that you can rely on it to automatically get updates.
The solution is simple: make lockfiles opt-in. That doesn't mean that you can't aggressively require a developer to lock their versions once they have opted in - it just shouldn't be a feature that is enabled by default.
> But repeatable builds are a big deal, and managing dependency ranges so you can still use semver while getting repeatable builds is a big deal. Even for smaller organizations.
For many people, it really isn't. Comparing the tradeoffs of manual upgrade management vs. the chance of occasional breakage due to non-determinism, many people will pick the latter.
And I'm not just talking about "organizations" here either - there's plenty of development going on outside of any organization, on personal projects, and that is precisely the kind of development you'd want to encourage.
Oh that sucks..How about "yarnish"? It's available on npm, sticks to your roots, and you can play on varnish if you like. Or you can always go the thesaurus route and use fleece, wool, cotton fiber, flaxen thread...
You don't. If you want to be able to do stuff like that (and inevitably run into "hmm, works on my machine"-type bugs once you start sharing your code with others), you need to keep using npm.
This kind of liberty is the sort of stuff npm was designed for. It is also why npm has been causing trouble in many cases.
I was under the impression that this was about speed.
There are many use cases for this functionality, the design of a software package shouldn't instill dogma into its users. Users use software for their needs, not to sign up to a religion.
No, speed is bonus. Yarn is about reproducible dev/test/build setups. Same yarn.lock - same node_modules content. That's the primary goal.
If you think consistency across computers is a religion then I don't think we're going to find a lot of common ground here though.
Finally, yarn is not intended to replace the npm client. It is intended to replace it for a certain use case, which happens to be one many companies have. Unlike Yarn, Npm is highly flexible and obviously you like it like that. I strongly doubt it's going to stop being developed all of a sudden because some other people made Yarn. So what, exactly, is the problem?
It still doesn't solve the biggest problem the node.js and javascript ecosystem has IMHO: dependency hell, thousands of trivial packages, just utter chaos.
I'm so excited about this. Having wasted lots of time migrating Shopify to use npm for JS package management, it seems that yarn fixes everything that had been a nuisance with it. Woo!
Rather impressed with this with the few tests I've run. Kudos to the creators and can't wait to see this project progress (and to contribute into!) in the near future.
This is a somewhat unrelated question.
I'm a systems programmer (I spend most of my days in the land of C and Go). I do understand the basic concepts of web apps and how you develop one. I have even built a few websites using Rails.
Can somebody please explain to me:
1) How does npm fit in in the process of developing a web app. How does it manage the js files / libraries and where does it place them?
2) How is yarn going to make your life easier
1. npm == bundler. One difference: it puts the source of your dependencies into a node_modules folder in the project itself, rather than in some global spot
2. npm doesn't have bundler's lockfile, but this does. It's also faster.
Yarn shouldn't be installed from the package repository.The inconsistency between the version of nodejs which yarn is running on and the version of nodejs which npm is running on brings significant effects.
https://github.com/yarnpkg/rfcs/issues/9
Any idea how to reach out to one of the authors?
Very interesting! How does the lockfile work with optional dependencies? We've run into this issue[0] on npm with a bad interaction between optional dependencies and the shrinkwrap. We develop on OS X and our CI server is linux, and fsevents works on one but not the other.
A big +1 for yarn from me. I had no end of trouble with npm needing internet connectivity at all times and its non-determinism. Thanks to all involved!
It also seems to imply that this is edited by the `yarn` command so it would be invalidated if you manually edited your `package.json`? Dunno if it's destructively invalid though.
I wish some kind of "auto-bundle" feature prevented the creation of multiple hundred MB's worth of tests, readmes and docs on disk when something like single "babel.bundle.js" file would do.
Is it really important that one knows that a tool dependency uses left-pad ? What could be the drawbacks of such bundling?
Lock file contains hashes of packages, so it works like many "trust on first use" systems (e.g. SSH): if you installed leftpad@0.2.3, you can be sure that futher installations of this same package will be exactly the same — if malicious npm server gives you something else, this should be detected and rejected.
Looks like they use SHA-1 hashes, though. Please, let's get rid of SHA-1 already.
actualyl at this point, unless there is a legal request or a serious security issue, we don't allow any unpublishes. we encourage transferring the package to the npm user and deprecating it. yay immutable registry!
If by "social" you mean bad decisions on settling a naming dispute, yes it was a social issue. That issue hasn't really been addressed publicly (the official policy still doesn't specify how disputes are decided by npm when no amicable agreement can be reached -- other than what essentially boils down to "we'll do what we feel is right").
The actual disruption however was solved by npm Inc disabling unpublishing (except for some very specific circumstances) and by creating a dummy account unpublished modules are reassigned to to avoid abuse (whereas previously anyone could simply claim unpublished modules and publish new versions).
I think it's more secure because you can deliver to it ("somehow") a global cache of packages.
For e.g. Facebook, I bet they have some online filesystem like NFS or similar, that hosts the global yarn cache for their continuous integration system. This way, their CI system doesn't call out to the cloud for every build.
This is something I really want to explore more through less-compatible modes. It was the original way yarn worked, but it wasn't compatible enough to be the default mode: https://github.com/yarnpkg/yarn/issues/57
If you do, would love see experiments on that. I do wish NPM would start adding badges or some other meta data to packages that signify that a module has constraints like:
- No native code
- No module state (No multiple singletons, and easily load variants of a module)
- Pure installation, aka no build steps and is fully cacheable via hashing/immutable module patterns
- Is all pure functional
- 100% code coverage
- Etc.
Yeah, that would be much better. But unfortunately the node_modules structure (which yarn is attempting to be fully compatible with) makes that impossible.
The reason is that each package only finds its dependencies relative to its own location. So your second level dependencies cannot vary from project to project unless you do copying.
(Example: AppA and AppB depend on LibX. LibX depends on LibY. Through their deterministic lockfiles, AppA and AppB disagree on which version of LibY to use. There is no way to symlink things together to satisfy that case without copying LibX or altering node's package resolution algorithm.)
It appears that yarn.lock resolves exactly one hash version of a package for one semvar version of it. What happens when your project depends on packages A and B, which both depend on the same semvar version, but due to manipulation by the author, rely on different hash versions?
JavaScript has enough issues that I prefer to not touch it, focusing more on backend Python work. Yarn is one of the first steps I've seen that makes me interested in JavaScript again. Excited to see where this leads the community.
The recommended installation method is actually to use the package/installer for your environment. Currently there's a Windows installer, Debian package, and RPM package, with Homebrew coming soon. See https://yarnpkg.com/en/docs/install
Just a heads up: someone tried to comment out the Windows choco instructions with HTML comments but the angle brackets were escaped so the instructions are still there (with the escaped brackets).
yarn
yarn install v0.15.1
info No lockfile found.
[1/4] Resolving packages...
error https://registry.yarnpkg.com/babel-code-frame: socket hang up
at createHangUpError (_http_client.js:252:15)
at TLSSocket.socketCloseListener (_http_client.js:284:23)
at emitOne (events.js:101:20)
at TLSSocket.emit (events.js:188:7)
at TCP._handle.close [as _onclose] (net.js:493:12)
info Visit http://yarnpkg.com/en/docs/cli/install for documentation about this command.
I think it's naive to not review dependency updates.
Also if a minor or patch update touches tens of thousands of files, you are dealing with a monster. And monsters need to be in cages.
It's incredible that a tool that we've tolerated for years, npm, is instantly and unceremoniously dumped for a superior offering, yarn. Technology changes on a dime.
I get NPM for backend development but I don't understand why frontend developers require NodeJS just to utilize it's package management. It really does not make any sense to me considering that NodeJS is server side javascript. If my backend is Python or Go or .NET or Java or whatever, now I'll need NodeJS for the frontend and only for NPM (and now this). It's a reason why I don't enjoy dealing with "modern" frontend technologies.
It doesn't make sense to require a backend tool for frontend technologies. That's just me though. So that's why I'm asking. Couldn't they just create a package system that's not based on some backend tech?
If you're writing a tool that's written in Javascript, you need an implementation of Javascript to make it work. Node, while focused on being "Javascript on the server", is also a really convenient way to get going with a Javascript runtime.
So yeah, in theory, you could create a whole 'nother binding to a runtime just for front-end tooling. Nobody's found node lacking enough to do that.
I guess I don't understand. How would you fetch/build/etc frontend JS files without some sort of backend technology?
(Also don't forget that frontend JS has to target dozens of subtly different browsers, so we unfortunately need to "build" JS files to account for backwards compatibility, etc)
We have that and it's called `bower`. We moved away from it because having two package managers to manage javascript (one for front-end and one for back-end) is more insane than the issue you described.
developers use node to compile whatever version of javascript they are using with a version that will run in most browsers without issues. It's also useful when concatenating and minifying assets. It's called an asset pipeline where you build a release, just like when you compile and package your Java,.NET or Go code before deploying it. Now of course javascript is a dynamic language and should not need all that stuff ... in theory ... but for some reasons developers love complexity. You can design the most simple system the most simple language, developers will find way to add layers of layers of complexity one way or another ... before going back to square one. It's interesting you are talking about Go. right now Go is quite simple, You bet your ass it will look like Java in 10 years just like Javascript is becoming more and more complex with tons of tools, transpilers and what not.
yarn's immediate acceptance by the community over npm appears to be a wholesale rejection of semver and an embrace of dependency determinism. Every dependency is locked down to the patch level with yarn. Instead of major.minor.patch, it might just as well be a single incrementing number now.
Sure, not saying there aren't any benefits but as a casual user they don't feel big enough to justify a switch. I may be wrong though (hence why I'm asking).
It took less than 20 minutes for me to change over a decently large project. The builds on the CI server increased by about 25 seconds by switching from npm install to yarn install. It was definitely worth it for me.
(Edit: Note: I mean you don't _have_ to. You might like the performance increase, but if you only use it casually, even that might not be worth the trouble.)
>We decided to zip the entire node_modules folder and upload it to an internal CDN so that both engineers and our continuous integration systems could download and extract the files consistently.
The 'attempts at scaling npm client' section is incredibly cringy even to read. Those are real complex problems that every one who works on a massive scale will face at some point in time.
> Finally, updating a single dependency with npm also updates many unrelated ones based on semantic versioning rules. This makes every change much larger than anticipated, and having to do things like committing node_modules or uploading it to a CDN made the process less than ideal for engineers.
You shouldn't check in node_modules. You should use a shrinkwrap file to indicating the specific sub-dependency versions you're using.
> They address pretty throroughly why that does not work for them in the linked article.
Not really. They just mention it in passing:
From the article
>> Shrinkwrap files aren't generated by default and will fall out of sync if engineers forget to generate them, so we wrote a tool to verify that the contents of the shrinkwrap file matches what's in node_modules. These files are huge JSON blobs with unsorted keys, though, so changes to them would generate massive, difficult-to-review commits. To mitigate this, we needed to add an additional script to sort all the entries.
Comparing node_modules to shrinkwrap isn't necessary. Building the app will re-generate node_modules and tests should catch whether or not the app works as intended.
The sorting issue for shrinkwrap is legit, but the answer to that is sort it! Sure you'll get some deep diffs if the dependencies change, but that reflects the real changes to the app.
The simpler answer seems to be to add a build test to mandate the existence of a shrinkwrap file, verify it's sorted, and (optionally) verify it's up to date. The latter doesn't even have to be done every build.
I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once. npm solved this problem elegantly, but still struggles with non-determinism. I had resigned myself to thinking that maybe these were just fundamental tradeoffs in package management that could never be resolved.
Then I had the opportunity to use Cargo, the package manager for Rust. It synthesized what, in my mind, are the best features of npm and Bundler into a single package manager, with terrific speed to boot.
Yarn looks like it will be Cargo for JavaScript. After Cargo improved on many of the great ideas of npm, those ideas are returning to help improve the JavaScript ecosystem. Cross-pollination at its best.
This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.
I'm also happy to see that a significant amount of work has already gone into the governance model. Despite the largest contributors being Facebook employees, it looks like they've really outdone themselves making sure this is a community-run project. A BSD license, no PATENTS file, and an Ember/Rust RFC process[1]; this has all the hallmarks of a community open source project. That's critical for open infrastructure projects like this, and they've nailed it.
[1]: https://github.com/yarnpkg/rfcs
I'm very much looking forward to using Yarn in my own projects because it looks like it solves a lot of real problems I encounter every day. Thanks for all the hard work!