I recently started working seriously with node.js (I've tinkered over the years since it was launched, and we provide some support for it in our products, but never actually built anything with it). I went looking for a library to deal with logins, authentication, password resets, etc. Normal stuff that most web frameworks have some solutions for.
I found a package on npm that sounded like it did everything I wanted (plus a few extra things, but I figured I could ignore those). It took longer than I expected to install...so, I did a little digging. It had installed over 53,000 files, and the resulting directory was 110 MB in size!
I was absolutely flabbergasted. I couldn't believe installing one package, for something seemingly simple, could balloon up that large. I won't name names, as I did a little more poking around, and realized that most npm installations pull in thousands of files via automatic dependency resolution, though this one was a particularly egregious example. I've gotten to where I only install stuff via npm when I'm on a free connection; I normally work on mobile broadband, which is very expensive (and adds up to almost $300/month even before I started playing with npm).
Now, to be fair, it was pulling in a web framework...maybe Express or Hapi, I don't remember which, and all of its dependencies, so it was actually a lot more than just the login module. The kind of annoying bit was I already had a global installation of both of those frameworks from following tutorials, but it still seemed to insist on pulling in its own preferred versions of stuff, and putting them into the project directory.
I come from the Perl world, where if you don't spend at least half your time looking for and evaluating libraries before you start writing code, you're not being very productive. I'm, frankly, overwhelmed by how big and unfiltered the npm ecosystem is. I've found myself relieved to start tinkering with more "all in one" libraries and frameworks, because I don't have the time or knowledge to evaluate libs on my own. I ordinarily prefer a more a la carte approach, where you just pull in what you need, and so big libraries and frameworks don't fit that. But, I can't make sense out of the ecosystem without some guidance. There are over 70,000 npm packages! Curation really has turned out to be one of the big problems in computer science.
Really recommend it. As you can see, react-router's deps are actually extremely conservative.
Webpack, another awesome project, gives a good example of a bigg dep tree: http://npm.anvaka.com/#/view/2d/webpack
It caches the package listings and the packages that you download. It will act as a pass through that with a limited TTL on the cache, but there is an option to fallback to the cache if you can't connect to upstream.
Granted, you have to have already installed something for it to work as an offline cache.
Also, part of the problem with all those files is that npm allows packages to installed pinned dependency versions. If package-a requires lodash 2.x and package-b requires lodash 3.x, then both will be installed within the respective package's directory. For example let's dive into the node_modules/ in one of my projects.
This seems like maybe a really big area for evenagelism in the node community. I've watched a ton of talks and tutorials lately, and several of them made a strong point of saying, "use versioned dependencies" because libraries aren't practicing good semantic versioning, so even minor version changes can be breaking. So...maybe there needs to be a lot more attention being paid to semantic versioning being used religiously.
I'm new to this ecosystem, so I'm definitely not an expert, but it's certainly been an intimidating point for me; maybe the most difficult thing to wrap my head around. I'm used to being able to spelunk into my project, and read everything I'm depending on, or at least skim it and kinda grok where things happen. How would one even do that with 53,000 files? How can anyone trust any application they build with these tools? I mean, the security implications alone are breathtaking, to me.
You're being unfair. You're obviously using npm 2.x, we've since moved to 3 for a very long time now where the dependency tree is flattened and this issue is avoided.
The problem isn't pinning, the problem is that it's routinely accepted for dependencies to disagree about which version of a library is ready for production use, because npm doesn't treat that as a disaster that should block deployment until the community coordinates their acceptance testing.
Basically, if lib1 and lib2 each use lib3, I don't want to upgrade anything until both lib1 and lib2 agree that a newer version of lib3 works.
If a new version of lib2 moves to a half-baked version of lib3, I want to stay on the old version of lib2 (and lib3) until the community agrees that the lib3 change is good and everyone has migrated to it. Having two different versions of lib3 in process is begging for bewildering bugs, and it was a serious mistake for npm to let that happen (much less by default!)
While I'm not defending running multiple versions of a library, I've never seen it cause bewildering bugs, or any bugs at all. The deeply nested style of npm may have issues, but it makes it really easy to avoid thinking about dependency problems.
I found a package on npm that sounded like it did everything I wanted (plus a few extra things, but I figured I could ignore those). It took longer than I expected to install...so, I did a little digging. It had installed over 53,000 files, and the resulting directory was 110 MB in size!
I was absolutely flabbergasted. I couldn't believe installing one package, for something seemingly simple, could balloon up that large. I won't name names, as I did a little more poking around, and realized that most npm installations pull in thousands of files via automatic dependency resolution, though this one was a particularly egregious example. I've gotten to where I only install stuff via npm when I'm on a free connection; I normally work on mobile broadband, which is very expensive (and adds up to almost $300/month even before I started playing with npm).
Now, to be fair, it was pulling in a web framework...maybe Express or Hapi, I don't remember which, and all of its dependencies, so it was actually a lot more than just the login module. The kind of annoying bit was I already had a global installation of both of those frameworks from following tutorials, but it still seemed to insist on pulling in its own preferred versions of stuff, and putting them into the project directory.
I come from the Perl world, where if you don't spend at least half your time looking for and evaluating libraries before you start writing code, you're not being very productive. I'm, frankly, overwhelmed by how big and unfiltered the npm ecosystem is. I've found myself relieved to start tinkering with more "all in one" libraries and frameworks, because I don't have the time or knowledge to evaluate libs on my own. I ordinarily prefer a more a la carte approach, where you just pull in what you need, and so big libraries and frameworks don't fit that. But, I can't make sense out of the ecosystem without some guidance. There are over 70,000 npm packages! Curation really has turned out to be one of the big problems in computer science.