I usually don't respond to anything which I feel is just another "language war" provocation, but whenever I see these type of reviews I'm mystified.
I've developed in many languages and frameworks, both well known and lesser known - decades of client and server side of C/C++, Javascript, Lua, Java, Python, PHP, Perl, Lisp. Pascal (just to name a few) in projects of all sizes, and not once did I have the thought "this language sucks". I think there are a few reasons for this:
1) Even after all these years, I'm still passionate and excited at the ability to sculpt logic, regardless of the "material" I need to use.
2) A "keep it simple" approach - no way to overemphasise this. Know the advantages and limitations of the language and stick to what works. Keeping things simple should be like a fractal - existing at all levels of abstraction.
3) I'm very wary of hype. New, shiny and trendy does not necessarily mean better, especially when the hype is in conflict with keeping things simple.
I find that when you understand the playing field, mark the areas to avoid and keep things as simple as possible, the elegance of the design and implementation usually makes the advantage of language X over Y insignificant, and I feel that blaming failure on the language used is like blaming a bad novel on the word processor used to write it.
The article doesn't appear to be blaming Javascript as a language, but rather the Node.js ecosystem as a server platform.
Not all tools are appropriate for all niches. The author made it clear they don't have an issue with Javascript as a whole or even with Node.js necessarily, but rather the use of Node.js as a major server platform or as a platform for large applications.
These things aren't pain points in the browser the way they are in Node. I have never felt the need for an ORM in the browser. I have never dealt with client-side code that was using so many libraries I had to worry about whether exceptions would be handled via exceptions, or the first argument of the callback, or rejected promises. No one (at least, no one I know) is installing node modules like isArray to use in the browser.
Yes, these things _could_ technically apply to the browser, but it's not commonplace. In the node world, these are all things you deal with consistently.
> No one (at least, no one I know) is installing node modules like isArray to use in the browser.
Erm, well, considering the adoption rate of Browserify and Webpack, I'm going to disagree with you there. Especially considering React's momentum practically requiring some form of module builder.
Startup speed is terrible with that approach, each browserified module adds like 10x more code as boilerplate than something like isArray implementation would take. Then every module dependency is resolved dynamically at runtime, which will also quickly become a performance problem even when you aren't using micromodules.
I haven't touched browserify, but with webpack what you said could not be more false...
Webpack doesn't bundle at runtime, it doesn't add any amount of code overhead per module that I can easily measure, and it doesn't trash startup speed.
Same goes for browserify. You can have lots of tiny modules and (watchify especially) is fast to re-build. And yeah perf of 'require' statements at run-time is a non-issue.
But there is a good argument that your server components (or at least a good portion of them) should be built in a language that requires some more up front work because it makes maintenance easier down the line.
You are a lucky person if you have used so many languages and never missed a thing in a language you are using at the moment that is readily available in some other language you have used in the past and that would make everything so much simpler!
For example the OP got fed up with Python and switched to Node. Now he is fed up with Node and switching back. I can say that focusing on the downsides of the tool I am currently using and the upsides of the tool I am not using is a fairly common thought pattern for me too. Of course the way to combat it is to remember that it is very rare for some tool to be strictly superior to others and just focus on the task at hand.
Choice is a very dangerous thing. When reading the book "Coders at work" I was struck by how primitive some of the old computing environments were. But that did not stop those people from doing amazing things with them. I now think that was partially to their advantage because they were freed from choosing the language/framework and so could concentrate on doing, you know, actual stuff.
Keeping it simple works wonders but you can't do it alone. A language with a culture of making things complex so as to achieve some standard of perfection, and/or with many developers simply not knowing how to stay out of trouble and not being able to tell what's fragile and what's trustworthy, will make it harder to "keep it simple."
I agree that when programming mostly alone one can get by very nicely in any reasonably popular language since its popularity necessarily means that it's workable in the areas where it's typically used. I disagree that languages can not get in your way if you know what you're doing once you're not doing it alone.
I didn't get the impression he hated the language or even the node.js ecosystem. Love or hate for languages have no real place in production. I work at a place with 5000 employees and I sometimes work with technologies I'm not fond of because they are the right choice for the job.
Things like long term support, stability and security rate in how mature different technologies are for production. Considering a lot of the problems the author ran into with node.js aren't secret, I think he might actually have liked it quite a bit to go on to try it in production for a full year.
> I feel that blaming failure on the language used is like blaming a bad novel on the word processor used to write it.
Sure. If you're a good novelist you can write something great even on a keyboard with a broken 'e' key (btw, your 'I' key is about to break; you should replace it asap).
But novelists don't have deadlines and when the novel is done, they usually don't take reader requests to change this or that part of the story.
"The advantages and limitations", what works and what doesn't, "areas to avoid" are precisely the point of that kind of review.
> (btw, your 'I' key is about to break; you should replace it asap).
haha - touché, point taken :-)
Deadlines seem like a great excuse for compromising quality. Sure, life is complicated, the boss is demanding, the mortgage has to be paid, the children need to be supported etc. etc. but compromising quality and enthusiasm (they usually are correlated) because of the "terror" of a deadline will just leave you at the mercy of the next "terror", only this time you'll even have even less enthusiasm to fix the spaghetti. Doesn't sound like an enjoyable existence.
Of course, very few people have the privilege of never having to compromise, but it's never black or white and there are many more degrees of freedom to choose the path with more quality than are implied.
>"The advantages and limitations", what works and what doesn't, "areas to avoid" are precisely the point of that kind of review.
The "area to avoid" in the review is Node.js and considering that large and impressive projects have been written in it, it seems that this is another case of throwing the baby out with the bathwater.
> But novelists don't have deadlines and when the novel is done
I hate to "well actually" you :) , but a great number of authors have contracts that demand one or two novels a year over a fixed period of time. I know Iain Banks did with his publisher, he mentioned it at a one of his readings of a new but yet unpublished novel I was lucky to attend back in 2007'ish.
> they usually don't take reader requests to change this or that part of the story.
Even well known authors are reigned in by their editors and proof readers, and sometimes things have to be changed.
It's more like hundreds of thousands of writing teams from different cultures and levels of experience sprung up to write collaborative epics in Latin because it had suddenly become cool to do so while bringing the language to the modern world by inventing a new vocabulary and arguing on the correct pronunciation of everything.
Now all of a sudden, having types and some standards to gather around doesn't sound like a bad idea anymore ;)
I agree with one of the commenters: Lessons already learned by older engineers (who went through similar woes with other languages/tools) are being re-learned again and again.
The software industry is in a sorry state.
Unless you are a very disciplined team with a very strong sense of writing modular code, don't use Node.js for any larger project.
And even then, the single-most useful function in an IDE 'Show Call Hierarchy' will never be available when using a dynamically typed language.
That is not an issue for smaller projects. However, long before you even get close to the the million lines of code project size, your tools will fail you.
Your debugging/refactoring times will explode and adding a new feature will seem unsurmountable.
Instead, let's just re-write everything from scratch because the cool hipster that wrote your backend a year ago has left for greener pastures...
I won't even try to guess the amount of technical debt produced with Node.js and the likes each day in the bay area.
And, yes, I just used Node.js to write a Slack-bot. It was fun, took me two hours and got me up and running quickly.
That's the beauty of it. Just be aware of the dangers.
However, having witnessed (different shapes of) the wheel being reinvented over the past 2 decades, my take is slightly different: each language/environment has its strengths. If you have the luxury of having a senior team that's decently versed in several environments, then you can have your cake and eat it too: for each tool, micro-service, component... use the right language for that. Then tie it all together on the network. Since the system as a whole is likely to be a network product anyways, embrace it to your advantage.
Joe Armstrong wrote a good post[1] about connecting programs a few months ago, that's quite relevant.
Node.js is a great environment for many microservices. Got a task to do which is 80% done in some npm module? Make a node.js service around it. Same idea for Perl & CPAN, C libs, Java tools, etc.
I've worked in three million+ loc codebases, in PHP, Python and Java. I don't share your opinion that you need static types in these circumstances. You need discipline, modularity, and most importantly you need to have been blessed with gardeners and maintainers throughout the life of a project and not just after a mess has already taken hold.
Did you read what I wrote?
I already said that you need a disciplined team.
Good luck keeping that team together for years to come.
Not sure if you are disputing the fact that keeping code around is a challenge, or not.
> And even then, the single-most useful function in an IDE
> 'Show Call Hierarchy' will never be available when
> using a dynamically typed language.
True when "never" = "40 years ago". Smalltalk IDEs have had features for displaying call hierarchies statically and dynamically since essentially forever. And of course, in a dynamic environment the IDE vs. program distinction doesn't matter, so if you want to really know what's going on, just insert a self halt. and check in the debugger.
Smalltalk also introduced automatic refactoring, another canard of the "impossible to do in a dynamic language" variety. Really weird when "impossible in" = "invented there".
Yes, we can talk about how static type information can make some of this easier, but it also makes other parts harder (and the RB guys did talk about this, so the information is out there).
Please: if you want to diss JavaScript, diss JavaScript, don't say "dynamic languages". If you want to diss dynamic languages (please do!), inform yourself first.
Lessons already learned by older engineers (who went through similar woes with other languages/tools) are being re-learned again and again.
If only that were true! I think the JS (and more generally web development) ecosystem today is more a case of those who do not learn from history being doomed to repeat it. The thing is, if you actually are Google or Facebook or Microsoft or Mozilla or Apple, you can throw huge amounts of resources at problems and produce sufficiently useful solutions regardless, and then you can do the same again a few months later if you need to. These are the kinds of organisations that set the tone for the whole community because of their sheer size and influence. However, their goals are not necessarily aligned with the rest of the web development community, nor will their approach necessarily work well for someone else.
I think you're both saying something similar, but there's a "why" in there to connect:
> Younger engineers are ignoring lessons learned
Why ignoring them? Beyond the 'youth' factor directly, the signaling from the larger companies mentioned (google, etc) is that those lessons don't necessarily matter, and... look, google did XYZ, you can too (ignoring that they threw potentially 3-5x as many people at problem XYZ than you even have, much less justify).
Maybe neither of you meant that, but that was the connection I just saw between your two comments.
Yes, that's basically what I meant. I think some of the highly visible projects from tech giants today are succeeding despite their approach to software development, not necessarily because of it, but a lot of the new developers coming into the industry lack the experience to realise that and think of these projects as examples of how things should be done.
Every time I see some young developer posting on HN about how the code we write today only has to work for a year or maybe two at most, or how 10,000 lines of code is a large program, or how some high profile company's web site that is still fundamentally just forms and tables and an API for talking to a database is a complicated UI, I weep a little for the future of our industry and our lack of ambition. Of course things like performance and stability and standardisation and portability don't matter very much if all you ever do is hop from one small throwaway project to the next every few months.
I think the culture around HN is unfortunate in this respect, because from a commercial point of view, it is an alluring idea to build some sort of trivial MVP, try and get crazy amounts of funding, and then if (and only if) you succeed, to throw it all out and start over. As a business strategy for building the next WhatsApp or Instagram, that's rational and perhaps quite effective. But it also contributes to the everything-is-expendable mindset that plagues our industry, and since the overwhelming majority of software development projects don't have the luxury of starting over every five minutes, I think it breeds a lack of respect for professional programming and the skills required to build good software without the benefit of an effectively infinite funding source from benevolent investors (or from the goose that keeps laying golden eggs elsewhere in the organisation).
That might be OK for Facebook or Google, but I wonder how many other startups that fail could still have been very successful, albeit not in unicorn territory, if their developers had paid more attention to some of these issues. I wonder how much time humanity collectively wastes today just because software quality and reliability aren't taken seriously by too much of the industry.
maybe i haven't read this thread closely enough ... the parent to this post mentions "lessons of the past" and its parent mentions typed languages, so uh, what lessons are being discussed here? certainty not "the past teaches that typed languages are best," b/c lisp
For what it's worth, I don't think Lisp is a very convincing example of how useful a language with a limited type system can be, because to a first approximation no-one uses it.
But more generally, the modern web development world is only just waking up to the idea that if you want to build software sensibly beyond a very small scale, you need tools for modularity and composition. Separation of concerns is important to preserve developer sanity and keep maintenance manageable. It is helpful to have well-defined, stable interfaces, whatever the current underlying implementation. Having an expressive language is good, but having an expressive language built on sound foundations and with consistent patterns and clean semantics is much better. Standards for protocols and formats are important. Performance matters, in many different ways. Quality matters. Portability and standards compliance matter. Longevity and stability and backward compatibility matter. Having a few good tools that work well and offer substantial benefits is much more valuable than having a million tiny things that change or become unsupported within weeks. Frameworks can be useful for getting going quickly, but you lose flexibility down the line if your requirements evolve beyond what the framework was designed to support and that can hurt. Libraries offer more flexibility, but you often incur overheads converting to and from their conventions and those overheads need to be reasonable if the library is going to be useful. Reuse is good but not always the best choice.
These and many other lessons in real world software development were brought to you by the words "developers older than", the number 30, and experience of programming outside the JS bubble. :-)
late reply, but something occurred to me to say more as a (hopefully funny) witticism than an actual talking point:
if we approximate usage in terms of a poly sum w/ coeff derivatives wrt time (Taylor series is the word, I think) then one might estimate lisp to be a very much used language
I'm mostly lamenting the overall lack of progress in creating reliable, bug-free software.
You would never let a software engineer build a house, would you?
For some reason, the discipline of architecting and building houses is far more advanced than building software.
Why is that?
We keep inventing new programming languages to tackle various aspects of making software better (safe concurrency, preventing memory leaks, etc. etc.),
but we keep introducing 5 bugs per Function Point and the rate of finding and removing bugs has stayed pretty much the same the last 20 years.
There needs to be a fundamental change in how we create software.
A lot of the Node issues seems to be due to the fact that it's a very new language. Sure JS has been around for a long time but the problem domain was small and very different from what Node is trying to achieve. Couple that with some issues other languages don't have (JS's standard library is quite thin) and I'm not surprised at these issues. I felt bits of this playing with Clojure and Go when they were new.
I imagine a lot of these issues will settle down in the next few years as one or two sets of conventions become de facto standards.
The main problem with most typed languages is basically that types are nominal by default.
What I want is free-form records and structural types by default, nominal types only when necessary. I get more than half of that with TypeScript (it does lack nominal types but there are ways to get around that), and honestly its the half I prefer having.
Most serious statically-typed languages have some sort of "anything" type where you can check the actual value type at runtime using some sort of reflection. For example, the direct Golang equivalent of a Javascript object is a map[string]interface{}.
If a particular library doesn't have one, you can write it yourself. In that case theres no need to model the entire library - you can only model the subset of functions / methods that you use.
With TS do I have to have a spec for every third party code or can I import something and use it as-is, but not get the benefits of TS on that part of the code?
The Netflix.com site and webapp runs on Node (and talks to a number of services written in mostly JVM based languages). While we encounter challenges just as we would with any other language -- it works for us and I would argue that it's a pretty big application.
There's always a multitude of ways to get something done, and it's up to you to decide what tool will do it best. Don't treat any one language as an end-all-be-all and you might find yourself much happier and more productive. Of course, YMMV.
Speaking only for myself, a middle-weight .NET dev who was, not too long ago, working desperately to find my footing in all this:
1. I think the learning curve to doing good work with Node is really shallow at first, and then it quickly gets almost vertical.
2. I think that ALL of the other server-side language/framework combos (Ruby/Rails, Python/Django, C#/.NET, PHP/Symfony, etc.) make it significantly easier for a "merely-good" dev to make really good Web applications.
3. I think the kinds of devs that work at, say, Netflix are often the kinds of devs that can do amazing things with almost any dev stack, but that leads directly to my last point:
4. I think that most junior devs are making a tactical mistake with their growth, and with their careers, when they choose Node for their first forays into Web app development. The relative chaos and churn, and questionable suitability of Node for many-maybe-most use cases, make Node more trouble than it is worth for someone still struggling to make sense of the basics.
> 2. I think that ALL of the other server-side language/framework combos (Ruby/Rails, Python/Django, C#/.NET, PHP/Symfony, etc.) make it significantly easier for a "merely-good" dev to make really good Web applications.
All of those frameworks are threaded/synchronous - they are maybe 10 times easier to use and potentially 100x slower (less concurrent) than Node. The ruby equivalent isn't rails, its eventmachine. Moving from django to twisted is probably just as hard as django to Node. 9/10s of the python ecosystem assume synchronous code so you end up with threads, processes, and callbacks.
I haven't found the overhead of threaded frameworks to be anywhere close to that -- in practice it can be close to even, maybe 2x slower.
And a big advantage of threaded is that it is (obviously) easier to scale to multiple CPUs, and I don't have to worry about one connection taking more than a handful of milliseconds and jamming up my server.
Node is fine, but I feel many of the speed benefits come from V8 vs cpython, rather than threaded vs async
Surprisingly (to me at least) scaling single threaded code to multiple CPUs is actually very easy: just spawn the node process N times (using cluster, naught, pm2 or similar project) and that's it.
Elixir/Phoenix, Go/, Rust/ ... should all outperform node.js easily, plus they're actually parallel and not just concurrent AND are much more solid languages to begin with.
But aren't all Node.js deploys fronted by a reverse proxy (like nginx) to multiple node processes? That gives them actual parallelism in practise, right?
Or is it remotely common to just have a single node process and never use more than one core?
splitting the load through multiple node processes is fine for simple usecases and in general you're fine if you do mostly IO stuff.
But there are usecases that have to crunch numbers or do other CPU intensive algorithms that can be parallelized on the algorithm level. This is where node.js just falls apart. Additionally, if an algorithm requires quite much memory (multiple gigs), node.js performs very poorly.
Most companies that do actual CPU intensive work have services written in other languages to handle this, and "we're using node!" mostly means that they just serve results or do other IO-Tasks with node. But not the actual work.
> All of those frameworks are threaded/synchronous
"Threaded" and "synchronous" are orthogonal concepts that don't deserve being lumped together with a slash. They can have significantly different effects on performance depending on the scenario. C#/.NET isn't explicitly threaded (or single-threaded) or synchronous, and it's right in the middle of your list. It has more in common with Java than with the other three. Keep in mind that there are node packages for threads, too.
Node evangelists often don't seem to get that non-blocking behavior isn't a Node exclusive. You can do asynchronous non-blocking code with C#. You can do an event loop and play node all day long with any number of threads you want. It's not that opinionated about how you run your code. There is a lot of old code that is synchronous. All languages have significant amounts of old code hanging around that drags us down. But things like the TPL, async/await and IOCP can make C# code perform significantly faster than Node. And it's in the middle of a serious public overhaul.
Java and the JVM can do this as well. Java itself might be a bit behind on language features, but it tends to outperform the CLR (also uses more memory). And libraries like Netty and Vert.x are seriously impressive. You can do incredibly amazing things with the JVM, and it's been one of the leaders in performance among GC-ed environments for a long time. You've also got the option to use languages like Scala and Clojure on the JVM.
I'll give you that Node applications might outperform Ruby and Python apps when running on the default environments. But those also can run on separate runtimes which can significantly boost performance. And we're ignoring less common server languages like Erlang which are quite suitable for certain situations.
You should also note that Javascript (and node) performance will be a bit questionable for a while, anyway, as the recent ES6 support has introduced a lot of performance wrinkles (https://kpdecker.github.io/six-speed/) that will take a while to iron out. ES6 adds some really attractive features, so expect node performance to drop (albeit perhaps only temporarily) as your package dependencies start adopting ES6-isms.
Just as there are hundreds of people jumping on the Node train, there are hundreds jumping off. Javascript tends to prototype rather quickly, which has huge drawing power. The constantly shifting ecosystem and the weak tooling, among other things, can affect its staying power. Some people are leaving due to code maintainability, others are leaving because of performance concerns, others are leaving because of ecosystem things like left-pad. There's a lot of variables involved in choosing a language, Performance is only one of them and no languages is a silver bullet.
We can pick on other languages all day, but there's always something to make fun of about our own language/ecosystem. So let's let people pick the tools they want and maybe critically evangelize a lot less. If they pick something ill-suited to their requirements, they'll just have to switch later. It's a pain, but it's not the end of the world.
They are probably 10x faster. You'd be surprised that Node is actually pretty slow by itself and only selling point is async which for the most part is not even used e.g. callbacks inside callbacks.
And if you leverage parallelism of those languages they become 100x faster since they can do multi-threaded processing unlike Node.
If they would switch to more efficient languages, they could probably run on half of their current servers.
But there's the huge cost of rewriting something you already have.
Will that affect your total cost significantly, if you can run on less servers.
I.e. it's too late to switch now.
There needs to be a significant business reason to switch.
"Stuff-that-kinda-works but needs more hardware thrown at it" is probably not a good reason.
I think the 'big application' bit is what challenges many people. From what I understand, Netflix is essential a huge group of micro-services, and I 'think' this is where Node.js shines.
What are your thoughts on that? When you're developing a service at Netflix, does it feel like you're building a single large application? Does the mentality of micro-services change how you approach code and therefore the validity of the original posts arguments?
One of the core tenets of the Netflix culture is "Freedom & Responsibility". In this context it means that each team is free to make calls like that as they see fit. Promises, Observables, Generators, Callbacks -- it's up to your team to determine what makes the most sense.
That said, we do make heavy use of Rx on the API layer (RxJava/RxGroovy). My team isn't currently using RxJs, but I expect that to change in the future. There are a lot of Observable fans here :)
I watched one of the recent Netflix engineering videos where they moved to a customized version of React and I could have sworn in that talk they actually moved away from Observables[1]. What was confusing is that the other talked released at the same time they talk about enhancing RxJS.
> By the way, what is Netflix's take on Promises vs RxJS?
We've been doing tons of research into the Netflix methodology and teams are highly compartmentalized (albeit communicate richly). This means that when you ask "What do you do at Netflix?" you should really be asking "What does team X do at Netflix?" or "Does any team at Netflix do X?"
For instance, unless things have changed or I have the story wrong, JVM is used for the streaming because that's the tool which that team chose. The other teams merely communicate with it over an API (regardless of client language).
How big is the Netflix webapp, though? There's what, 5 or 6 main content "views", then around 20 user-editable forms? Plus the help site. Am I viewing this incorrectly? It doesn't just doesn't feel like Netflix is a large app from the UI perspective - maybe that's just good design.
Hope this doesn't come off as "I could do this in a weekend", just questioning how big the actual webapp is.
I believe you would be surprised were you to look under the hood.
There are a number of different "UI areas" in Netflix.com -- homepage, titles, signup (multiple flavors of this, depending on how you're signing up), onboarding, member browsing/viewing experiences, account management, numerous partner integrations, etc... not to mention all the work that goes into per-user customization, AB testing, and internationalization.
No offense taken on the question. IMO, if you look at the site and view it as simple then we've done our job correctly. :)
If I am not wrong, Netflix folks make heavy use of 'stream based programming' using RxJs. This greatly simplifies a lot of the standard problems faced in JS - like callback hell and error handling, if one adopts it consistently across the stack. Not many folks have discovered this as yet.
These types of articles make me laugh. Typically a dev with many many years experience with one language, learned all it's quirks, standards, etc decides to try Node.js because it's the "new hot fun toy", and expect it to work like their old language, and realize that is not how it works, doesn't know where to find what and fails real hard to realize that JavaScript in general is in a huge influx of updating at this moment, which by nature propagates to Node.js.
The end result is they get frustrated and go back to their old language.
Yeah, so some js devs might need to stop overselling javascript to everyone, pretending it is the one language that people are waiting for years...Just saying.
You should try to surround yourself with developers who don't have such attitudes. I am a fan of JavaScript, I love Node.js and I will often times suggest it to newbies. But I don't pretend it's the be all/end all of languages. Just like any other language it has its strengths and weaknesses. It's up to you to decide if it's the right choice for you.
You should try to surround yourself with developers who don't have such attitudes.
The trouble is that while you can do this for yourself to some extent, you can't necessarily do it for the people you have to work with. I've worked on a project where one guy came in to do a simple database back-end for an embedded system and decided Node and its ecosystem were appropriate. He seemed to have some experience with those tools, so the rest of the team went along with his decision. A year later, there was still little useful code actually running on the required platforms. One of the other guys got fed up and wrote the basic case using C and SQLite in less than a day. That guy did have some relevant experience and had been questioning the use of Node for that project from early on, but because the Node hype was so high-profile, management hadn't known which developer to believe. This is why it's useful to have articles like the one we're discussing, where people who have actually tried a tool in real life for a significant period of time share their experiences, even if those experiences weren't quite what they'd hoped for at the start.
The difficulty of getting Node itself up and running on the various platforms involved was a significant part of the problem. These weren't Linux boxes built around Xeons. It turns out that as soon as you go outside of the mainstream, the tools and documentation for building and running Node are awful compared to a lot of other languages.
The lack of standardisation and stability and the relatively poor dependency handling within the Node ecosystem were also major factors. Way too much time was spent just trying to make sure different machines really did have identical packages installed, figuring out how to pin everything down to make reproducible builds, and so on. Again, if you're happy just to shrinkwrap, npm install on your local machine, and wait if the repository is unavailable, that's one thing. However, as soon as you start talking about preserving exact configurations within 100% local build environments or building images to install on different machines, all kinds of problems crop up that have been well addressed by now within a lot of more mature language ecosystems but not so far with Node.
Another problem was that Node and its ecosystem are just so heavy overall, and this one is nothing to do with the variety of platforms in use. It's just a big runtime, and a lot of overhead because of the way libraries and dependencies are handled. This would be an issue for anyone operating with limited resources, whether it's trying to run something on a Raspberry Pi or trying to keep costs down by using the smallest possible instances of cloud-hosted virtual servers.
There did also seem to be an element of wanting to use the shiny new toys, and of stubbornly sticking with those toys for far longer than should probably have been allowed, so in that sense there were issues with the developer as well.
But there was also a willingness among the wider team to trust that developer, at least to begin with, because with Node's high profile and large ecosystem, no-one expected it to be as immature as it turned out to be as soon as you left the mainstream. For comparison, porting code written in traditional systems programming languages like C and C++ was done routinely, but even among the dynamic/scripting family, the more established server-side languages have caused far fewer problems than Node.
> The tools and documentation for building and running Node are awful compared to a lot of other languages.
Can you be more specific?
> It's just a big runtime, and a lot of overhead because of the way libraries and dependencies are handled.
Compared to what exactly? I don't see any mainstream dynamic language except Lua doing better. Ruby? Python?
If you don't need native modules (which is actually the case in a surprising number of situations) there is no build process, except making a tarball of node_modules and the project. If you do need native modules, the situation is slightly worse than C or C++, but not significantly: you generally need to build on a machine with the same architecture (Setting up cross-compilation with C isn't very easy either)
> A lot of overhead because of the way libraries and dependencies are handled.
This has some truth to it, but in my experience its overblown. Being just a little bit careful with your dependencies goes a long way.
On the other hand, if you want to run node on something like this: https://www.olimex.com/Products/OLinuXino/iMX233/iMX233-OLin... that will definitely be a problem. Thats about the lowest specced hardware that can (barely) run it, but 64MB RAM just isn't enough, really.
Sorry, it wasn't my part of the project so I can't offer much detail. I know they never got as far as a Node build that passed the full test suite on some platforms. Many of the problems seemed to be around how floating point is handled on different types of CPU.
Compared to what exactly? I don't see any mainstream dynamic language except Lua doing better. Ruby? Python?
The total footprint for Node and supporting packages was taking up a few GB more than the equivalents for some of those other languages you mentioned. If you're working on a system where your non-volatile storage is something like a microSD card or flash chip, "a few GB" can be a huge difference.
If you do need native modules, the situation is slightly worse than C or C++, but not significantly: you generally need to build on a machine with the same architecture (Setting up cross-compilation with C isn't very easy either)
Sorry, our experience has been very different on both counts. Building on a machine with the same architecture isn't necessarily a reasonable option if you're targeting platforms that aren't general purpose computers, and at best probably means jumping through a lot of hoops and figuring out a lot of the details for yourself. In contrast, cross-compiling a language like C is straightforward and reliable, the tools for doing it are mature and well documented, and many developers who have worked in embedded systems are familiar with the techniques.
No. The person doing the Node version was trying to use some of the NoSQL tools, and seemed to be spending much of their time documenting and implementing things you get as standard with a SQL database. The person who wrote the C based alternative just used SQLite as a demonstration of how much easier and more robust everything could be using a different database instead.
One of the striking ironies of the project was that although in principle a dynamic language like Node doesn't need a compilation step, just the JS source, while C code has to be compiled for every target platform, the reality was that getting the C code running was dramatically easier. A well-made C library like SQLite could be installed, cross-compiled and running on all the required platforms within literally a few minutes, while Node and several major packages were so flaky that the equivalent level of real world functionality was never achieved even after several months of work. Again just for comparison, some of those other dynamic languages you mentioned were in between but much closer to the shorter end of the spectrum. The Node ecosystem, in our experience, proved to be an exceptionally bad choice for this sort of work.
I'm afraid that based on what you said, it only made it clearer that this wasn't an issue with the ecosystem but with that particular developer. I don't really see how you can extrapolate that this was the node ecosystem's fault, when you have a developer that is trying to implement SQL on top of NoSQL and pull in a gigabyte of dependencies for an embedded platform. If a C++ developer was trying to implement the app on top of MongoDB's source code, that says more about the developer then about C++ and its ecosystem.
I might concede that the node ecosystem does have a disproportionate number of people with this mindset though :)
By the way, cross-compilation is also an option, although I haven't personally tried it:
A plus would be that only native modules have to be compiled, and only once. After that its possible to set up downloading them as binary blobs.
You got me curious about the application, by the way. If you let me know what it does, it might be fun to try and implement a prototype in node (with sane dependencies and easy to deploy on a RPi)
Just to be clear in case I've given the wrong idea, my point here is not really to criticise Node in general. I'm just saying, with the benefit of hindsight of course, that it was not the ideal choice for that particular job. In particular, because of the hype around Node, I think the people making the decisions (including the developer who was working on it) honestly didn't expect that, and perhaps weren't sure how to proceed when the problems I mentioned started mounting up.
(Edit: That said, the difficulties with size, portability and dependency management in the Node ecosystem that were exposed by this particular project are more general flaws with that Node ecosystem and not specific to this particular developer.)
You got me curious about the application, by the way. If you let me know what it does, it might be fun to try and implement a prototype in node (with sane dependencies and easy to deploy on a RPi)
It really wasn't anything very complicated by database standards, just storing some settings in a central place. Those settings were a bit more complicated and inter-related in this case than you could conveniently store using something as simple as a plain text file, so a lightweight database was the next logical step. As mentioned before, it was the kind of thing where you could implement the basic infrastructure required within a few hours using more suitable tools.
Okay. I've had vastly different experience with node on ARM. Much more reasonable dependencies size then what you report (by about 2 orders of magnitude) and excellent performance from the JIT. Combined with the ability to first write dynamic code then switch to static types, as well as the memory safety, its been a lot more pleasant than working with C.
So honestly, I really can't relate to most of what you said here :)
Please keep in mind that the Raspberry Pi was only an example. We're talking about embedded here, so there are all kinds of weird and wonderful CPUs and storage systems and I/O components being used in various devices that need to run the code I'm talking about. Certainly not all of them were ARM chips. I don't know which chips caused the big problems with porting Node, nor which specific packages, so it's entirely possible that whatever you've worked on was closer to the mainstream and supported just fine.
This "I love [some language]" thing seems to be quite common in JS circles. Maybe that is part of the problem. Being in love is not good foundation for making objective decisions.
It is one of my most cherished professional goals to avoid ever working with people who think like this. The message this sends to junior devs is so backwards and wrong-headed that it defies discourse.
But I see your point as being precisely backwards: it's the JS crew who constantly evangelize for their way of doing things, and that way of doing things is completely inappropriate for anyone who isn't already expert.
"Choose your own tools" is advice for experts, and literally nobody else. It smacks of the cowboy attitude, and that attitude is wildly unhelpful to almost anybody doing professional work in software. Frameworks and coding standards exist for a reason: not to be unreasonable strictures, but to provide guidance and sanity in a staggeringly complex field of endeavor.
Many new devs and junior devs are flocking to the JavaScript ecosystem, and it is one of the most troubled and chaotic ecosystems in all of software right now. Anyone who is not a complete JS badass is simply not going to find Node to be even a decent choice for learning best practices pertaining to the larger world of application development. And I feel I'm stating that politely.
So: yet another JavaScript Pro tossing out the "choose your tools like a Pro" advice-morsel is part of the problem, as I see it. I don't want to work with people who toss the kids into the deep end and hope a few can learn to swim real quick. I think people deserve a helping hand and a reasonable set of expectations.
The JavaScript ecosystem has become self-parodying. Anyone who is oblivious to that fact is inherently NOT a trustworthy witness. That is not to say that the whole thing is rotten and worthless, but it IS messy as heck, and refusing to acknowledge that is a sign that one is in denial about some pretty blatant facts.
> JavaScript in general is in a huge influx of updating at this moment
Isn't that the case basically all the time? Which is a perfectly suitable reason to avoid it at all costs except for the bare minimum required for front-end..
No, ES5 to ES2015 is a HUGE leap. ES2015 to ES2016 is a very minor jump as it only adds 2 new things to the spec.
> Which is a perfectly suitable reason to avoid it at all costs except for the bare minimum required for front-end..
And honestly, that is not my job to tell you not to do. If you want to avoid Node.js go ahead. I feel Java is something I should avoid at all costs. Doesn't make Java any worse or better. Same with Node.js.
You seem to enjoy switching tools and methodologies and see it as part of embracing Node.js.
Others don't have that luxury and just can't afford that.
And if your software needs to be around for more than 5 years, you want to keep your costs low.
How long would it take you to find a bug given a stack trace from a Node API with close to no clue which part part of your app registered that callback that isn't called because of the error?
Chances are, you don't even measure the time spent in code archeology as I like to call it.
The time probably is not significant right now as you kinda know your way around your code base...yet.
> You seem to enjoy switching tools and methodologies and see it as part of embracing Node.js.
I've been using the same tools for the last 7 months outside of switching from Angular 1.5 to React because I'm not a fan of Angular 2.0.
> Others don't have that luxury and just can't afford that.
Then don't? What is this sense of needing to change everything because something else comes out, that developers have?
If your current toolset works, why change it? Why rewrite a monolithic app in another fancy language or tool just because someone wrote a blog post boasting about that new tool/language?
> Coming from other languages such as Python, Ruby or PHP you’d expect throwing and catching errors, or even returning an error from a function would be a straightforward way of handling errors. Not so with Node. Instead, you get to pass your errors around in your callbacks (or promises) - thats right, no throwing of exceptions.
Promises let you throw errors normally. They will propagate up the call stack in a similar manner. With bluebird, you will also get full stack traces in development mode and the performance penalty for that isn't too bad.
> The last thing that I found frustrating was the lack of standards. Everyone seems to have their own idea of how the above points should be handled. Callbacks? Promises? Error handling? Build scripts?
Promises are in ES6 (i don't think it gets more standard than that) and have well defined semantics, including error handling, shared between libraries: https://promisesaplus.com/
I know that Bluebird's promisifyAll might seem like a bit of a hack, but just try it out. It works surprisingly well, and its really painstaikingly tuned for near-zero performance loss. It will probably be both less painful to do and more performant than any manual attempt to wrap a callback based API into a promise one.
At the time the solution was "use domains", but I think domains are deprecated now. It was painful enough that I have stuck with Rails since then. I'm glad to hear that Promises are an improvement!
Domains have been deprecated since at least 0.10. As of yet there's no replacement for them and all node apps should be using them. There's no other way to catch "But ... but ... that can't happen!" type errors.
There is no replacement because they're a fundamentally broken idea. They require the following to happen, in that order:
* V8 needs to optimize try-finally
* Node core needs to add try-finally at every single place where callbacks are invoked and make sure all state and resource cleanup is properly done to support domains
* Popular libraries need to also add try-finally handlers for the above.
As to why this is a problem in node and not so much in other languages, its because with node callbacks, the call stack goes both ways. In other languages, libraries mostly call their dependencies' code. In node's CPS style, you call the library but the library also calls your closure code. The semantics for the 2nd part aren't well defined in node - the loose law basically says: I wont call you twice, I'll try not to call you synchronously, and you wont throw (and if you do the behavior is undefined).
With promises there is a contract and its enforced by the promise implementation. Since Promises actually have error semantics, you can build resource management strategies on top of them. http://promise-nuggets.github.io/articles/21-context-manager... - and consequently there is no reason to crash your server on errors.
Kind of related: One thing that has bit me a few times with promises is accidentally missing a return in the promise body. As such whatever called the promise just sits there forever, waiting for something that will never come. Is there a way in Bluebird to have it throw an exception or similar if the function executes without returning anything?
I sort of don't understand your problem. If you forget to return a value won't the thing consuming the Promise just get undefined as the value (and perhaps throws an exception expecting the value to be something other than undefined)?
I agree with this article and find the Node.js community, and to a lesser extent Javascript itself, exhausting. It seems like every 2 minutes there is a "more" proper way to do something, which tells me that the architecture is not yet mature, even though it's pretty old by now.
On a related note, it seems like every time you find something that doesn't quite work correctly or conveniently in Node.js there is a "fix". Don't like callbacks? Force Node.js to look more like other "normal" code and use Promises. Having trouble getting Node.js to concentrate on one thing at a time? Force Node.js to look more like other "normal" languages and use the "async" library. And it just goes on and on and on. If I have to use all of these other pieces and parts to be productive in Node.js I may as well just use some other language.
I really wanted to like and use Node.js, but Javascript and the community are holding it back.
Yes, the Javascript world is quickly evolving both on the front end and backend, and of course that can be exhausting but I have to disagree with both conclusions that (a) this means the tech is immature, and (b) the community's readiness to make changes is "holding it back".
The js world is very unique in its ability to evolve quickly and things have improved massively over the last few years. Now, apart from the churn itself and precisely because of that rapid evolution, front end and backend development in javascript is amazing compared to most other options (especially on the front end). So while you do have to be realistic about the cost of the evolving ecosystem, it is just a tradeoff for rapid progress, not a flaw.
If you crave stability, agreed, this rollercoaster probably isn't your ride. But the evolution of node, the emergence of react/redux, es6, etc is amazing and beautiful IMO and I'm totally enjoying every bit of it compared to the staid mediocrity of my Rails, C/C++, and Java history.
I'm not saying that Node.js is bad and shouldn't be used. Rails was in the exact same boat several years ago; everything was changing so quickly that new people on the scene had to work pretty hard to keep up. Being on the other side of that process with Node.js, I can now understand the Rails newbie's frustration. :)
If you are comfortable with the constant change and evolution in a platform/language/framework, then by all means continue to use it. Just don't disparage people that come forward and point out the pains they have and say that everything is fine. Just my $0.02.
I'm still not all the way on board with promises - from the sounds of it this article says there is already a successor down the pipe.
As you've pointed out some of these libraries make things behave a lot more similarly to other languages. And in such a case you may be better off just using one of them. However you're giving up on a lot of the power and benefits javascript provides, so of course you'd want to use something else.
For all of its pitfalls and annoyances, javascript is too powerful a language to ignore. Asynchronous operations and streams are just too good. They are difficult to balance but once mastered you can really do a lot with them.
My recommendation would be to skip libraries where possible. Use raw ES6, or Typescript. Something javascript has been suffering from for a long time is bloat from external libraries. You don't really need to use them, especially now that the language has more or less settled down.
All of the best libraries were used to normalise things and that's not really needed anymore.
How is JavaScript so special with async and streams? JS lacks any syntax for async, so it's not on par with languages that do. What's special there? Same for streams: they're implementable in any language, and JS has no special capability there. Or am I unaware of something JS has that other languages lack (besides popularity)?
When javascript performs an asynchronous operation it doesn't block the thread. It stacks everything on a queue and then just works through everything as fast as possible. Any other language would require a separate thread for each operation.
I'm sure there's one other language out there that does it too but I don't know what it is.
Event based programming effectively means that as long as it has something to do it's doing it. You tell it to wait? It says great I've got all this other stuff to do.
Streams you may have me there, I didn't realise those were so widely used in other languages. Javascript works pretty well because you can run it on the whole stack. Share code between the front and back end. Stream from one place to another and around the corner.
Vert.x basically gives you everything that node.js has - and it spins up several event loops, making your code scale without having to run several node processes.
It also allows you to mix in worker threads that are allowed to block (for I/O, for example).
You can develop in JavaScript, Java, Ruby, Ceylon, Groovy.
I think it's more a matter of it being the only way to get things done in JavaScript and Node.js specifically (an overstatement, but that seems to be the sentiment).
Never had the problems with Node described in the article. Node forces thinking about modularity and composition though, which I'd wager is what the author is actually struggling with. Write functions that do one thing and they're pretty easy to compose, even if they're asynchronous. It's really not hard to debug what went wrong in the stack trace when you name your functions.
And I think exception handling is much worse than passing errors up through callbacks. It forces you to think about edge cases. Not sure how python handles this, but I can't tell you how many times I've seen Java or C# code swallow exceptions which is much harder to debug IMO.
People really seem to have a problem with there not being "the one true path" in Javascript, but it's not something that gives me much anxiety. Javascript is incredibly moldable, which is part of what makes it so powerful.
Author actually wants to use Python. Used Node.JS regardless, for whatever reason.. It did not work the way Python works. Author is frustrated. Complains that JavaScript is not Python.
> You use Grunt!? Everyone uses Gulp!? Wait no, use native NPM scripts!
Although couched as a criticism this is actually the community fixing itself. The evolution from Grunt > Gulp > npm scripts is movement away from needless complexity towards simplicity. Npm scripts are effectively just Bash commands that build and manage your project, which sometimes employ small, unixy tools written in Node.
This self correction was pretty quick, it happened within a few years.
> Unfortunately, there isn’t any one “standard” (like everything else in Javascript) for implementing or using Promises.
Yes there is. It's called the Promises/A+ spec, and its built into ES6.
So somebody found a project somewhere on the internet with an exceptionally complicated build process, and you use it to say npm scripts are broken? Sorry, that's absurd. Looking at that particular build process, I don't think a Makefile could have been crafted to make it much simpler or smaller. In that example, the problem lies with the complexity of what they're having to do, not the tool.
Npm scripts are really just shell scripting, which means all the real progress happens in the unixy Node tools that do the heavy lifting, where it should be. It's a future proof and scalable approach for the vast majority of projects imo.
It's an inflated example of what all npm script projects become, imo. First you just have "test", then you add "build", then you separate your "test" into one for the browser, one for Node, one for CI, then you need scripts that combine those together; then you create different "start" versions depending on environment... It blows up quickly.
> Npm scripts are really just shell scripting, which means all the real progress happens in the unixy Node tools that do the heavy lifting, where it should be.
That's fine, but you're missing critical features that Make provides; make won't even rebuild a target if no files have changed.
Admittedly, my first 2 or 3 npm scripts based build processes did start to get a bit ugly. But I'm much better at writing them now, so they stay pretty sane.
> That's fine, but you're missing critical features that Make provides; make won't even rebuild a target if no files have changed.
WebPack does this for me too. Also my ava tests don't rerun for files that haven't changed while watching.
Genuinely curious, what scenarios precisely do you find this feature of Make useful?
Also, Make isn't truly cross platform ... and since I sometimes work with Windows devs this would be a problem.
I completely disagree that this is what all npm scripts become
>then you separate your "test" into one for the browser, one for Node, one for CI,
For the love of God why. Don't turn every language into Java.
Plenty of tools will not rebuilt a project if no files have changed, but the dumb, quick and easy way that requires no deendendencies is simply to delete and rebuild.
The fact that this functionality is served so well by a simple dependency shows that you don't need overly complex monolithic tools to accomplish small tasks.
Whether the Promise/A+ spec is bare bones or not doesn't matter, the reference implementation is the ES6 native Promise, which is part of Node and now natively in browsers too. This is concretely what a promise _is_, use a different sort of promise at your peril.
I've spent a lot of time writing Javascript on the front-end in the last year using both React and React Native. I've found the React ecosystem to be a sane, productive and enjoyable development environment.
Interested in sharing more model logic between our front- and backends, I also investigated writing some new backend features using Node (we're currently developing with Rails). But after days of research and playing around with the available options, I came to a conclusion similar to Gavin's -- any reasonably complex backend requires you to either roll your own everything, or try to cobble together literally hundreds of tiny dependencies that weren't built to go together, and then somehow keep track of all of their regressions and breaking changes.
Node's fantastic performance and unique ability to share logic between client and server are enticing, but I just don't trust the community and "best practices" around it enough to bet the farm on it for now.
The same problems you cite for node.js are the reasons why a lot of devs love node.js.
Its much easier to do your own research and find the best module to solve a particular problem you are having then to shoehorn into some larger monolithic framework.
Also its a lot more fundamental then that - Node.js has prolly the fastest iteration cycle for any platform out there since its so easy to create your own module - it leads to some sort of Cambrian explosion of innovation and experimentation.
EDIT:
Also OP seems to think callback hell and async programming is bad. The important thing is those things are problems for python/.... too !
Its just that python doesn't have a good programming model to even begin to address those concerns.
JavaScript at-least tries to say - "hey this is a problem we need to deal with - concurrency is a issues we all face "
So when devs complain about callback hell - its just that they have never tried to use python to do async in a neat way.
> python doesn't have a good programming model to even begin to address those concerns
This isn't even close to true. Anything JavaScript has to express logic in the face of asynchrony, Python has too. There are a half-dozen asynchronous web servers written in Python.
There's not as much of a culture of writing APIs that way in Python because it's generally a terrible way to program, and threads/OS processes are good enough for basically everything except HTTP servers with absurd numbers of concurrent connections.
> There's not as much of a culture of writing APIs that way in Python because it's generally a terrible way to program
I would love to see evidence for you making that statement. Almost every programmer would put out their fav programming language as the 'right' way to program.
> everything except HTTP servers with absurd numbers of concurrent connection
Once you introduce async operations in your code - you need to follow the execution path through. http request can be async - but then what if the http request results you doing a db lookup or some form of file handling ? you need to make the whole thing event driven.
It's not like we didn't have cooperative multitasking for 50 years. Having threads/processes and a scheduler is easier and safer, full stop. Potentially long-running portions of the program don't need to be arbitrarily chopped up to yield control back to the server, because they are pre-empted. Your system is no longer at the mercy of the worst code within it.
Both nginx and Apache's event MPM handle HTTP connections with events while the app backends are still using preemptive threads for running the HTTP handler code, so it's clearly not the case that "you need to make the whole thing event driven." You just need programmers who don't think, "well since the browser doesn't expose threads to JavaScript programmers, clearly they are useless."
That Cambrian explosion is fun when you are programming, but if you are writing a product, you will want to make sure you don't pick a dinosaurus or trilobite to build it on.
* Async libraries make it easy to make things high performance.
* I like Sequelize as an ORM, once I figured out how the async everything works.
* The testing support is pretty good, both mocha and e2e testing using selenium
* The angular-fullstack generator was really helpful for getting started and setting up the deploy to Heroku.
* everything is open source. If I get confused with what a library is doing while I'm debugging I can just stick a print in the lib temporarily.
Things I don't like:
* "undefined is not a function". When something goes wrong. This is the error I get 80% of the time.
* async stuff silently swallows exceptions unless I put try{} catch(err) { console.trace(err) } everywhere
* There's a bit of a learning curve with promises.
* Needs a lot more automated testing than a really strongly typed language like Scala.
* Single Threaded. I know how to program using threads, so I view this as a disadvantage.
* No types. I have a lot of type checking asserts at the beginning of dao functions.
If I did it all again and my teammates would oblige, I would have probably chosen Play/Scala. I actually reimplemnted things from a Play/Scala project I did a while back (login/signup/forgot password/confirm account) and it took less time in Play/Scala, even without passportJS and friends. I have about a years worth of experience learning Scala before I started that project, so it would probably take longer for a new Scala developer.
The async stuff is not really any better performing than e.g. async in C#, which is mature, easy to debug, and has been around a while.
I don't know why people keep saying node.js is "high performance", the numbers I've seen basically put it in the same ballpark as similarly architected apps in other languages, as long as the other language has a reasonably performant toolchain. Perhaps people are used to old Ruby on Rails performance problems?
> async stuff silently swallows exceptions unless I put try{} catch(err) { console.trace(err) } everywhere
Huh? That's not how async errors work in Node. Try/Catch is not async, the catch block will not magically transfer to your callback function.
You check for the error as the first parameter in your callback, that's the standard way of error handling. Throwing errors in Node is considered by most to be an anti-pattern.
That's nice in theory, but third party libraries may throw exceptions if unexpected things happen at runtime. You'd still have to put the try catch block there and call the callback in the catch block everywhere.
That's only for non-async stuff, which is very rare. If you're using some wacky 3rd party code which throws an Error instead of the accepted convention of (err, response) callback arguments (or returning a promise), I suggest not using it.
We couldn't even stand 6 months let alone a year. Broken pack depts make it hard to CI, poor performance, memory leaks, huge docker base images due to deps, lots of single cpu only tasks that were too hard to scale out, an ugly language compared to ruby or Python and on top of all that how poor the package management with NPM has been. We've just ditched it and gone back to Python/Django/flask and ruby for the ops tooling.
Same experience as the author. Tons of reinventing the wheel, and hundreds of dependencies.
The majority of established companies that say they use Node in production are doing so as a fancy proxy. All the "serious" stuff is done on backend services written in other languages.
Moreover, asynchrony is a concept far more advanced than most people think.
We will likely continue to leverage Node as a fancy proxy. Adding Typescript will only help. But it's likely that as Node grows into a mature platform, other platforms will continue to fill in the gaps that Node filled.
See for example Node constantly adding [foo]Sync versions of methods, while Python adds first class sync support.
I've been on a similar learning curve with Node over the last year, and it has certainly been a rougher incline than other languages I've used.
The whole async situation needs to settle down, it's completely unacceptable to write code with callbacks, promises, etc. This is because they are not just challenging to deal with, but intrinsically wrong in concept. I have to wait for a database query to complete, then pass the next thing to do to the callback of the query? That can't be right.
I find that JS often seems to tie programmers in the most extraordinary knots just to implement even quite simple logic, because of the single-threaded nature of the language.
In the programming model used by most other mainstream languages today, if you've got some work to do that interacts with some external system and might take a while, you'd probably start another thread for that task. You'd write the required logic in the usual linear fashion, and just let the thread block if and when it needs to. Modelling this using fork/join semantics and techniques to co-ordinate access to shared resources from different threads are reasonably well understood ideas.
Because there is no general support for concurrency and parallelism in JS, you only get one thread, and so in most cases you can't afford to ever block it. Consequently, you get this highly asynchronous style that feels like writing everything manually in continuation passing style, just so you can carry on with something else instead of waiting. That in turn leads to callback hell, where you start to lose cohesion and locality in your code, even though usually you're still just trying to represent a simple, linear sequence of operations.
Async/await help to bring that cohesion and locality back by writing code in a style that is closer to the natural linear behaviour it is modelling. However, even those feel a bit like papering over the cracks in some cases. Async/await kinda sorta give us some simple fork/join semantics, but as the blog post linked from the parent shows, we have a lot of promise-based details remaining underneath.
Fundamentally, the problem seems to be that JS is increasingly being used to deal with concurrent behaviours, but it lacks an execution model and language tools to describe that behaviour in a natural, systematic way as most other widely used languages can. Being strictly single-threaded avoided all the synchronisation problems in the early days, when the most you had to worry about was a couple of different browser events firing close together and it was helpful to know the handler for one would complete before anything else started happening. I'm not sure it's still a plus point now that we're trying to use JS for much more demanding concurrent systems, though.
> Modelling this using fork/join semantics and techniques to co-ordinate access to shared resources from different threads are reasonably well understood ideas.
Writing thread-safe code is anything but easy in languages that support threads.
Dealing with shared state is not always easy when you're working with multiple threads. If you can't reasonably avoid that sharing because of the nature of your problem, and if your choice of language and tools only provide tools on the level of manual locking, then I agree that writing correct, thread-safe code has its challenges.
However, there are plenty of scenarios where you don't need much if any state to be shared between threads. That includes almost every example of JS promises or async/await that I've seen this evening while reading this discussion and the examples people are linking to.
There are also plenty of more sophisticated models for co-ordinating threads that do need to interact, from message passing to software transactional memory. These are hardly obscure ideas today, and I don't think anyone could reasonably argue that for example message passing makes things complicated but async/await/promises make things simple.
I have function requestFromService and another readFromDB. On a single thread, I issue one request to a service and one to the database server, and you think it makes sense for the thread to just wait around? In javascript, the second I issue those two requests, the thread is available to now serve the next incoming request or do literally anything else while those services take as long or as little as they need.
The code itself could not be more clear as to what will happen, except for maybe the syntax Promise.join(), which was only there to illustrate how easy it is to do concurrent, asynchronous tasks.
You don't have to think twice about what to do while those requests carry out, and it's not like you had to write your code in a complex manner to take advantage of the asynchronous nature of JS, it just inherently works that way. Maybe I have gone and served 5 more requests while the db server was responding, and I didn't have to even be aware of the concurrency whatsoever.
I'm not disputing the architectural choices behind Node's async model. I'm talking about syntax. It's cool that you can process multiple promises at the same time, but most of the time I need one promise to complete before continuing. In those cases the way it inverts regular code flow is very unnatural. Like I said, I think more support/usage of 'await' will help.
Callbacks are ugly, but are probably the semantically simplest way to handle asynchronicity. Promises are ugly too, but are semantically the same thing as async/await. I agree that promises and callbacks are not pleasing to the eye, but they are completely logical ways to do things.
And if you're using regular callbacks, forget it: you'd have to nest do_other_things in the callback of get_username(!). It just makes things very awkward.
I understand what you mean about sync vs async calls. I won't pretend I have a better solution. I don't mean to say callbacks are illogical, just a bad way to write programs. So maybe not 'wrong in concept', I can concede that. But I think async/await can make things more readable again, i.e.:
var username = await get_username();
/* carry on... */
Event streams are similar to Promises in that they provide a value for something that will happen in the future. In the case of event streams it is a value for an event. In the browser this might be a user clicking a button. This event can occur many times. It might never occur. Promises are values for something that will happen in the future, but only once.
Use Promises for things that happen once, event streams for things that happen 0 to many times.
I'm not sure I'm seeing your argument against using promises over observables for single events. Observables are a superset of promises, sure, but you can map them into situations where you'd use promises and then still have the flexibility of observables and all their operators. along with this you get the consistency of using the same async paradigm everywhere.
I find you run into trouble when you start mixing them. If you use observables for 0-N events and promises for 1 time events you're always converting between the two.
Yes, I too find callbacks an obstacle to readability, among other things.
If one uses mongodb (popular on node.js), and one needs multiple queries to 'join' several documents, the cascade of callbacks is an obstacle to readability, let alone debugging.
If you're writing a cascade of callbacks then just use promises. There's no reason to be needing to cascade callbacks pretty much all libraries now support promises and the rest can be promisified.
I'm not calling it a Panacea but Promises are idiomatic Node Javascript now and there's almost no reason for mutliple levels of nested callbacks anymore.
sorry if that came off harsh, but that is actually the feedback, and it's valid. whether or not javascript/node is better or worse than python, it's pretty clear that bringing a python style approach to nodejs is going to cause problems, especially with error handling and async stuff.
I have used Node.js in production for about 5 years now and I must agree with the sentiment that JavaScript is "Easy to learn, impossible to master". Yes, error handling is a little confusing at first but it stems from JavaScript's asynchronous nature which is naturally complex for the linear mind.
My personal sentiment is to just use Promises, like everywhere. ES7 async/await will really help with this too.
For web applications, I've been very happy with the up-and-coming Phoenix[0], a framework for the Elixir language.
Very well-designed and thought-out, fast, productive. Leans functional and immutable rather than object-oriented and mutable. It's kind of like Rails but without most of the problems.
I've found Elixir to be a delightful language with very palatable syntax compared to ruby. It has been a really enjoyable transition, with some vague reminders of the parts I really like about JavaScript.
I've been using Node in production for a few months now, having come from Ruby (Rails & Sinatra) immediately before, but having used JavaEE and PHP before that. I find it... fine.
For error handling, bluebird's typed error catching (http://bluebirdjs.com/docs/api/catch.html) is working well for me and I'm finding it analogous to my experiences with Java and Ruby.
I'm rather used to using ORMs as well and I use Bookshelf (http://bookshelfjs.org) on top of Postgres for this as well. It definitely has room to grow, but it's also fine.
I think the biggest concerns on which I'd agree with the author are the pace of the community and lack of agreement on things which are well-decided in other, more well-established development environments.
That being said, there's huge potential with Node because of that. There are more coders in the world than ever (I'm assuming) and Javascript is a great low barrier to entry language that encourages people to explore various runtimes. In 20+ years of coding, I've not seen this level of excitement and engagement in a development environment. While it may be rocky for another few years yet, I suspect we'll end up with a very productive platform, simply because of the amount of involvement. Of course, it's totally understandable to want to wait for that before jumping in. :)
As a relatively new Node developer, I'm much more concerned about the single-threaded nature than the development environment, but so far even that hasn't been a problem.
A lot of this is residual effects of the (slow) evolution of JavaScript. It was thrust into the spotlight missing a lot of features and these features have only recently been fixed by the language itself.
But Node has been around since 2009. 2009 JavaScript was missing a lot of features. It was basically a runtime only advanced users should use. The Node maintainers had to make a lot of decisions that now conflict (to some degree) with fixes that have come later to the language. They chose their callback style; now we have Promises. They chose to throw in methods rather than return an error in the callback (this makes it awkward to use fs with Promises without a wrapper library). They chose to implement their version of CommonJS, now we have the .mjs issue arise.
Been on Node and Mongo for 4 years now. Both have worked well for me.
JS has evolved a lot in the past few years, and with it came all the new shiny tools that left us confused.
I think a year was too short for the author. Sounds like they were chasing after every cool thing to make life easy.
Error handling is a pain yea, I've seen amateur folk try catch this and that, I think that lends itself to being terrible.
One of the things I appreciate the most about JS is JSON. Crafting tens of classes in Java irritates me. I find Python sometimes tricky also when dealing with structs VA lists.
I've always stuck to the basics when I was learning how to JavaScript with Node. I used only callbacks for 2 years until I understood what my code was doing. Granted, I'd have spaghetti at the end of complex async queries, but I understood what was going on. I moved to caolan::async and have been using async whenever necessary. I barely use promises as I got confused by the early adoption craze.
I learnt how to use Backbonejs, and a bit of Angular+React+Ember, but I found myself comfortable using vanilla JavaScript. Only thing I use is Underscore templates. I know I could benefit from shadow DOMs etc, but I'm content where I am.
I think a good way to learn is to take things at bite sized chunks. I've started using RxJS recently, and I'm loving it! I'll keep using JS as my primary tool, but I'm slowly moving to Kotlin.
If this article is true, it paints a concerning picture. I don't want to research libraries, understand the pros and cons of them, run into problems, then switch to another library, and so on. I want there to be a default that works out of the box for the majority of use cases. There can be alternatives, as long as there's a default that works out of the box for most people and most use cases.
My recommendation is to just use songbird (which exposes the forthcoming promise API from core, built on bluebird), async/await and the async `trycatch` library so you don't have to worry about a 3rd party package's choice of asynchrony. It also comes with optional long stack traces.
in my experience, async/await is a great way to layer indirection and obfuscation over what is still callback hell. it may look better in the editor, but it's hell to debug.
When I initially encountered event -based processing (libevent in C), those callbacks were indeed difficult to wrap my mind around. But I learned to structure the code in the editor, keeping everything together which made it manageable.
I 've worked with node for a couple of months now, and I find promises to be more confusing, because it obfuscates the callback in my mind, and makes it look like ordinary function calls.
The advantage of promises is that you can return them as a type, you can't do that with a callback. If you fetch something from a database , your data access object can return a promise and let the client code deal with the result. Promises are composable by nature, callbacks are not.
async/await is tiny amount of sugar over generators and promises (virtually the only thing that changes is `yield` -> `await` and `function*` -> `async function`), not callbacks.
EDIT: Oops, I misread the parent. You're right, async/await isn't easy to debug, but it's getting better, and it's a tradeoff. For now, I recommend debugging the compiled code (using generators) without source maps to avoid some of the quirks of source maps. In my experience, the tradeoff is worth it for new node.js developers because it offloads the asynchrony contract (e.g., calling a callback OAOO) to the language.
Really, just learn to use callbacks properly. Well, wait, actually, you should skip that and start getting used to using promises instead--a big improvement. Well, I mean, until next version is ready, and we can start using async/await...until wasm makes better thought-out languages available.
Despite the sounds of this, I do like the idea of having an experimental platform with which to gain experience using wildly different approaches to the wildly changing world of web apps. I don't take it for granted that old language concepts will turn out to be the most useful for the web platform.
So, I'm okay building a website for the PTA or ceramics club with this, and I'm very interested in the experiences of others using a wide variety of technologies and approaches, but I'm not sure Node.js would be a sensible foundation to build a business on.
This article definitely matches with my experience. I did two not all too complicated projects on the side with Node.js and the the first steps where so easy that it completely convinced me to go with it.
After a while I tried to dig deeper and went to meetups to see whether I do things right as in a community accepted way and if I use the right tools and so on. Since then I refer to the Nodejs community as the most hipster programmer community I've ever seen. As soon as a framework was getting near a 1.0 version nobody wanted to use it anymore, experimental features were used in production code, it was horrifying.
For me that ended this endeavour, I just could not keep up with the pace. I always wondered if it would be different if I would have worked full-time on Nodejs projects.
Interesting analysis. I just finished a year of using Node for implementing an HTTP API and a chat server, and found it to be actually pretty pleasant. I'm not chasing the latest and greatest things, there's no ES6, no ORM, and I'm on an older version of Node. But it works and has actually been quite stable! The things I've missed are static type checking at compile time, and execution speed (which is less of an issue when you're talking with databases all the time). I'd be happy to write in more detail if anyone has any questions, but I found I had the opposite experience of this author. The situation makes all the difference though.
Most of the problems described in the articles can be solved by simply using promises. Error handling is centralized in your chain and any function can throw and just like Python or Ruby you can catch anywhere you want. As for consistency between callbacks, promises and generators, just pick one. We switched from callbacks to promises and it was a great move. Error handling greatly improved and the code looks a lot nicer. No more callbacks hell. Ever.
We switched our backend from Python to Node almost 2 years ago now and it was a great decision for us. If you handle a code base that deals with a lot of async requests Node is definitely a top contender.
I think by reading the comments here and the article, its apparent that Node.js just isn't as mature yet. If you know what you're doing, it can be great, but for the noobs the right way to do things is not easily apparent. Couple this with the huge choice in libraries and frameworks, makes Node.js harder for now. I think the base is good though and given more time it will become more easy to wield. This is typical of any new tool. And yes Node.js has been around the block for a while, but it is still newish compared to Python, Ruby, PHP, etc. So you're going to pay a new adopter tax still.
Can't fully agree with the article re: using nodejs in production. A couple of years ago I decided to use nodejs for a rewrite of a web/database app that had gotten to be complex and hard to maintain.
As so often said, it was true for me that the abundance of modules and choices in node was at first very confusing.
I eventually figured out that keeping things as simple as possible was my best approach. What I came up was a server relying on very few module dependencies and written using consistent if not so elegant components. Sure it's kind of verbose and far from totally DRY but fairly easy to understand, modify and extend.
Key issue was node's "callback hell" style of async programming. Of course, it's not just node, other languages (Scheme, FP, etc.) can be mind-bending in a similar way.
The callback "inside-out" locality inversion was initially hard to grasp, but once I caught on it was possible to get the server working the way I needed it to. The more recent development of promises, etc., certainly provides reasonable ways to reduce the high barriers implicated in using the nodejs style of async programming.
In my opinion, JavaScript is a toy language. Looking at it, and the Node.js ecosystem by-extension, that way has really helped me be effective with it.
Tooling has never been more important to my productivity with a language as it has been with JS. I constantly search for tools to paper over the warts and potholes.
The only clear, objective advantage of using nodejs is that if your application is I/O heavy (e.g. tons of sql queries which can be executed in parallel), nodejs event system is helpful and everything you need is pretty much out of the box. Other thing... it largely depends on personal taste and a matter of convention.
Reading his "Why I’m switching from Python to Node.js", doesn't seem like he was having an issue with that. And I don't really buy into the "same language everywhere" argument because come on, how hard is it to learn python, ruby, etc enough so that you can be productive? Not hard at all, unless you have hundreds of cubicles filled with drones.
Anyway, it's good to see that he made some reasonable conclusions after the experiment. That's a good sign :)
Same language, in theory, is great. Reuse structure definitions. Reuse rendering logic or even validation logic (perform checks on both, but make it easy to get into client-side). In general, keeping things "in sync".
Also can make it easier to write in SPA style but offload rendering to the server when you need it (particularly first-page or reloads).
Whether or not tooling is good enough to allow this (either with JS or compilers) is another issue.
We use NodeJS pretty extensively at Yahoo for both front end and back end services and it works well. While some of the complaints are valid, it's not worth flipping tables over.
Promises are standard in ES6 and it is "the way" to handle errors. At least if you want to stay sane.
Though I agree with what you all explained with the big issues with NodeJS, you have to understand its not all of Javascript. Its only server side JS. The creators have clarified, for heavy CPU intensive services, go away from NodeJS. As far as frontend is concerned, AngularJS and React are just sugar candy for user interaction and structure.
Node filled the gap with an async network application in pure JS without the heaviness of Python or traditional languages.
It is a hack as in every day people are finding new ways to use NodeJS but I agree, there has to be best practices and less boilerplate (plus less silly npm packages for trivial JS tasks).
I think the switch to an async back-end can be more initial work than many expect. It may take some time to feel as productive, but promises become powerful and became a game changer for me over my previous work with callbacks. Error handling also becomes manageable.
What I really enjoy is jumping into new community and getting to work with tools that have built. Choosing the right ones can make or break an experience. I personally enjoyed working with Express and Bookshelf.js/Knex.
I appreciate the authors perspective, but I also don't think this should deter anyone from trying out Node. I personally have no overwhelming preference to using a Python or Javascript stack.
After ruby orms, Bookshelf felt very, very underdeveloped. Anything I tried to do beyond "hello world" only brought me pain, especially dealing with associations, but honestly just about everything. I guess I've been spoiled, but getting anything done in express/bookshelf combo seemed like a chore.
There's a very fine line between between removing too much essential complexity, and leaving programmers with too many limitations. Or the other end where the language overhead overshadows the original task at hand.
For instance multithreading should be handled by languages and frameworks 99.9% of the time. Take reading a file (something that should be handled by the language/ framework). Read it asynchronously, structure any dependant code clearly (which JavaScript does pretty well), if any problems are detected they display/ log the appropriate error. You shouldn't be kneck-deep in callbacks.
The biggest problem is dealing with callbacks (and yes even promises use callbacks , and generators need to be wrapped in a cor-routine framework in order to work as cor-routines ).
I want to write a quick script doing some busy work , I now have to think about synchronicity even though the script does not need to be non blocking. Of course in these circumstances, I want to move back to Ruby or Python, which actually let me code the thing I want to code without forcing callbacks on me.
So when you have to do 20 i/o operations in sequence, using nodejs becomes really tedious.
Node is one of those platforms that makes me shudder every time I go looking for solid best practices. It seems to change every 3 months. It's scary (and cool) how fast things are changing.
I think the basic problem is that JS is not the right solution for every task but for the web, it's often the only tool available.
I had similar experiences to the OP. I had lots of code written in JS that I was happy with to some extent, but all of it would have been easier / cleaner / more maintainable in a less crap language.
Using babel without async await is a mistake IMO should solve almost all your problems and errors.
Also the main argument for using js in the back end is that you have the same team working in the front-end too and in any case in the same langage.
tl;dr: "I really miss Python's bondage and discipline, so I'm going back to it, and I'm going to hate on Node on my way out because I genuinely can't wrap my head around the idea of a paradigm other than the one with which I'm familiar and comfortable."
It's not even about Node. It's about anything that isn't Python, and doesn't have the Python community's strong "there is exactly one way to do it" tradition. I'm glad OP has realized that's what works for him. It's a shame he lacks the perspective to understand that it's about him.
This articulates my fears around switching to node very clearly. I've played with it in the past and felt like it would become too difficult to manage with a large project and lots of contributors.
A year ago one of the reasons to leave Python was poor MongoDB & JSON support, 12 months later the same author complains about the lack of a decent SQL ORM library in js <scratching my head/>
while JS is far from perfect, his problem was he was a little bit ahead of time- babel, standard promises & sequelize solve most of the problems.
I think Python is still superior for serverside, but JS isn't as bad as portrayed in this post, if you slowdown for a couple of weeks to learn how to use it properly (just like any other popular language or framework).
I mused on similar things after getting a very small project shipped. It is based on my experience with Node, Ruby, and Node as of 4 years ago (when I first learned it).
Small company ops guy here - a bit over a year ago, I used to joke with my devs, asking them "So, what's this month's recommended way of installing node?". More recently we've been experiencing dependency hell, which is exacerbated by our small team not having enough time to upgrade to the latest LTS release. Node is definitely a language that you have to manage. It doesn't sit in the background and let you get on with writing stuff.
Is it the right tool for X or Y? I can't say, I'm not a dev. But it does require a lot of hand-holding and keeping current with the zeitgeist.
Node.js, meteor, et al. is a great example of what happens when you let inexperienced developers design a platform and run an ecosystem. Almost every ounce of focus in this community goes to increasing developer productivity. Operational concerns like scalability, security, monitoring, are given the bare minimum of focus. In many years of programming in many languages on many platforms I have never seen a worse platform and ecosystem than Node.js in a very popular language (obviously some obscure languages hardly have anything built around them at all).
I really think it is setting the programming world back a lot. Not to mention what a thoroughly shitty language JavaScript is. Being able to code your web app in one language is a neat trick, and the ability to talk back and forth between client and server so seamlessly does kind of feel like magic, but otherwise, this is a meaningfully worse language/platform that Visual Basic.
- The dependency instability can be avoided by specifying specific versions of your dependencies inside your package.json.
- The fact that the ecosystem is evolving quickly is a good thing. Node.js is still one of the fastest growing software development platforms according to Google trends so you should expect it to change faster than other ecosystems.
- ORMs suck (in every language) - They always sucked; ORMs are a massive hack intended to fix the impedence mismatch between relational DBs and RESTful APIs. If you used a NoSQL DB with Node.js (such as RethinkDB), your life would be much easier. Nobody in the Node.js community except your grandma cares about ORMs because they're considered legacy technology. If you don't like it, then you can stick to your COBOL and Oracle database.
- Node.js lets you choose how you want to handle errors. JavaScript offers an Error class which exposes a name property (which you can overwrite for each error type) and you can also attach custom properties to Error objects (to carry back more detailed info specific to each Error type). JavaScript is really easy to serialize/deserialize so you can even design your error handling system to be isomorphic (same error handling on the client/browser and server). I really enjoy error handling in Node.js/JavaScript - You just have to put some thought into it.
- The way of writing async logic is changing. The fact that Node.js is keeping up is a good thing. There is no single right way to handle async logic. Most well-maintained libraries will keep slowly evolving to use the newer features of the language but it's mostly backwards compatible (many libraries support both callbacks and promises).
- The standards are not bad; they're evolving and you can choose your own styleguides for your projects. Most JS developers will adapt to new styleguides as they move between companies/projects.
