Rather than changing express's syntax, I would much rather have a better express boilerplate generator with an opinionated stack with good documentation/justification.
I recently started learning express/node and wasted weeks googling for blog posts for each dependency to figure out how to configure them. I guess this is somewhat valuable for end-to-end understanding but not for productivity.
There are a bunch dependencies I am now kind of decided on (not for technical reasons, but because they are clear popularity contest winners) and figured out how use:
body-parser
compression
serve-favicon
express-session
csurf
helmet
cors
dotenv
moment
selfsigned
chalk
debug
But still a bunch that I haven't gotten to googling and still need to decide on what to choose:
logging (morgan vs winston)
validation (express-validator, joi)
auth (passport jwt, local, social)
session storage (redis vs keeping it in database?)
database (mongo/mongoose vs pg/knex/bookshelf/sequelize, graphql)
storage (s3 vs gcs)
ajax requests (request vs axios)
mail (mailgun, sendgrid, mailchimp)
error handlers
rate limiting
geo ip
I would be so happy if someone took away my power to choose and just forced me to use something that will "just work".
Totally a tangent but, don’t use moment.js! It inherits some of the parsing and other quirks from the awfulness that is the JS Date object, the mutability almost certainly will lead to being bit by a couple of bugs before you actually internalize it, and the time zone support is kind of tacked on as an afterthought.
I highly recommend js-joda, particularly if you’ll ever be computing/showing things to your users in different time zones. It actually treats dates and times rigorously, and has an api that makes it clear what kinds of operations make sense to do on a zoned vs local datetime and makes things like converting between timezones vs transposing them explicit and simple.
Yeah it looks nicely simple & modular, but it probably works best in Node.js since your servers can all be in UTC.
In general I see the appeal of using a small shim around a standard library thing rather than re-implementing something totally new, but JS Date is bad enough that you're better off staying away altogether. It's just hard to use correctly since there's no "timezone unaware" object available and it always assumes the local timezone, so users' browsers in different timezones treat them differently. Lots of seemingly simple things (e.g. a time + timezone input picker) are easy to mess up because you end up accidentally implicitly converting things to the local time.
I've always been of the opinion that timestamps should be stored UTC datetime or unix timestamp on the server side and then displayed to the proper locale by the client's device. Is this not a good method?
It is not good enough. All the major browsers that I tried do not properly handle the timezone offset, as I discovered a few days ago (I put a more complete answer about this in a stackoverflow answer, but I don't know if putting the link would count as self-promotion so I will leave it out)
In short, the ES5 specs [1] say that
The implementation of ECMAScript should not try to determine whether the
exact time was subject to daylight saving time, but just whether daylight
saving time would have been in effect if the current daylight saving time
algorithm had been used at the time. This avoids complications such as
taking into account the years that the locale observed daylight saving time year round.
This means that if your country handled daylight saving differently in the past, converting the timestamp to string may give you a wrong answer (this would be the case of a number of european countries during WWII).
The spec has been improved starting with ES6 [2], but to my knowledge no browser has fixed the issue:
An implementation dependent algorithm using best available information on
time zones to determine the local daylight saving time adjustment
DaylightSavingTA(t), measured in milliseconds. An implementation of
ECMAScript is expected to make its best effort to determine the local
daylight saving time adjustment.
NOTE It is recommended that implementations use the
time zone information of the IANA Time Zone Database
http://www.iana.org/time-zones/.
A solution is to use a JS library that has its own timezone database.
It's good enough (and correct) to store unix timestamps. For displaying the date - your comment applies...
Unix timestamp refers unambiguously to single point in time. Interpretation of this point in time in different timezones is complicated but it's outside of it.
I agree that storing unix timestamps is a good approach.
My issue was with "then displayed to the proper locale by the client's device."
I mean, by itself it is ok, but in the case of a web client one cannot unfortunately just use the Date object and convert the timestamp to a human readable string.
There are some cases where you not only care about the exact time (unix timestamps are fine for this) but also the timezone in which it was specified. I work on an app where users specify "people in california should see this at 3pm PDT" and get confused when they reload the page and it has been switched to "6pm EDT".
As an aside, moment-timezone it a godsend for dealing with timezone discrepancies. I would pay money for it.
Thank you for this comment. I've been suffering for JS Date quirks for a long time, and haven't found a solution that alleviates the issues I've been having. Looks like js-joda is exactly what I need.
For some reason reading the API docs of js-joda gives me flashbacks to Java programming. Maybe it's all the "public static" stuff. Maybe it's that I can't immediately see how to do something useful. Some examples would be nice.
Never ask PHP/Node/Ruby/Python to do low-level network routing. Your throwing megabytes of memory at a simple connection decision. If you need advanced logic for rate-limiting (not just access counting) then using nginx + memcached/redis as the marker. Have your app mark the ip in memcached/redis so nginx knows not to bother Node/PHP/Ruby/Python next time.
I guess you could simply rate-limit your login/signup if your not worried about a DoS and just want to keep people from brute-forcing a password.
On the other hand, it's not simply "a performance optimization" as there is no way for node to handle a DDoS without crazy amounts of hardware relative to what iptables|nginx can handle.
Couldn't agree with you more. Have been involved in a few projects this year where a bunch of freelancers who had never worked together were thrown together to get something up and running quickly. The results were slightly horrifying precisely because everyone had their own tooling and way of doing things. Opinionated boilerplate would have solved so many problems for us upfront.
No, he and others used to more complete (for lack of a better term) frameworks are looking for the Django/Rails equivalent in Node.js. I got my first taste of Node.js development a few months ago (https://engineering.edx.org/serverless-984cee7797e1) and found it confusing that so much of what I take for granted in Django is kinda all over the place in Node.js. The one that stood out to me was internationalization (i18n). In Django, I mark my strings, run a couple commands to extract/compile translations, and I'm pretty much done. The most popular options for Node.js involved writing a crawler for my site. It makes no sense me that folks are writing crawlers when the static text is on the disk, marked up, waiting to be parsed.
The diversity of the Node.js ecosystem is great because there is clear change and innovation. It's bad, however, because some of that innovation is simply taking things done by other frameworks and poorly implementing them in libraries or other frameworks. The lack of general consensus potentially hurts adoption by those of us who work with Django/Rails and perpetuates the myth of the JS world changing every hour/day/week.
I think you're guilty of the trope that everything should be ported to Node/Javascript.
More likely is that the people who want the features of Django/Rails just use Django/Rails, so there's no goldrush to recreate them in Node nor a monolithic community around the attempts so far.
The Node ecosystem is like the Clojure ecosystem: all-inclusive frameworks just aren't as popular as library composition.
It could just as easily be said that you are guilty of assuming all Node developers want a million tiny modules and all of the decision making overhead that goes with it.
Express is a nice small framework and is good for some things. Sometimes though you really want a monolith. If you are a Node developer there isn’t a clear choice for this.
I actually enjoy Sails, but I can see some decisions they’ve made that probably aren’t attractive to new developers (such as still using Grunt by default).
Waterline is nice as an ORM on the surface, but you run into walls as your queries become significantly complex.
Nope. I want well-defined best practices that make it easy for me to focus on the real engineering challenges rather than boilerplate. The issue I have is that there are numerous, sometimes conflicting, methods of solving the same problems that, in the Django/Rails worlds have one generally-accepted solution. As I mentioned earlier, there are numerous methods of solving i18n and it isn't clear which is "the best." A framework is a potential solution to this problem. Simply getting folks to all align on supporting a few libraries is also a solution. The challenge, of course, is that my "best" isn't the same as your "best," thus we end up with conflicting solutions.
Actually it sounds like OP is looking for something in NodeJS.
I'm also interested in a NodeJS version of something like that even though express is very straightforward once you've worked out all the modules you need.
I much prefer frameworks like Adonis (https://dev.adonisjs.com/docs/4.0/installation) - it feels a lot easier to just strip out what I don't need from an MVC framework, than to end up rebuilding 90% of the same functionality from a bare-bones Express / Koa application.
I'm working on something like that in my spare time for myself. I have a bunch of websites and it was quite annoying to have to update the same thing over and over again if there was a bug with one of my middleware so I just extracted out into it's own project. It uses Koa under the hood, but all it essentially does is bring in the most popular Koa middleware and mesh them all together.
Good list to ponder trade off Node js Express packages , all in one place !!!!!
> But still a bunch that I haven't gotten to googling and still need to decide on what to choose:
logging (morgan vs winston)
validation (express-validator, joi)
auth (passport jwt, local, social)
session storage (redis vs keeping it in database?)
database (mongo/mongoose vs
pg/knex/bookshelf/sequelize, graphql)
storage (s3 vs gcs)
ajax requests (request vs axios)
mail (mailgun, sendgrid, mailchimp)
error handlers
rate limiting
geo ip
For those who don't have too too much time to spend comparing stacks, looking at download trends is sometimes a decent heuristic for weeding out which you might not want to throw into a production system yet. Shameless plug for a site I made to help visualize that
I think it's always just going to be a different-strokes thing.
For example, I left the Rails world hungering for tiny, composable solutions like Clojure's Ring and Node's Koa.
I ended up preferring to just see glue code in my git diffs. For me and the small teams I work on, there's a lot of productivity to be gained when you can just look at the code and understand what's going on.
You end up with bespoke glue code per application, but the glue is generally simple so I didn't reap much reward from using a framework that tries to hide it at all cost.
> I would be so happy if someone took away my power to choose and just forced me to use something that will "just work".
That's basically what Google did on the front end with Angular for people who had the exact same problem with React. Maybe we'll see something like this happen on Backend Javascript, I mean there is Sails, but it doesn't "just work" as well as Angular does on the frontend.
Well if you want this and you’ve ever considered using TypeScript for a JS project, you could try C# + ASP.NET Core. IMO it’s probably the best web stack out there. Opinionated, but allows you to stray from those opinions fairly easily. Secure by default. Great support and stability.
If you already have a database, I recommend storing the session in the database. I personally use connect-pg-simple when using Postgres. In other words, I wouldn't add another major dependency like Redis just for session storage.
At this point I have a repo that I keep somewhat up to date and then fork the repo for any new projects.
Every time I start a new express app I feel like I'm working at the 'wrong level of abstraction' to put it in Dan Abramov's words about the React ecosystem. I end up writing the same boilerplate over and over.
The only solution I have is a bare bones repo that I keep up to date myself. Not a great solution, but it's a solution.
The problem with Sails is that unlike Rails or Django, it's not very popular. This not only means a lot less documentation and help for you but also help for the maintainer (financial or code-wise). This means a slow rate of development, bug fixes, and updates.
I do agree with another poster in that if you want a opinionated just works stack, use Ruby or Python.
Why not? Figuring out DB access, session storage, i18n, etc. is not core to my business. Why not abstract that into a framework, and let engineers focus on solving their business needs?
Most of the things you mention are already solved in several useful and applicable to different situations methods in JS anyhow.
Mega-frameworks are, in my experience, one of the biggest sources of encouraging Bad Practices just because their the favorite idiom of the developers.
Rails is infamous for this.
Node (and JavaScript) already has a big enough problem encouraging good code practices that a Rails framework would do untold damage.
The closest it has now is Express (obviously not the same) and that’s already pretty bad for doing what I mention (vastly superseded by Koa and Hapi anyhow).
These things create a very dangerously myopic gravity.
The micro module route Node, JS , and npm uses is - I feel - much much more flexible and beneficial long term.
Bingo! I've seen things go wrong with Backbone. The lack of opinion is great if you have an experienced developer starting the project and defining the filesystem layout and overall architecture. We, unfortunately, did not have that for our marketing site. I actively avoid working on our marketing site for this very reason.
Software development should be about thinking, creating, solving problems, and yet a large percentage is grunt work that barely requires more than monkey intelligence.
In some ways we benefit. Sometimes its harder to replace us not just because of skill, but because of countless layers of byzantine and mundane information we’ve memorized about a system. It just not cost effective to stuff all of that crap into someone else’s brain if you don’t have to.
However whenever there is a way to reduce the cruft to productivity ratio, the power is staggering. That’s your “10x” developer right there, someone who gets to actually think a higher percentage of the time.
It is built on top of express to keep all of the good things, but it adds basic middleware and some more advanced functionality like websockets through socket.io and security through helmet by default.
It is designed to work perfectly with Promises and async/await, the new ES7 features that makes asynchronous code awesome. Also, I put a lot of work into making the documentation extensive and clear and I'll keep working on that for the future while preparing for the 1.1.
I like the design of the API, but I don't understand what you mean by "built on top of express?" I see you have express in a plugin module, but I'm not sure how it's being used.
Why socket.io instead of just plain websockets? Socket.io adds a lot of complexity in the client and the server. What commensurate benefit does it provide?
Socket.io adds rooms/namespaces as a feature, automatically reconnects with effort to not have all clients reconnect at the same time, covers compatibility edgecases, heartbeat, timeouts, custom events, multiple transport methods, server side implementations in multiple languages, supports non-ws fallback for reasons that might not be browser compatibility, and a good API.
For one, it has message_id ackbacks so that you can have a request/response rpc style system. A basic feature for any websocket-heavy application.
socket.send('login', [uname, pass], (err) => ...)
And there are socket-io clients for most languages that can interop with a socket-io server, once again supporting message_id callbacks.
Reconciling my own message_id + reconnect buffer system is not something I want to build on the client + server every time I build an application.
People talk shit about socket.io and then never recommend an alternative that implements this fundamental feature. Without message_ids, I get flashbacks to working with IMAP.
>Why socket.io instead of just plain websockets? Socket.io adds a lot of complexity in the client and the server. What commensurate benefit does it provide?
The real reasoning is browser support. Socket.io supports automatic fallback to AJAX long polling if the client doesn't support websockets. It also standardizes the interface across clients. It's essentially to websockets what jQuery is to the DOM.
I'd argue koajs was the first modern express alternative, although it's arguably more of a connect alternative. I've been happily using it since generators were available under a flag with node 0.11 and it has always been a joy. There's a huge list of middleware [0] on their wiki, and you just pull in whatever you need.
It means that your initial setup takes a bit longer, but it forces you to understand all the moving parts.
The session in production tutorial makes no mention of secret rotation, which I'd consider an important detail for a production environment. The docs don't mention the feature either, but if you look at the underlying express-session library you see that the secret can either be a string or an array of strings.
Although redis is an amazing tool, I disagree with recommending it as the first choice for a session store. Unless you really need it or plan on using it for more stuff, you'd probably be better off keeping sessions in whatever data store you're using for the rest of the application. That lets you avoid having to setup and maintain yet another tool.
No documentation at all for how the socket.io stuff is supposed to be configured. Chat tutorial assumes a single-node environment, which I'd consider unreasonable for anything outside of a hackathon. Wouldn't you at the very least want a process per core?
I didn't know about that middleware list, thanks for sharing.
About session rotation, it was my impression that it is a smaller problem compared to how it can be exploited if we're using cookies [1][2], could you share some more info about it please?
About Redis, I totally agree. You can add any store that you want with the plain `{ session: { store: ... } }` option. There is an issue though for some of them that need the original `session` passed in which I'll have to fix. So the main fix would really to improve the documentation to explain how to use the appropriate store.
Finally about socket.io, I also agree. I am not a large-scale system expert, so this is part of my limitations and that's why I recommend server.js for small-to-medium sized projects. Long-term I am working on improving on my knowledge here, but not the highest priority right now (compared to security for instance). Also, socket.io right now is not stable officially, so use with care. I'd love any help in here if you want to share some of your expertise.
I think around express 3 the guys decided to go with a modular approach and decouple the server from any of those heavy ( render ) libraries, as well as body-parser ( once it was core module in express itself ).
My really friendly feedback is that I think express carries so much experience that it will be really hard to replace it with something so monolithic. I use express for REST APIs as well as full web app, where I need no render engine and having this installed on my server is an overkill.
Best approach I would think of is having someone rewrite express from scratch and call it express 5, while keeping the same balance of core and extra functionality, as well as new async / promise goodies.
I totally agree, somewhere in the homepage I explain what this package is for. I decided not to bomb myself and not writing "where you should NOT use it" but I still hint to it. Server is good for small to medium projects where you don't want to be writing the same thing again and again. Think of it as express with a bunch of sane defaults and some extra functionality.
An extra render engine does not affect on anything except on file size (and a tiny bit of install time), which is cheap nowadays. If you are making a single big project server might not be the best option, since you might want to fine-tune many things. If you are making several projects per year with Node.js then it is perfect, since you don't want to waste time doing the same thing again and again.
As others commented, express 5 with async/await = Koa.js
BTW, bodyParser was just added back into the core of express.
Really? Hapi has its request lifecycle and built-in auth that makes auth much simpler than injecting authentication/authorization middlewares on every route.
Koa was created by the very same person as Express itself [https://github.com/tj] TJ Holowaychuk.
The initial idea is to do the next level express framework using back-then generator functions. What is missing ( for me ) is the smaller community and popularity. So it makes it tough to argue about the business impact between choosing koa over express.
Honestly, being a nodeJS freelancer for the last decade there was not a single project, where koa was in use. Mostly I've seen express or sailsjs [http://sailsjs.com] which is quite a full-blown MVC Rails style solution.
IMHO, and one of the reasons I decided to go with server, simplicity is a feature and the advantage of express over Koa. It is hard enough for new users to get started on Node.js, but to have to handle all of the new middleware concepts, PLUS to have to learn a lot more advanced features like generators is a no-go for my framework of choice.
- server: return something to reply to the browser. Can use `async` for more advanced features.
- express: reply.send() is fairly straightforward. next() for middleware is a new concept but conceptually in the right place (when you are digging a bit deeper in the middleware).
- Koa: yield, function with asterisk, etc.
So I think Koa is fine for devs who have been a while in JS dev, but the learning curve is too steep to get started. Same as what happens with React and the reason create-react-app has become so popular.
As long as you're on Node >=v7.9, you can also use `await` in Sails, Koa, or any other tool that supports promises. There's some work we had to do to make it feel nicer, and we had to update docs and stuff, but for the most part it was pretty painless-- all thanks to the JavaScript engine.
And `await` is so much easier and safer than callbacks in userland code! Really excited about Node.js in 2018, now that `await` is supported in the LTS release. Callback hell is finally dead.
Fastify is a similar project that aims to be faster than Express and adds a number of API improvements like baked-in schema validation for routes, logging, and async-by-default APIs.
Haven't used it in production but replaced Express with it in some side projects.
Since server.js is built on top of express it might even be possible to build a variant of it on Fastify instead and get those benefits as well.
This might be somewhat negative, but I don't quite see what problem the library is solving.
It looks like as if Express wanted to become more like Koa (obvious from the context object).
Frankly, it has missed out - koa's simplicity and elegance is unmatched.
The routing and middleware syntax is particularly clunky.
No problem, didn't sound negative. The main difference with Koa would be having a starting point with default middleware, security and websockets. I do agree with your points, Koa is probably the most elegant library out there, probably followed closely only by express.
But that is not why I created server. It is not about library elegance and simplicity, it's all about making it easy to use. Both of Express and Koa involve external middleware that the user has to manage manually, and there is a subset that is really common for most situations. Server.js is all about usage simplicity, including this subset by default and making things work by default.
Context was inspired initially by the way Promises work with a single return value, but then expanded further (and named after) Koa's context.
Isn't koa flexible enough for you to build a framework on top of it to accomplish your goals of elegance and simplicity? That way you could take advantage of the ecosystem that already exists, instead of trying to bootstrap your own.
Coming from Python, my understanding is that Express is sort of like flask (add your own everything), and something like sails.js is like Django (bundled, convention over configuration). I am wondering, why, unlike, Python and Rails, Django-type frameworks did not take off in Node.js? I would like to have a popular “Django.js”, so I am rooting for this project.
My opinion is that Node.js follows a lot more the philosophy of small reusable modules, which is a lot more flexible with the disadvantage of a lot more initial work. I still don't think server.js is Django.js-like, since it leaves most decisions up to the developers. The main difference is that Django, Rails, Sails, etc are opinionated about both the libraries AND the code structure, while I tried to keep server opinionated only about some basic functionality, but leaving the more advanced and code structure totally up to the devs using it.
But I do try to do batteries-included, because express-like workflow involves following a bunch of steps that are basically the same for most projects.
1. Node focuses on modularity. You pick your express clone, you pick your ORM, you pick your front end, you pick your logger, you pick your passportjs plugins. Because so many great things are just unopinionated modules, there's less need to make packages like express-mongoose, koa-mongoose, express-postgres, koa-postgre, express-pug, koa-pug, etc. (although I bet those exist anyways, they are simply gratuitous). If you want redis, you just pick your redis module from npm.
2. You could say npm is the framework.
3. You could say Adonisjs did kick off (pretty good if you wanted a laravel clone)
4. Mainstream webdev in Node is younger, so a clear rails clone is not evident but could already exist under your nose.
5. Maybe front end frameworks are the new "MVC frameworks". You can make both SPAs, MPAs, and hybrid SPA/MPAs with nuxtjs and nextjs. Do you need a full backend MVC framework to do what your frontend framework already does, but better?
Ah I didn't even think about it being a thing, thanks for the feedback. I did make server so I wouldn't have to worry about so many middleware pieces any longer and could work on my actual projects, but I see how this is fairly common wording.
No, I find it funny. Even hilarious, given that code with initial version
published mere six years ago is considered "not modern" somehow. I write in
a language that was published twenty years ago and hasn't changed very much
since then, and is unequalled to this day.
In Koa, handlers don't respond to the request. Instead, they update a representation of the response and return a promise. The response bubbles up to the top-level where the `res.send()` is done.
This way you have real middleware. You can post-process the response or do something else entirely after the downstream has run.
Are there docs or maybe a blog post that identifies the value-add on top of Express? While I see the API design example (which applies to me), we already do that with Express v4 (it's more code organization than anything else.)
I'm super-interested, but I'd like to know at a deeper level what this gets us over plain Express.
I think a lot of people still haven't seen https://feathersjs.com/. Has been around for a while and offers all of this and more AND is drop in replacement for Express.
Is there any Node.js web-framework/app-server like Express, that uses XML as backend, rather than JSON? Or, that can have this plugged in, at least? I'd like to use BaseX (XML document database) as a backend to something like Express.
Only 37MB? I expected more TBH, I guess I am too used to the dev side of it. 37MB is really cheap compared to the hours that you save by not doing everything manually, even more when you consider the quality of the underlying libraries from a security point of view.
Long-term I would like to reduce this and have some ideas in mind, but right now this is not an issue.
Does it support automatic documentation generation for the API (kinda like what swagger + express.js does)? That's the biggest missing feature in Express.js for me.
Passport can be quite easily used with any framework, but of course an existing library can make it even easier. Though to be honest Passport.js is trivial enough that I don't see a need for it usually..
Exactly, it was initially intended to be a core feature but after searching around (and from my experiences) I found out it is rarely done at the language-level.
It was not available! But it was abandoned and without any content, so after talking with npm for this specific scenario they decided to give it to me.
Asking the real questions! I wonder if deleted package names are made available again. It makes sense that they did, but it's also a security issue, if one package is removed an attacker could add a new package with the same name and with a malicious functionality.
Apparently the author registered the package name back in 2012. From `nam info server`, it shows that the initial development started on 9 Aug 2012, but stopped very soon on the same day. Then the author picked up the work again last year.
I recently started learning express/node and wasted weeks googling for blog posts for each dependency to figure out how to configure them. I guess this is somewhat valuable for end-to-end understanding but not for productivity.
There are a bunch dependencies I am now kind of decided on (not for technical reasons, but because they are clear popularity contest winners) and figured out how use:
But still a bunch that I haven't gotten to googling and still need to decide on what to choose: I would be so happy if someone took away my power to choose and just forced me to use something that will "just work".