So basically their biggest problem was that node.js machine-gun hammers the filesystem with stat calls when looking for modules. This was made worse by running on a shared folder.
Vagrant has a mode called rsync-auto which will synchronize directories into a local vm. Or you could just always have the code local to the node process. It makes more sense to have your editor work with remote files.
Next biggest problem was relational database. I think that tests requiring a SQL database are usually a bad sign, and the statement "we'd like to use transactions but our orm doesn't support them" makes me sad. To paraphrase Ben Franklin, "those who would give up transactions for a framework deserve neither". Here's some ideas for speeding up your tests, in order from best to worst.
- Make most business code rely on a service or interface. Swap that our during tests for a trivial in-memory implementation.
- Use sqlite :memory: for tests
- TRUNCATE TABLE
- Set your database to running with scissors mode (disable fsync)
Disagree about DB tests. If you want to test queries, then it's bad idea to mock DB. If you don't want to test queries, don't call your tests coverage full.
The best way to do db testing is with an empty db and in a specialized dao test. Try to make these dao tests non-overlapping so that the dao test is the only test that hits that part of the schema. Also, if your dao tests are the only thing that hit I/O you can safely do all the other non-integration tests concurrently.
Service layer tests should have mocks of the dao and web service calls. You should do most of your testing here and with unit tests for tricky algorithms.
Integration tests should hit the full stack, but those should run on a continuous integration server asynchronously and also take place in their own empty database.
The key to dao and integration testing is the empty database. Don't let it fill up with test crud between tests. PostgreSQL is pretty fast when your test db only has a couple of rows.
Not production DB, obviously :)
Generate data before tests, collect list of created records during tests, remove collected items after all tests - if DB is not empty after it, you can see something unexpected has happened. If it's not possible to collect records (because of too opaque ORM), just flush DB.
This is by far my favorite way to test; not something I invented but something I picked up start with an EMPTY DATABASE (with tables setup), then every test: begins a transaction,asserts the db is empty, sets up the data (via factories and DAOs), performs its queries (via DAOs) rolls back the transaction for the next test.
It's a fast approach that gives some test coverage, but some of the most important things to test are around the interaction of multiple transactions, which you can't do if you're using a single big transaction to run your test.
Same. Not only to ensure I wrote queries or scopes correctly, but also because I like to add extra validation constraints to be sure my application code can't sneak junk into the database.
How can I really ensure my CHECK constraints / DOMAINs, unique indexes, RAISE EXCEPTION in insert/update triggers, etc. are correctly implemented if I'm not testing against a real Postgres database?
I would recommend writing tests specifically for testing the PostgreSql changes, ideally you can write them purely in PostgreSql, maintaining a nice separation of your code and your SQL. Also, the only thing that should really need proper testing are stored procedures, your check constraints can't possible be so complicated that they warrant tests.
The unit tests that test out your business logic should use mock data - it's just much more reliable, reproducible, and faster.
That doesn't mean your data/db access needs to be in the same project, or tests as the rest of the application... if you haven't touched the db bits, don't run the db tests.
It's about creating appropriately sized discrete modules.
One of things unit-tests gives is fearless refactoring. When you can easily change behavior in one module and tests shows other modules are not affected (or affected). When you test only one module/layer you don't see the whole picture and such tests are much less representative and useful.
Bollocks.
If your database schema changes right at the heart of the data model, a load of unit tests are not going to save you from some serious hassle.
I run a script that mounts a RAM drive and copies my postgresql database into the drive. I use tablespace to force Postgresql to look at the RAM drive for the database data. That way I am testing my queries against the same database engine but get the benefit of avoiding disk IO.
Where I work, tests have two modes: quick and long. Quick mode gets in memory treatment. Long mode doesn't. Anything that goes live will undergo a long mode test beforehand.
We would run those tests a couple times a week, and the memory tests idealy on every successful build (we chained the builds so we never bothered to run the DB tests if the unit tests had already failed).
Strictly speaking it's also possible to get subtle differences in behavior across OSes. Especially if you have a race condition you don't know about yet. I always tend to encourage more diversity (carefully spent).
May not work. Given the use of ORMs these days, this is a smaller issue. Though it still requires attention. Having the tests run quicker allows for better use of the programmers time
> Given the use of ORMs these days, this is a smaller issue
The ORM can't abstract away all differences between databases. For example, SQLite doesn't have Numeric type, so all of the Numeric(10, 2) fields on (e.g.) my SQLAlchemy (a Python ORM) models will trigger warnings when saving to a SQLite database.
Hence the sentence that follows after the one quoted. It does require attention. Just a bit less. In a sea of issues, reducing the importance of any might probably ease development in some way.
SQLite might be the easiest to configure, but some other RDBMSs have an option to run in memory too (e.g. MySQL's MEMORY storage engine). Also, as a last resort, you can always configure the DB to write to an in-memory file system.
This strategy is, in essence, mocking the FS layer in DB tests.
Well, those sorts of things cause the tests to immediately fail, and you realize the error of your ways. It's more deadly when you tests pass, but behaviour on production is different.
Hi Shane, thanks for the tip about sails-memory. We've seen a lot of inconsistencies in the way Waterline behaves, so we're hesitant to run it against a different backend than the one we use in production. We'd also lose all of the uniqueness constraints that we use in Postgres.
In general we're trying to write and run more tests that only create and manipulate objects in memory, especially for testing business logic. Unfortunately we still have to have some level of integration with the database to test things at a higher system level.
Re: transactions, most of our database code uses the ORM, so we thought other test speedups would give a bigger benefit. We recently added an interface for getting/querying with a `pg` connection directly and we're using that to implement transactions in a few critical places. I'm planning to release the code soon - hopefully in a blog post in a few weeks.
There should be some optimizer for npm modules. The current system is silly. Popcorn Time installs around 5000 files as part of its node_modules. How is that useful?
There should be some way to combine things into a single module/file or otherwise glob it all together, while removing pointless test, docs, and Adventure Time animated gifs from your deps folder.
Apparently imminent npm v3 "flattens" node_modules/, which should help with this problem. At the same time I worry that version compatibility might become undecidable...
Although I'm sure there are module maintainers shipping AT gifs, we can't really blame npm for that.
By its design _browserify_ can be very powerfull on the node space (there are dedicated options in this way).
We use it a lot (with grunt & co) to pack & deploy production code (with excellent result)
Should add, a problem with `rsync-auto` is some files in node_modules (bcrypt, among others) are compiled at runtime, and produce different outputs for Mac/Linux - you can't just copy them from Mac to Linux and expect it to work. Probably some things we could do to hack around this but it didn't seem worth it.
Running against a local mysql instance running on a ramdisk and emptying the DB tables between tests yields "fast enough" performance while avoiding test pollution: ~300 full-stack scenarios in ~5sec start to finish.
May want to check out the patch - after a syscall in a directory failed because the directory didn't exist, node would issue 4-5 additional stat() calls for the other filetypes (.js, .index.json, .json, .coffee, etc) even though the directory didn't exist. Solution was to abort after the first failed stat(). https://github.com/nodejs/io.js/pull/1920
In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch. Based on that, I'm not sure advising for more stubbing is a good advise. Maybe I'm wrong but that was my experience building UI apps.
The distribution of when bugs are caught, and what bugs are caught, by different kinds of tests is very different. Units tests catch bugs while they are being written and when the units they test are being refactored. The bugs they catch are trivial, silly, 'would have been caught anyway'. Outside of those activities, they rarely catch bugs. On the other hand, integration tests more consistently catch bugs and catch bugs you didn't foresee, have to investigate before understanding.
You don't remember the last time a unit test caught a bug, because on average it is much longer ago and if it was a test for a unit you wrote this morning, you usually blame it on 'work in progress' and think you would have caught that trivial bug anyway.
I think the nature of units tests make it easy to underappreciate them.
This is too true. A team I was on started seeing way more value in integration tests and ended up writing way more integration tests than unit tests.
What ended up happening is that the code quality ended up worse because they stopped running tests very often because there were too many integration tests and they took too long to run. Sad part is, a lot of those tests would have made a lot more sense as unit tests.
An added benefit of using unit tests over integration tests is that unit tests shows you what went wrong very clearly. If an integration test fails I have literally no idea where to start hunting for the bug. It could be located at any level in the stack.
I highly recommend this brilliant talk by J.B Rainsberger on how integration tests are a scam: https://vimeo.com/80533536
If you don't have that much time to write tests I'd recommend integration tests. Then at least you have some tests. Treat integration tests as smoke tests. If there is smoke then you can start looking for the fire. Integration tests are always better then no tests at all. At least they tell you there is a fire even if it is vague in telling you where it is.
As someone who always writes unit tests before any production code, I don't know how I would have the time to not write tests. It drives the design and reduces time debugging by a ridiculous amount. I work with information systems though, might be different for other fields.
Since I've switched to a language with a richer type system (Scala) I've found a lot of what I used to do with unit tests can go into the type level instead (see e.g. http://spin.atomicobject.com/2014/12/09/typed-language-tdd-p... - but in Scala the techniques are a lot less verbose to apply).
I am most familiar with Javascript & Python, but I think you're onto something having fiddled with Java for 6 months or so. Some logic obviously can't be replaced with only a rich type system, but a lot can.
It depends a bit on the problem you are trying to solve and the structure of your code. I have projects where only end-to-end tests find bugs and others where unit tests are helpful. What is never helpful is imposing some sort of semi-religious "thou shalt do x" in every case.
That's why you don't stop with unit tests. You make sure to automate testing of ever layer of the application. It does require a bit more effort, but it's it's a savior when the UI breaks, and it's not something obvious yet breaks the user experience.
I prefer to write end-to-end tests first, then fill in integration and unit-tests on a as-needed basis. If I am finding it complicated to write, or if experience from production says it is a problematic area.
After all, the important thing is whether your application/service/library does what it promises to its users/consumers. Whether some function/class deep inside works or not is mostly irrelevant (can help in fixing the issue faster though).
Unit-testing-first has a tendency to make the tests very fine-grained, often need changes when you refactor internals (how do you know test did not break?), and needlessly keeping rigid internal details that one should have the freedom to change. Have seen plenty of code being kept around because it had lots of tests - when it was not actually used for anything at all!
Note: Above approach is coupled with a pretty strict adherence to keeping projects small and independent, preferably <10kLOC.
Yeah...for me, unit tests are the gift that you give to the developer who will be maintaining and improving the code 6 months from the time you write it. Beyond documentation, it gives him/her the ability to refactor the code without fear of it breaking horribly and tells that developer which behavior of the application is intentional and which was accidental. And before you say, "Screw 'em...I've got a code freeze to meet," consider that the person maintaining the code 6 months from now is probably you.
How would you have good unit tests if you have bad application code? It's writing the tests that forces you to improve the application code, because otherwise you won't find it straightforward to write the tests.
Admittedly not in node, but I've found that writing end-to-end tests but substituting different 'units' for in-memory or not depending on the test marries the best of both worlds. In-memory tests tend to focus on behaviour.
In the rush to shoe-horn tests into a taxonomy, or 'integration' vs 'acceptances' vs 'unit' we often loose sight of the actual value proposition. Assisting design and building confidence to make changes.
"Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code."
Unit testing and TDD are quite separate things. Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation.
> Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation.
Nice snark there. Feel free to keep it to yourself.
To add actual value to the conversation (as opposed to your contribution), I can very much recommend the book "Working Effectively with Legacy Code"[1] for how to handle unit-testing in the scenario of existing "legacy" code-bases.
It's full of useful tips and methods to get testing in place "anywhere" and has a pragmatical (as opposed religious) approach to getting it done.
To spark some interest: The book defines "legacy code" as any code not covered by unit-tests.
It may be seem dated (from 2004 and all), but it's been the most useful book I've read on unit-testing by far.
You came along and dismissed the parents views without any real thought.
If you write the tests afterwards you can better gauge how well they catch bugs. It sounds like the parent has tried this and has found them pretty useless.
I've personally found that unit tests are not worthwhile for many components, and yet are critical for others.
I've been meaning to buy that book for a long time, thanks for the refresher :)
I've found that there are way more books for "greenfield" software development and not so many books for what 60% of people actually do (maintaining other people's projects, legacy or otherwise).
my experience is like yours that unit test developed in development are rarely useful.
however, having a framework in place helps a lot when working at user submitted issues: those are the issue one should strive to replicate trough unit test. In a six month of production, you get a fairly robust suites of test on all corner cases undiscovered in development.
Your sentiment is strange to me - I've only ever seen unit tests as absolutely worthless. The reason integration tests are flaky is exactly the same reason they are valuable.
Someone who checks in broken code will also take flaky tests as an excuse not to look at the build results, or to convince themselves that the failure is caused by flakiness and not their changes.
If I had a dollar for every time I asked someone why they weren't fixing their shit, and they answered "Oh, I thought the build was failing randomly again." I'd have a lot of dollars.
The fact that your tests are flaky at all means that the integration tests have revealed a bug that you need to fix. The tests have done their job. The fact that people aren't looking at the results speaks to a larger cultural issue.
Not if it's a timing error in the test harness. That's usually what people mean when they say flaky tests. Selenium, especially three or four years ago, would simply fail occasionally no matter how good your tests are.
Yes, there are bugs that look like bad luck, but are really flaws in your logic. I have fixed a lot of bugs that look Random, but eventually you're left with, for instance, selenium just refusing to click a button.
While bringing down test runtime 1000x is certainly a nice feat, I don't think this is setting a right precedent. I, for one, would be very afraid. Tests are not supposed to be written with speed in mind. Metrics seem to dominate the world of tests, where the spirit is supposed to be catching real world bugs.
For example, the following can be done without destroying the spirit of testing:
* Stub the sign in (big savings)
* Keep a tab on fixtures (big savings)
* Parallellize (big savings)
* Buy a better machine (big savings)
* Truncate with transactions/container-snapshots/etc (medium savings)
* Tune database for speed than safety (medium savings)
* Tune GC (medium savings), with enough RAM (16GB) disable it entirely (big savings)
* Disable logging and instrumentation, yes including timing measurements, unless intentionally needed (small to medium savings)
* Mount entire codebase, database and resources in tmpfs (small savings)
The below, while they might improve performance, go against the spirit:
I'm planning to write a tool, which given a Class, will pump out lines and lines of tautological tests, line for line (a.a should == a.a), method for method (a.should receive save, a.save), branch for branch (with some metaprogramming). Coverage will report a cool 100%, tests will be blazing fast, and that should calm the lead devs down.
> Tests are not supposed to be written with speed in mind.
There's always extremes.
If people write tests with no regard for speed (as I've seen) you can have "unit-tests" run for 45 minutes before completing.
That might be OK to guard against regressions on a nightly-build, but it will in no way be a test-suite I'm willing to running prior to every commit I ship.
If the test-suite runs in less than a minute, I'm much more willing to do so before submitting code, and in that way avoiding regressions which needs to be fixed later on.
If the tests runs in less than 10 seconds, I'm probably going to run them for every major change I do, even if I'm not planning on submitting anything yet, just for peace of mind.
There's IDEs where tests can be run automatically in the background as you make changes and let you know the second you broke something. For that to work efficiently, your tests has to be sufficiently fast.
So maybe the aim shouldn't be that tests must be fast above everything else. But if the test-suite can be run quickly and casually, it's value will increase tremendously.
> Tests are not supposed to be written with speed in mind
This is why the tests took 100 seconds in the first place. While I agree that speed isn't the number one priority for everything, Reducing the length of time your entire test suite takes to run to be doable on every single run of your framework is absolutely worthwhile.
This is somewhat more reasonable. If your core business logic encompasses flushing data to lying disk controllers and being robust in the face of kill -9's, the root filesystem filling up, not corrupting the database on segfault, and not falling over on multi-terabyte datasets then your tests might take a long time.
I just about choked on my coffee. If you're serious, is the issue just the size of the test suite, or are the tests themselves just woefully inefficient? By the time you're done, you've forgotten whatever it was that you modified.
I expect they have subsets of the suites working as "fast tests". The full sqlite test suite is 3 orders of magnitude bigger than sqlite itself and takes hours on beefy machines, but they're not run "online" by developers. Developers generally run the "veryquick" suite which takes a few minutes (and is sufficient to catch the vast majority of issues)
Database test suites tend to be on the ridiculously thorough side, for good reason. For example, the SQLite test suite ( https://www.sqlite.org/testing.html ) all but beats the code with a sledgehammer trying to break out bugs. Some software can't be upgraded once deployed, so it only makes sense to ensure that it is well-tested.
This post is full of great general advice that's not specific to Node. In Ruby-land, we've been talking about the same things for the past few years, complete with 'omg how many stat calls are we making again?'
I'm sure others were doing the same before Ruby too. The wheel keeps turning...
I worked on a perl application server for a big ecommerce company. We'd stat so many files. Pulling in the meta-object library ('use Moose') was guaranteed to make your program take a long time to start.
The core issue is that Rubygems adds the `lib` directory of every gem to your path, so you end up calling stat a zillion times.
I am admittedly a bit out of the Ruby loop these days, so it's possible that this has been improved. The fastest way of course is to not load a zillion gems...
A decade ago I brought the test time from 86min down to a few seconds.
An intern had thought that synchronizing things with "sleep one tenth of a second then check that flag again" at the heart of the product and reading and splitting by line a file where some lines might be 86Mo long were good ideas.
Wait, Shyp uses Sails.js for their framework? Huh, they might be the most well-known startup to use it. Sails development has really slowed down lately, especially since one of their core creators/commiters got into YC with a different project; I wonder how Shyp feels about that...
"Avoiding Sails isn’t possible in every situation, but we’ve found it’s a useful design principle - it helps us separate our business logic and our data storage layer, and will make it easier for us to move off Sails in the future."
Yep - we've run into a lot of really unexpected behavior with it and with the ORM (Waterline), so we try to restrict ourselves to "the minimum possible framework subset that gets the code to run".
For example, we disable `dynamicFinders`, don't use policies, don't use blueprints, don't use Waterline's associate/populate/destroy, don't use the "turn any controller function into a named route" setting, don't use globals, disabled all of the Grunt/websockets junk, and overwrote all of the error response handlers.
The result is that day to day we're writing something that resembles an Express app with its own `routes` file. We're looking to move off of Waterline/Sails entirely, it's slow going though since we're a small team and it's tough to prioritize that, as it "works" now.
Every company has parts of their tech stack that don't do a great job - we're somewhat fortunate this one is easy to mitigate.
Sounds like you guys pulled out everything except for Express. I too found the more I used Sails the less useful it became, as though it was missing a solution for anything lying outside of the very basics. Compare Sails to say, Laravel, and you begin to see how much work needs to be put in to bring it up to date.
That happens with every Framework.
That's why we migrated from Django to Playframework.
Django is great, except for the thing we do / the expertise I / we have. Play framework is way more "raw" than the most of the framework's I've seen. Also I use Slick but only the SQL part of it and the Codegen. It's so great to write RAW Sql while persisting objects easily and type safe without a big layer in between.
You are correct Playframework still does too much for the most people, however we are using a lot of things from the framework, basically json, forms, templates, the new DI layer, the ws library (which still is not usable on spray :(:() and the plugin layer.
Mostly I think we could easily switch to Spray, however currently I really liked Play in his current version.
Being able to move "off" of Sails, interestingly, is a design principle of Sails itself. Sails is built on common node.js modules (express, socket.io, etc) and uses standard express middleware/APIs so that pieces of a Sails app can be ineroperable with other frameworks and tools. Waterline (the ORM layer) is designed to work with any database so that you aren't locked to a single vendor. It'd be hypocritical if we wanted to liberate you from other tools, but try to lock you into using Sails, wouldn't it? We feel that that's a feature, not a bug.
There are much larger companies (Fortune 100) that use Sails.js. This isn't a knock against Shyp at all, and I'm glad they're using Sails, I'd just like to clear up whatever the source of your misconception is. Sails, Waterline, and the many related modules are all in active development.
(Disclaimer: I work for Balderdash, the company that created Sails)
Well, to be fair, one of the patterns I've seen with Rails projects is to move as much out of Rails proper as possible, and treat it just as your web/(maybe)database interface.
Never fear, our new project compiles Sails.js apps-- the core of what we're trying to solve is exactly that (the transition between using a more heavy-handed framework, and being able to customize/optimize/understand minutia as needed). More on the tech here: https://www.youtube.com/watch?v=nZKG7hLhbRs
Also, in case it's helpful, figured I'd mention that we just finished rearchitecting sailsjs.org and the doc compilation process to improve the permanence of links and make the experience of searching for stuff from the docs on Google better (latest updates here: https://github.com/balderdashy/www.sailsjs.org/commits/maste...) Kevin, I know that was something you (rightfully) have a lot of energy about.
I don't know what it is with this site, but it made Chrome (Android) crash four times already. Which means that, thanks to that wonderful browser, my two billion research tabs are gone since the first crash.
Anyway, I can't imagine how running unit tests could possibly reach 100 seconds. Usually, that means that the tests are coupled to the environment (database, filesystem, ...). So without being able to read the article, that would be my guess. There are tons of good (and old) reasons why you should decouple unit tests from the environment as much as possible. Execution time is only one of them.
I don't think it's your fault, though. Chrome loves crashing and throwing away all of my research tabs. That wasn't the first time. It's only the first time I could reproduce it with a specific website. Still, as I said, I don't think it's your fault.
I'll just finish reading the article when I'm back at my PC. Test optimization is a great thing.
At my startup (35 devs on the main product), we'd spin off a dozen AWS instances so the tests would run in 20 minutes in parallel. And only tested commits could be merged into master. We didn't feel it as a bad situation.
It seems like renting computers by the minute is pretty cheap these days. I'm surprised more people don't just have one machine per test, run them all in parallel. Testing seems to be embarrassingly parallel in that usually your tests don't depend on other tests at all.
> We run tests 400-500 times a day; a ten second improvement in every test run translates to an hour of saved developer time, every day.
Is it just me, or does that seem like running a lot of tests per day. I mean, how many lines of code do they write per day? 1000? Do they run the test suite ever time they write two lines of code?
That is probably on a CI server. During development, I'd probably be running way more than that, at every production or test line written, having a task that watches for file changes and re-runs all tests. Then, having a fast test suite is key.
While speeding up the process is always good, tools like lucifer or spork are only masking the problem: loading the whole environment for unit-level tests. Sails, like Rails, promote tight coupling between all parts of the system, which harms test speed, thus harming another development tool: TDD.
The more decoupled and isolated you write your business logic, the easier and faster it is to test drive it. Then you simply plug an HTTP interface (with routes and whatnot) that calls into your core.
If it so happens that people who are having problems with the running time of tests have something like on the order of as much test code as they have "actual" code, I'm not surprised that they have to optimize the test suite. Apparently that was a trend some time ago (I don't know if it still is).
Vagrant has a mode called rsync-auto which will synchronize directories into a local vm. Or you could just always have the code local to the node process. It makes more sense to have your editor work with remote files.
Next biggest problem was relational database. I think that tests requiring a SQL database are usually a bad sign, and the statement "we'd like to use transactions but our orm doesn't support them" makes me sad. To paraphrase Ben Franklin, "those who would give up transactions for a framework deserve neither". Here's some ideas for speeding up your tests, in order from best to worst.
[edit] I just took a look at the ORM used, waterline https://github.com/balderdashy/waterline . Why wasn't step one moving to sails-memory for running tests?