I've seen the addition of unit testing is a big cause of complexity. Previously simple classes now have to be more abstracted in order to unit test. Add mocks, testing classes & test frameworks. Some unit tests are handy, but I dont think it justifies the additional complexity. For the apps I write I'd like to see more emphasis on automated integration testing and fewer unit tests - so we can write simple classes again.
The "threat" of having to add unit tests should force developers to write their classes and components in a way that is easy to reason about. In particular:
* put as much functionality into pure functions
* depend less on statically-linked globals
* import all significant collaborators across seams that can be mocked.
* keep state in a small atom, rather than strewn about
If you write code like that, you get many of the benefits of unit tests, whether or not they are actually written.
Perhaps it's a good idea to write a test harness (e.g. larger integration tests) for old so that you have a reasonable chance of catching it if it becomes broken, and focus on writing new code in a testable fashion.
In my experience, it's usually not the existence of unit tests themselves that's causing an issue, but that most of them are badly written. One telltale sign is when writing the unit test becomes overly painful (like too much code setting up mocks), it usually means that your class is not simple enough or has too many dependencies.
Proper unit testing also complements integration testing in that corner cases can be handled at the unit test level, therefore reducing the amount of integration test code which arguably is much more brittle, runs slower and more complicated to write.
Many unit tests are just written to test code, which is at best irrelevant. At worst your codebase is 2-3x bigger and more abstract than it needs to, where useless tests keep code alive and useless code keeps tests alive.
Test functionality, as close to the promises given to outside consumers as is feasible. Be it API or UI for other people/projects/services.
This is the stuff that needs to work (and thus often need to be stable). No-one cares whether a function deep down inside the code, used a part of the implementation of promised functionality works. Delete it if you can.
Only case where I'd support "unit tests" as typically practiced (small units, isolated functions/classes) is around core competence (defined as narrowly as possible). But then I'd argue that this functionality should be put into a library anyways, which is used by products codebases. And then the tests are tests for the functionality promised to the products.
I'm not arguing against writing integration tests, they are as important if not more important, as you've said. Maybe I've only seen badly written ones, but my issue was against integration tests that check for example if this ever so important, but hidden, flag is being set properly after an API call when that can be checked at the service level. Someone eventually decides that flag is unneeded, and a whole host of tests fail and someone has to dig several levels deep to figure it out.
I guess I shouldn't have used the word 'brittle', but this is what I was thinking of.
And of course, I think unit testing anything and everything is absurd and not a good use of developer time.
I don't think you can avoid meta-debugging. That is, debugging your asserts or tests that you hoped would detect bugs instead of being the bug. Sometimes because more realistic tests unveil a bug, sometimes (as in your example) because underlying code functionality has changed. This is unavoidable but also often enlightening. To my mind, it's even okay if most of your bugs are meta - because these are usually very fast fixes, and it probably means you have a lot of checks. But by the same token, I would agree with you that all such tests have to be well-written, not mailed in, for just the reasons you give. It's too easy to assume that writing tests is somehow a fairly trivial task. Until you end up debugging the test.
I've seen this too. Unit testing was mandated from on high and it's something developers never learned to do properly. My telltale sign is more than one logical* assert. A test should usually be only a few lines of code, a dozen lines should be all that's needed for 99% of tests.
*Logical meaning only test for one thing, not a single assert statement. So testing for null and testing if a value is set is fine, but testing if 10 values are set correctly is not.
If integration tests are brittle, then the integration is likely to be brittle. In my opinion this is something to fix, not workaround by testing lower down.
