I don't understand why PBT is above mutation testing. It seems like it's more of a popularity contest kind of thing, and not a matter of engineering tradeoffs or how useful it is.
In my experience, adding PBT once you've got a codebase amenable to unit testing in general is a relatively easy step: you can add property tests to your existing unit tests without changing much about your setup, and assuming your tests in general are quick, PBT won't add much overall time to your testing process.
But adding mutation testing tends to be harder: it's not just an extra thing you can add in without changing any existing code, it's testing whether the tests are useful in the first place. Which means when you introduce it, you'll probably need to spend some time fixing everything that's currently wrong. This makes it a next step in the testing process beyond just adding a new technique to your existing repertoire.
That said, I've used PBT for a while and not had as much success with mutation testing, so maybe this is a personal bias.
Maybe we're talking about different things then, but in my experience, MT is a reasonably involved procedure that requires configuring the MT harness to understand where the code lives, how to run the test suite, how to interpret the test suite, etc, then running the mutation tests for upwards of an hour as it repeatedly makes various changes and runs the tests. If you want to do something more complicated like only test a given region of the codebase, then the configuration becomes even more involved.
This is all significantly more involved than my experience with PBT, which tends to be something that can be added without much ceremony to an existing test suite when it makes sense.
To be clear, I love the idea behind mutation testing and I have given it a go a few times with limited success, but I think your comment is overselling its simplicity.
That said, I'd love your advice: how do you introduce mutation testing to a large codebase that currently has an extensive set of tests but hasn't used mutation tests yet? And how do you maintain the MT side of things? It seems far too slow to regularly run in CI: do you just run the MT tool every now and then to make sure that the tests are still covering all the mutations? Or do you have a more structured approach?
> MT is a reasonably involved procedure that requires configuring the MT harness to understand where the code lives, how to run the test suite, how to interpret the test suite
mutmut autodetects this with a majority of setups, but yea, if you need to configure all that a lot then it can be annoying.
> then running the mutation tests for upwards of an hour as it repeatedly makes various changes and runs the tests
Yea, MT is slow heh. But it can be simple to start with. I generally recommend to do it for libraries mostly since they tend to have small and fast test suites which makes it much more fun. Or extract the code into a throw away project, do MT there and then move it back. It's a bit crap, but it works.
> To be clear, I love the idea behind mutation testing and I have given it a go a few times with limited success, but I think your comment is overselling its simplicity.
I had to write my own mutation tester because I couldn't get the existing ones to work, so I do feel your pain there :P
> That said, I'd love your advice: how do you introduce mutation testing to a large codebase that currently has an extensive set of tests but hasn't used mutation tests yet?
I party answered above, but I would only test extremely limited parts that are critical. And I would make sure to not run the entire test suite.
> And how do you maintain the MT side of things? It seems far too slow to regularly run in CI: do you just run the MT tool every now and then to make sure that the tests are still covering all the mutations? Or do you have a more structured approach?
This is exactly how I use it yes. People ask for CI support for mutmut and I've accepted PRs for it, but I just assume they will get it working and then later throw it all away as it's useless. I try to convince people it's the wrong approach but I have trouble getting them to listen.
If MT was WAY faster then maybe you could use it to validate it regularly, but for mutmut at least it's too slow. I have an experimental branch of mutmut that is much faster but I just don't have the time/interest to make that a reality right now. I don't particularly need MT in my current job...
