On the other hand, there's more to testing than traditional unit testing in which a programmer enumerates a series of cases and verifies that what they think should happen (which may or may not even be correct) happens.
There's fuzz testing. There's QuickTest-like testing, which is tricky in this sort of situation, but powerful if you take the time. There's static analysis, which are basically a form of automated testing. In fact, static analysis can in fact test some of the things you claim can't be tested. Some of the other things are also more testable than you are saying, such as resource exhaustion, which can be both simulated via injection or via simply exhausting resources in the test case.
Further, even the relatively-weak guarantees that traditional unit testing can provide are a foundation for further analysis. What's the point of analyzing a bit of code for whether or not it uses entropy correctly if you miss the fact that the code is straight-up buggy? Given the demonstrated difficulty of writing code even before it's security code without solid testing, this is hardly the time to be claiming that testing isn't necessary.
This strikes me as another iteration of the Real Man argument ("Real Men write in assembler" being one of the originals). The truth is we need all the help we can get because we humans aren't very good at programming in the end... and of all the places I'm not inclined to accept the Real Man argument, it's in security code. This is the sort of code that ought to be getting covered by two or three separate static analysis tools, and getting reviewed by skilled developers, and getting fuzz tested, and getting basic unit tests.
No Real Men arguments here. I did not claim testing isn't necessary. I just wanted to challenge the blatantly religious attitude of pretending or believing that a particular form of testing (one with clearly written tests the poster above me would actually read) is all there is and that you cannot review code otherwise. You definitely can, and must.
As far as I am concerned, all techniques that help improve code are welcome and even necessary. This includes testing, manual review, static analysis, etc.
There's fuzz testing. There's QuickTest-like testing, which is tricky in this sort of situation, but powerful if you take the time. There's static analysis, which are basically a form of automated testing. In fact, static analysis can in fact test some of the things you claim can't be tested. Some of the other things are also more testable than you are saying, such as resource exhaustion, which can be both simulated via injection or via simply exhausting resources in the test case.
Further, even the relatively-weak guarantees that traditional unit testing can provide are a foundation for further analysis. What's the point of analyzing a bit of code for whether or not it uses entropy correctly if you miss the fact that the code is straight-up buggy? Given the demonstrated difficulty of writing code even before it's security code without solid testing, this is hardly the time to be claiming that testing isn't necessary.
This strikes me as another iteration of the Real Man argument ("Real Men write in assembler" being one of the originals). The truth is we need all the help we can get because we humans aren't very good at programming in the end... and of all the places I'm not inclined to accept the Real Man argument, it's in security code. This is the sort of code that ought to be getting covered by two or three separate static analysis tools, and getting reviewed by skilled developers, and getting fuzz tested, and getting basic unit tests.