On: Write Fuzzable Code

[dev_dull]

> In other words, it detects only crashes. We can do much better than this. Assertions and their compiler-inserted friends — sanitizer checks — are another excellent kind of oracle.

I'm trying to understand this one. Isn't a fuzzer useful for catching unpredicted exceptions? It seems a non-zero or premature exit of a program is one of the best scenarios.

[MaulingMonkey → dev_dull]

Fuzzing can feed all kinds of bad input into a program where the correct result is to print an error message and/or return an error code. E.g. if I feed a malformed PNG into a texture conversion tool, it probably should print an error and exit nonzero, so we can't automatically mark those as test failures.

On the other hand, if my same texture conversion tool hits a debug assertion about a corrupted heap, that should be a test failure. On the one hand, it was kinda predicted - someone wrote a debug assert to catch it after all - but on the other hand, it certainly wasn't supposed to happen, no matter what kind of bad input we were fed. It indicates we're corrupting memory, quite possibly in a way that won't always do the right thing of exiting non-zero, especially if the debug assertions are disabled in release builds.

Even when your encounter assertions in more predictable code, it's often the case that a non-zero premature exit actually wasn't the desired response. Maybe you wanted to gracefully exit with a better error message. Maybe you wanted to log an error and continue execution when your HTTP server recieves bad client requests, instead of creating a denial-of-service CVE. Maybe you had a bug, and your code is mishandling entirely valid input.

If you're using assertions to indicate "I believe this should never happen" instead of a general error reporting tool, it can be very nice to make them test failures when fuzzing.

[Sohcahtoa82 → MaulingMonkey]

> If you're using assertions to indicate "I believe this should never happen" instead of a general error reporting tool, it can be very nice to make them test failures when fuzzing.

I used to work for a company that used asserts as general error checking and it was awful. It was for embedded firmware, and I always ran tests (including security tests) with debug firmware. Non-debug firmware had asserts disabled. On debug firmware, a failed assert would cause a complete crash with the display showing the source code file and line number of the failed assert.

Some developers were using asserts on simple user input. For example, this firmware had a web server with a REST API. One of the APIs would expect a number as a parameter, and the person doing development added an assert() statement to make sure the API received a number and not letters or other characters.

It made security testing a nightmare and most of the assertion failures I ran into were completely pointless.

[redis_mlc → Sohcahtoa82]

> It made security testing a nightmare and most of the assertion failures I ran into were completely pointless.

Just to clarify for me, what you're saying is that:

1) instead of asserts for invalid input validation, they should have just returned the appropriate error code

2) so that the fuzzing harness could continue with the run rather than just keep dying?

> It was for embedded firmware, and I always ran tests (including security tests)

You had me going there for a minute. lol. :)

TIA.

[Sohcahtoa82 → redis_mlc]

Correct.

Basically, I'd see code that looked like this:

    assert(isNumeric(userInput));
    if (!isNumeric(userInput)) {
        return ERR_INVALID_INPUT;
    }