The winner of this contest built his own Linux-compatible system and wrote the kernel "designed to withstand zero-day attacks in userspace" from scratch(!) [1]. Linux binaries work on it without recompilation(!!).
I have, for a number of years now, found the 2008 Winning entry by John Meacham to be an important reminder. The trick isn't any preprocessor abuse, pointer-math problem, or any of the usual places in C where the complexity can easily obscure what is really going on. The program uses several idiomatic construct like
for(c = buf;*c;c++) { ... }
to walk the PPM data. It does exactly what it is supposed to do. (output black (zeros) for pixes in a given region.
To find the problem, you have to understand the entire problem and the written algorithm, as this isn't a "bug" - it doesn't implement the spec properly. It walks char by char - which looks fine at first - but that means it zeros "1 12 123" into "0 00 000", leaking the contrast of the original data.
I find that this is an important lesson to remember: no tool can catch you using the wrong algorithm, especially when the code "mostly" follows the design requirements that can never cover every possible corner case. Tools can catch interesting problems, but the inherent complexity in software says that you can never really be sure what a program does until you've read an understood the whole program... which isn't possible in the general case (halting problem).
So we can only hope to try and keep everything as simple as possible... as this winning entry shows that even 55 lines of clean code are enough complexity to hide what could be malicious code.
There's also a new contest for Underhanded Crypto -- https://underhandedcrypto.com -- that just finished what I think was its first year, though I know the Underhanded C competition has a longer history.
The 2014 Challenge: PiuPiu and the National Security Letter
The PiuPiu oversharing site allows users to create PiuPiu accounts and post 140-character messages. The federal government wants PiuPiu to surveil user activity on the site, by archiving any posts that match certain patterns outlined in a national security letter. Subject to the nondisclosure constraints of the letter, PiuPiu may not inform anyone of the surveillance request.
The underhanded goal is this: write surveil() in such a way that the act of surveillance is subtly leaked to the user or to the outside world. PiuPiu can not reveal the act of surveillance, but your function is technically able to edit the Piu or user structure during scanning. Find a way to alter that data (this alone is a bit of a challenge, since you are not supposed to alter the data, just scan it) in such a way that an informed outsider can tell if someone is being archived. The leakage should be subtle enough that it is not easily noticed.
[1] http://www.manux.info/en/