I like Frama-C a lot in theory, though in practice I found it hard to use except on very small code segments.
When its unable to prove a range the feedback you get from the solvers is inscrutable enough that its very hard to figure out what additional data it would need to satisfy the analysis. (Sort of like parsing C++ compiler template related errors)
I'd hoped that language features like the typestate stuff that used to be in Rust would someday make the work required to use sound analysis tools in production code smaller. I'm not sure if much thought has been given to what kinds of accommodations languages could give to ease static analysis while still being programmer friendly.
It seems that newer languages have actually moved away from analysis friendliness in some respects, however. E.g. in C a signed overflow is always a bug so if analysis can prove one is possible you have something to fix. Several modern languages have defined signed operations to wrap and so that obvious safety test is no longer available. (You could define in your own code that it should never wrap, effectively writing in a subset of the language, but as soon as you call into third party code you never know if an overflow was intended and safe or not— not without extensive analysis)
> I'd hoped that language features like the typestate stuff that used to be in Rust would someday make the work required to use sound analysis tools in production code smaller. I'm not sure if much thought has been given to what kinds of accommodations languages could give to ease static analysis while still being programmer friendly.
Well, since the Rust borrow check is basically a static analysis that's always on and is required to pass for the compiler to emit code, we've put a lot of thought into how to make it as programmer friendly as possible. The final trick that seemed to got it to fall into place was restricting data to only one mutable reference at all times—this was a restriction that's easy enough to understand and can be readily explained through error messages and tutorials. There's still a learning curve, of course, but I think we've got the system down to a reasonable level.
Right, the borrow checker does great things. But memory safety is only one aspect of program correctness— though a universal and very important one.
If a program does anything important, if it handles secrets, if it enforces invariants ("the robots arm must not be out of the safe area"), etc. then other aspects of the programs logic may be just as safety critical as memory safety. Frama-C can prove other things about program saftey: this calculation cannot overflow, this error state can never be reached, this operation will complete within X steps... even if the results of non-safety don't result in anything to do with memory errors.
Some of the programmers expectations can be extracted from the program— signed integers won't overflow, loops (unless otherwise annotated) will always terminate, pointers will not be extended outside of their objects, a pointer to an integer won't be called as a function pointer, behavior will not depend on the order function arguments are evaluated in, etc. Some of the expectations must be expressed as assertions.
Once as many of the programmers expectations are understood the analysis needs to know the invariants that allow them to be true. Some of these can be extracted, some must be asserted.
The gap that exists today is that a lot that could be extracted (e.g. by another programmer reading in a context free way, with high probability of success) can't be soundly extracted because there is a lot of technically valid behavior which is just usually unlikely to be what anyone intended. This means that anyone attempting to apply tools like frama-c has to spend a lot of time making the implicit assumptions explicit with assertions. Thats why I was saying that something like making singed overflow defined may be a step backwards because it takes a whole class of "you couldn't possibly have intended this behavior" back to "maybe you meant this".
Sure, I totally agree that there are many other invariants that you might like to prove. Many of them require full-on dependent typing—not going there for now is a choice we made to make the language approachable. (See your sibling comment for how difficult it is just to get memory safety!) But I completely agree with you that there is more to be done.
I played with Rust for a day and while I was impressed with the language as a whole, I felt the borrow checker left a lot to be desired. I don't think having lifetime of references tied to scope works well in general. This is most evident in pattern matching where you are forced to grab your reference in the pattern (even if you don't need to use the reference until much further on), and there is no way to release it until the end of the scope (even though it can be dead much sooner).
Making the borrow checker know about the liveness of variables outside of scope is on the todo list. Like all of the borrow checker, it's fairly tricky to implement properly, but it should be doable. I don't think this issue leaves "a lot to be desired" though; it's a relatively minor tweak on the overall semantics.
Having a borrow check is important, because otherwise you're left with iterator invalidation and numerous other memory safety holes.
It would be good to hear other HN:ers experience with various static analysis tools for C.
I have had good experiences with Flexelint (PC-Lint). It does not attempt to deeply analyze control flow, more like compiler additional warnings. It flags a lot of common mistakes and can basically turn C into a more strictly typed language. I feel a lot more confident in C code if I know that it passes lint, since it warns if you try to mix unsigned and signed ints, cast away const, call functions with wrong types etc.
Like many static analyzers it takes some work to set it up, and tune which warnings you actually car about. It is definitely business-priced and feel a bit old (although command line tools age well.
The is a clear lack of good open source tools. I tried all i could find, but Splint was the only one that would flag switch-cases without break. It was last updated in 2010.
One feature you can start using right now: prove an assertion with the WP plugin. [1] It's especially useful to prove the absence of some undefined behaviour, bounds on variables, or invariance conditions.
Klee, EXE, Forensic, cbmc, BAP all mix symbolic analysis with SAT solvers into the C parse tree, and can come up with input to contradict the assertions. (automatic testcases or exploits)
Frama-C still beats all with it's interface, but Forensic is so nice that it even tells how the fix should look like.
Frama-C does not say much, when do don't tell it what it should do. It needs to reason over symbolic facts, C code is not enough. It's not just another simple static code checker.
You can look at the Frama-C blog what it does with a major openssl competitor, polarssl. http://blog.frama-c.com/
When its unable to prove a range the feedback you get from the solvers is inscrutable enough that its very hard to figure out what additional data it would need to satisfy the analysis. (Sort of like parsing C++ compiler template related errors)
I'd hoped that language features like the typestate stuff that used to be in Rust would someday make the work required to use sound analysis tools in production code smaller. I'm not sure if much thought has been given to what kinds of accommodations languages could give to ease static analysis while still being programmer friendly.
It seems that newer languages have actually moved away from analysis friendliness in some respects, however. E.g. in C a signed overflow is always a bug so if analysis can prove one is possible you have something to fix. Several modern languages have defined signed operations to wrap and so that obvious safety test is no longer available. (You could define in your own code that it should never wrap, effectively writing in a subset of the language, but as soon as you call into third party code you never know if an overflow was intended and safe or not— not without extensive analysis)