Since virtually no production C code† can actually handle the case of a random malloc() call failing (just like with exceptions in C++, the code to reliably unwind an allocation failure depends on exactly where you're at in your allocation pattern), the simplest and most reliable way of handling malloc() returning NULL is to rig the program to abort.
You're right that assert() isn't the most reliable way to do this (programs have to work whether or not assert() is a no-op; that's the point of assert).
On most platforms, you can just rig malloc to do the abort itself instead of failing --- either with configuration or by preloading a wrapper malloc. Some very, very large shops do exactly this.
Another very common idiom is "xmalloc", which does the malloc/if/abort dance. But xmalloc() misses every place where libraries call malloc(); the most obvious example is strdup(), but the more pernicious issue is 3rd party code that can't know to use x-whatever().
Calling malloc and then immediately assert()'ing success is a reasonable shorthand. That exact same code can be made safe on any mainstream platform just by changing malloc's configuration.
† On general-purpose platforms; I know things are more complicated on embedded platforms.
Good point about the 3rd party libraries there, it's true that even if you do your own cleanup after manual mallocs, you can't do much about theirs.
Do you have an example of documentation that shows how to configure malloc to do some cleanup in the case of failure? Based on the Linux manpages I can't really see any obvious way to do this short of preloading some kind of malloc wrapper that's custom-written for the application.
I think it may depend on the distro (on Ubuntu, for instance, you can set MALLOC_CHECK_ to 2 to force aborts on errors --- but this also forces the use of a debugging malloc, and you should care about malloc performance).
So what I'd recommend is, grab Jason Evans' jemalloc package and use that (it's good enough for Facebook!).
You're right that assert() isn't the most reliable way to do this (programs have to work whether or not assert() is a no-op; that's the point of assert).
On most platforms, you can just rig malloc to do the abort itself instead of failing --- either with configuration or by preloading a wrapper malloc. Some very, very large shops do exactly this.
Another very common idiom is "xmalloc", which does the malloc/if/abort dance. But xmalloc() misses every place where libraries call malloc(); the most obvious example is strdup(), but the more pernicious issue is 3rd party code that can't know to use x-whatever().
Calling malloc and then immediately assert()'ing success is a reasonable shorthand. That exact same code can be made safe on any mainstream platform just by changing malloc's configuration.
† On general-purpose platforms; I know things are more complicated on embedded platforms.