If you don't have anything better to do, just abort().
One (naughty) option is to malloc a decent sized buffer at startup (50MB say), and then when malloc fails free it, and immediately display a warning box.
One most modern 64-bit systems, malloc will never fail anyway, you'll get killed due to memory overcommit.
The alternative, if you have no good way of testing the codepaths, or if you're running on a system with overcommitted VM, is basically segfaulting. And you're running on a system with overcommitted VM.
Overcommit is not actually relevant. An overcommit OOM-kill is a manifestation of the administrator killing the process for being a memory hog (it's just that the administrator has delegated this decision to some less-than-scrutable kernel code).
Library code should in general punt out-of-memory conditions back to the caller.
It is if you ever want to exercise the code paths. Overcommit means you never get an out of memory error. Your process just dies. Kind of like it would if you just aborted.
When you're running tests in development you can easily set up a test environment with and without overcommit enabled (and in fact there are other ways to limit allocated memory on a per-process basis that work even with overcommit, such as with rlimits).
...but that doesn't even really matter, because in order to fully exercise those code paths you need precise control over exactly which call to malloc() returns NULL, which in practice means you need to interpose malloc() with a debugging framework to force a NULL return.
One (naughty) option is to malloc a decent sized buffer at startup (50MB say), and then when malloc fails free it, and immediately display a warning box.
One most modern 64-bit systems, malloc will never fail anyway, you'll get killed due to memory overcommit.