The proper way to deal with a memory allocation failure, when a graceful return isn't possible, is to call abort() or similar, instead of trying to continue with a potentially inconsistent state.
Of course, it would be better to avoid getting into a situation where either a graceful return isn't possible, or poorly-tested recovery code has to be run to deal with the failure. I read somewhere that seL4's strategy is to divide the work in two phases. The first phase allocates all the resources it will need and checks all preconditions, but changes no state; the second phase has no allocations, does all the work, and will never fail. That way, any error recovery is a simple release of the resources it had already allocated.
Of course, it would be better to avoid getting into a situation where either a graceful return isn't possible, or poorly-tested recovery code has to be run to deal with the failure. I read somewhere that seL4's strategy is to divide the work in two phases. The first phase allocates all the resources it will need and checks all preconditions, but changes no state; the second phase has no allocations, does all the work, and will never fail. That way, any error recovery is a simple release of the resources it had already allocated.