This is really a different issue. You'd be extending the domain of factorial beyond what it's traditionally been used for (same as if you extended it to include other numbers outside the natural). That you can do it for some sets (gamma function) doesn't matter, because it also stands on its own. And if you extend it, it's not the factorial function anymore, which is defined to have the non-negative integers as its domain.
But conceptually the primary use case in combinatorics is for permutations, i.e. $n!=|S_n|$, which only makes sense for $n \ge 0$. You could even make an argument for n!= because $|S_0|$ should be 1 on its own (because there's exactly one permutation of the empty set) without saying that it makes formulas easier. (This, I note again, matters regardless of what the gamma function does.)
But conceptually the primary use case in combinatorics is for permutations, i.e. $n!=|S_n|$, which only makes sense for $n \ge 0$. You could even make an argument for n!= because $|S_0|$ should be 1 on its own (because there's exactly one permutation of the empty set) without saying that it makes formulas easier. (This, I note again, matters regardless of what the gamma function does.)