You need to divide by 3! and 2!. One way to look at it is that every selection of five from the 4M and 10F is equally likely. There are \binom{14}{5} (i.e. 14 choose 5) such selections. Of these, \binom{4}{3} * \binom{10}{2} have exactly 3M and 2F. Therefore, the probability is \binom{4}{3} * \binom{10}{2} / \binom{14}{5}, or (4 * 3 * 2 * 10 * 9) 5! / (3! 2! (14 * 13 * 12 * 11 * 10)) = 10 * 9/(13 * 11 * 7) = 0.08991...
Ahh indeed, I thought there were 5!=120 ways to order 3M and 2F, but apparently there are only 5!/(2!3!) = 10 ways: {MMMFF,MMFFM,MFFMM,FFMMM,FMMMF,MFMMF,MMFMF,FMMFM,MFMFM,FMFMM}. I made the mistake of thinking you can distinguish between individual males/females.
