The factoring algorithm is not "try every possible number". If it was, then your statement would be correct.