Why is there a limit at all? When you hash the password, it should be a fixed size no matter the password length. Make sure you're not just storing the password on its own.
I recently discovered that some password hashes take time proportional to the length of the password times the number of rounds. E.g. if bcrypt work factor is set to 16 and you have a 1M byte password then the time is proportional to `1000000 * 2^16`
Why it doesn't first hash the password so it's `1000000 + len(hash) * 2^16` is beyond me, and something a crypto expert will have to answer.
In practice this means that setting an arbitrarily high limit on password size would open you up to a DOS attack via (effectively) quadratic time password hashing.