Not using default ports will mildly confuse automated scans and do absolutely nothing to a determined attacker. Or somebody with nmap, which is not the same thing.
If you're whitelisting IPs, you may as well run it on port 22.
No it makes it harder and more of a pain. Trust me I have a friend who loves breaking into my personal server. That one trick two ssh running on different ports screwed with him for a long, long time. He is a genius of a hacker and has been doing it for a living for years. When he finally got in he was so pissed that threw him.
You are describing an anecdotal instance of a person whose capabilities are not established being thrown by something that nmap will catch on a normal scan.
Color me skeptical. I shall decline to "trust you."
Not my competence it his competence I trust and I got him good with that one since it never occurred to him that one stupid trick messed with him for so long. Lie 5 minutes a month.
One small benefit of using a non default port is that it keeps down the noise from automated scans. So any "real" suspicious activity will now stand out as it is not drowned out by the noise anymore.
If you're whitelisting IPs, you may as well run it on port 22.