You have that exactly backwards: if someone is hitting you with a password bruteforce from a single IP address (which is the only threat that fail2ban mitigates) then it is assuredly nothing personal at all.
A personal insult, if you are ever unfortunate enough to receive one, will be much more stealthy and neither fail2ban nor any other magical rock will protect you against it.
If anything, you're doing them a (miniscule) favor by keeping them from wasting more resources on failed login attempts. If you really hated them, you'd set up a honeypot.
More fun: setup a fail2ban actionban script that instead of banning the IP, shapes the traffic coming from it to have abysmal bandwidth so requests/responses takes really long time, so they'll have to timeout instead of getting failures.