I use it with ssh and password auth disabled, is there a reason not to? Might be overkill but the host is in my home so physical access if I ever get locked out is not an issue.
I think this is still reasonable, attackers may have a database of leaked keys (e.g. if you ever accidentally commited to GitHub, or ever ran a malicious script which uploaded it), which they then try on random servers.
