You can also use https://tinyssh.org instead of the sshd from OpenSSH. It only implements key based authentication and newer encryption algorithms. It's available on Ubuntu:
but it's not widely used, which makes using it risky. To my knowledge there hasn't been a formal audit. Also it doesn't implement the scp protocol, but you can use rscync over ssh