> First, with IPv4 this will have the potential to increasingly penalize innocent bystanders... Worst case, this will give bad actors the option to lock the original owner out of their own server if they have a botnet host in the same network.
So instead of looking, like the author of these new options, for ways to make life for the bad guys harder we do nothing?
Your concerned are addressed in TFA:
> ... and to shield specific clients from penalty
> A PerSourcePenaltyExemptList option allows certain address ranges to be exempt from all penalties.
It's easy for the original owner to find the list of all the IP blocks the three or four ISPs he's legitimately be connecting from to that exemption list.
I don't buy your argument nor all the variation on the same theme: "There's a minuscule risk of X, so we absolutely nothing but saying there's nothing to do and we let bad guys roam free!".
There's nothing more depressing than that approach.
Kudos to the author of that new functionality: there may be issues, it may not be the panacea, but at least he's trying.
> So instead of looking, like the author of these new options, for ways to make life for the bad guys harder we do nothing?
Random brute force attempts against SSH are already a 100% solved problem, so doing nothing beyond maintaining the status quo seems pretty reasonable IMO.
> I don't buy your argument nor all the variation on the same theme: "There's a minuscule risk of X, so we absolutely nothing but saying there's nothing to do and we let bad guys roam free!".
Setting this up by default (as is being proposed) would definitely break a lot of existing use cases. The only risk that is minuscule here is the risk from not making this change.
I don't see any particularly reason to applaud making software worse just because someone is "trying".
Sure but if I only used fail2ban for sshd why should I install two separate pieces of software to handle the problem which the actual software I want to run has it built in?
Normally I would agree with you, but fail2ban is a Python routine which forks processes based on outcomes from log parsing via regex. There’s so many ways that can go wrong…and has gone wrong, from one or two experiences I’ve had in the past.
This is exactly the sort of thing that should be part of the server. In exactly the same way that some protocol clients have waits between retries to avoid artificial rate limiting from the server.
As I’ve already posted, I’ve ran into bugs with fail2ban too.
Also adding firewalling to SSH is hardly “kitchen sinking” (as another commenter described it). You’re literally just adding another layer of security into something that’s literally meant to be used as an out of the box solution for creating secure connections.
If you want to take issue with the “kitchen sink” mentality of SSH then complain about its file transfer features or SOCKS support. They are arguably better examples of feature creep than literally just having the server own what connections it should allow.
> Also adding firewalling to SSH is hardly “kitchen sinking”
sshd is a service. It may be one among dozens of other services running on a host.
Now imagine for a moment, if EVERY service on the host took that approach. Every backend service, every network-facing daemon, every database, every webserver, voip servers, networked logging engines, userspace network file systems, fileservers...they all now take security into their own hands.
Every single one of them has its own fail2ban-ish mechanism, blocklists it manages, rules for what to block and how long, what triggers a block, if and when a block will be lifted...
Oh, and of course, there is still also a firewall and other centralized systems in place, on top of all that.
How fun would such a system be to administer do you think? As someone with sysadmin experience, I can confidently say that I would rather join an arctic expedition than take care of that mess.
There is a REASON why we have things like WAFs and IDS, instead of building outward-facing-security directly into every single webservice.
If you’ve been a sysadmin as long as I have then you’ll remember when services didn’t even manage their own listener and instead relied on a system-wide daemon that launched and managed those services (inetd). Whereas now you have to manage each listener individually.
That was additional initial effort but the change made sense and we sysadmins coped fine.
Likewise, there was a time when server side website code had to be invoked via a httpd plugin or CGI, now every programming language will have several different web frameworks, each with their own HTTP listener and each needing to be configured in its own unique way.
Like with inetd, the change made sense and we managed just fine.
Tech evolves — it’s your job as a sysadmin to deal with it.
Plus, if you’re operating at an enterprise level where you need a holistic view of traffic and firewalling across different distinct services then you’d disable this. It’s not a requirement to have it enabled. A point you keep ignoring.
> Likewise, there was a time when server side website code had to be invoked via a httpd plugin or CGI, now every programming language will have several different web frameworks, each with their own HTTP listener and each needing to be configured in its own unique way.
And still we keep all those webservices, be they in Java, Go, node, C# or Python, behind dedicated webservers like nginx or apache.
Why? Because we trust them, and they provide a Single-Point-Of-Entry.
> Tech evolves — it’s your job as a sysadmin to deal with it.
Single-Point-Of-Entry is still prefered over having to deal with a bag of cats of different services each having their own ideas about how security should be managed. And when a single point of entry exists, it makes sense to focus security there as well.
This has nothing to do with evolving tech, this is simple architectural logic.
And the first of these points that every server has, is the kernels packet filter. Which is exactly what tools like fail2ban manage.
> A point you keep ignoring.
Not really. Of course an admin should deactivate svc-individual security in such a scenario, and I never stated otherwise.
The point is: That's one more thing that can go wrong.
> And still we keep all those webservices, be they in Java, Go, node, C# or Python, behind dedicated webservers like nginx or apache.
Not really no. They might sit behind a load balancer but that's to support a different feature entirely. Some services might still be invoked via nginx or apache (though the latter has fallen out of fashion in recent years) if nginx has a better threading model. But even there, that's the exception rather than the norm. Quite often those services will be stand alone and any reverse proxying is just to support orchestration (eg K8s) or load balancing.
> Single-Point-Of-Entry is still prefered over having to deal with a bag of cats of different services each having their own ideas about how security should be managed.
Actually no. What you're describing is the castle-and-Moat architecture and that's the old way of managing internal services. These days it's all about zero-trust.
But again, we're talking enterprise level hardening there and I suspect this openssh change is more aimed at hobbyists running things like Linux VPS
> > A point you keep ignoring.
> Not really. Of course an admin should deactivate svc-individual security in such a scenario, and I never stated otherwise. The point is: That's one more thing that can go wrong.
The fact that you keep saying that _is_ missing the point. This is one more thing that can harden the default security of openssh.
In security, it's not about all or nothing. It's a percentages game. You choose a security posture based on your the level of risk you're willing to accept. For enterprise, that will be using an IDP to manage auth (including but not specific to SSH). A good IDP can be configured to accept requests from non-blacklisted IP, eg IPs from countries where employees are known not to work in), and even only accept logins from managed devices like corporate laptops. But someone running a VPS for their own Minecraft server, or something less wholesome like Bit-Torrent, aren't usually the type to invest in a plethora of security tools. They might not even have heard of fail2ban, denyhosts, and so on. So having openssh support auto-blacklisting on those servers is a good thing. Not just for the VPS owners but us too because it reduces the number of spam and bot servers.
If your only concern is that professional / enterprise users might forget to disable it, as seems to be your argument here, then it's an extremely weak argument to make given you get paid to know this stuff and hobbyists don't.
a system where sshd outputs to a log file then someone else picks it up and then pokes at iptables, seems much more of hacky than having sshd supporting that natively, imo. Sshd is already tracking connection status, having it set the status to deny seems like less of a kitchen sink and more just about security. the S in ssh for secure, and this is just improving that.
It's battle-tested for well over a decade, has accumulated 10.8k stars and 1.2k forks on github, so it seems to do something right no?
Not to mention that even if it were otherwise, that's not a reason to ignore UNIX philosopies that have served the FOSS world well for over half a century at this point.
Last but not least, there are any number of alternative solutions.
Just because it's 'battle tested' and has stars and is useful does not preclude it from being a hacky pos. Reading logs using regexps and then twiddling IP tables is not the cleanest method of achieving this result. I would much prefer if this functionality were either handled like ssh or if there was some kind of standardized messaging (dbus?) that was more purposeful and didn't rely on regex.
It's useful because you can hook it up to anything that produces logs, it's hacky because that means you are using regexp. If the log format changes, you're likely fucked, not to mention that regexps are notoriously hard to make 'air tight' and often screwed up by newbies. Add to that in a case where your regexes start missing fail2ban will stop doing it's job silently.. not great my friend.
It's been a useful hack for a very long time, but I'd like to see us move on from it.
The issue is that the log parsing things like fail2ban work asynchronously. It is probably of only theoretical importance, but on the other hand the meaningful threat actors are usually surprisingly fast.
Yeah, they exist because nothing better was available at that time.
It doesn’t hurt to have this functionality in openssh too. If you still need to use fail2ban, denyhosts, or whatever, then don’t enable the openssh behaviour feature. It’s really that simple.
UNIX Philosophy: "Do one thing, and do it well". An encrypted remote shell protocol server should not be responsible for fending off attackers. That's the job of IDS and IPS daemons.
Password-based ssh is an anachronism anyway. For an internet-facing server, people should REALLY use ssh keys instead (and preferably use a non-standard port, and maybe even port knocking).
It’s better if you want an out of the box secure experience. This might be quite a nice default for some VPSs.
If you have a IDS and IPS set up then you’re already enterprise enough that you want your logs shipped and managed by a single pane of glass. This new SSH feature isn’t intended to solve enterprise-level problems.
Plus if you want to argue about “unix philosophy” with regards to SSH then why aren’t you kicking off about SOCKS, file transfer, port forwarding, and the countless other features SSH has that aren’t related to “shell” part of SSH? The change you’re moaning about has more relevance than most of the other extended features people love SSH for.
> This new SSH feature isn’t intended to solve enterprise-level problems.
But service level security features have the potential to cause enterprise-level problems.
Sure, in an ideal world, all admins would always make zero mistakes. And so would the admins of all of our clients, and their interns, and their automated deployment scripts. Also in that perfect world, service level security features would never be on by default, have the same default configuration across all distros, and be easy to configure.
But, alas, we don't live in a perfect world. And so I have seen more than one service-level security feature, implemented with the best of intentions, causing a production system to grind to a halt.
It would be frustrating to be denied access to your own servers because you are traveling and are on a bad IP for some reason.
Picture the amount of Captchas you already getting from a legitimate Chrome instance, but instead of by-passable annoying captchas, you are just locked out.
I have fail2ban configured on one of my servers for port 22 (a hidden port does not have any such protections on it) and I regularly lock out my remote address because I fat finger the password. I would not suggest doing this for a management interface unless you have secondary access
what do you if you get mugged and you laptop and phone and keys are taken or stolen from you? or lost?
After this party, this guy needed help, he lost his wallet and his phone, his sister also went to the party and gave him a ride there but had left. he didn't know her number to call her, and she'd locked down her socials so we couldn't use my phone to contact her. we were lucky that his socials weren't super locked down and managed to find someone that way, but priv keys are only good so long as you have them.
You can and you should back up your keys. There isn't a 100% safe, secure and easy method that shields you from everything that can possibly happen, but there are enough safe, secure and easy ones to cover vast majority of cases other than a sheer catastrophe, which is good enough not to use outdated and security prone mechanisms like passwords on network exposed service.
I use a yubikey. You need a password to use the key. It has it's own brute force management that is far less punishing than a remote SSH server deciding to not talk to me anymore.
My keyboard has a built in USB hub and ports. They key lives there. They keyboard travels with me. It's hard to lose.
I have a backup key in storage. I have escrow mechanisms. These would be inconvenient, but, it's been 40 years since I've lost any keys or my wallet, so I feel pretty good about my odds.
Which is what the game here is. The odds. Famously humans do poorly when it comes to this.
If I present the incorrect key fail2ban locks me out as well. Two incorrect auth attempts locks out a device for 72 hours. The idea is for regular services which depend on ssh (on port 22) to work regularly (because of key auth) but to block anyone attempting to brute force or otherwise maliciously scan the system.
Doesn’t change the advice, if this is your only management interface, don’t enable it :)
Also you know you can have MFA even with pw authentication right? :)
What's the alternative? If you get onto a bad IP today, you're essentially blocked from the entire Internet. Combined with geolocks and national firewalls, we're already well past the point where you need a home VPN if you want reliable connectivity while traveling abroad.
What happens when your home VPN is inaccessible from your crappy network connection? There are plenty of badly administered networks that block arbitrary VPN/UDP traffic but not ssh. Common case is the admin starts with default deny and creates exceptions for HTTP and whatever they use themselves, which includes ssh but not necessarily whatever VPN you use.
Better internet isn't always available. A VPN on the ssh port isn't going to do you much good if someone sharing your IP address is doing brute force attempts against the ssh port on every IP address and your system uses that as a signal to block the IP address.
Unless you're only blocking connection attempts to ssh and not the VPN, but what good is that? There is no reason to expect the VPN to be any more secure than OpenSSH.
If you're using an IP address that's being used to brute force the entire Internet, it's likely that lots of websites are blocking it. If that doesn't matter to you and all you need is to get into a particular SSH server, and also the network blocks VPNs, you're still fine if the SSH is on port 9022 and VPN is port 22. If it's not your own SSH server and it's port 22, then you're still fine if your own VPN is port 22 (on a different host).
Hacking into the VPN doesn't get the attacker into the SSH server too, so there's defense in depth, if your concern is that sshd might have a vulnerability that can be exploited with repeated attempts. If your concern is that your keys might be stolen, this feature doesn't make sense to begin with.
> If you're using an IP address that's being used to brute force the entire Internet, it's likely that lots of websites are blocking it.
Websites usually don't care about ssh brute force attempts because they don't listen on ssh. But the issue isn't websites anyway. The problem is that your server is blocking you, regardless of what websites are doing.
> If that doesn't matter to you and all you need is to get into a particular SSH server, and also the network blocks VPNs, you're still fine if the SSH is on port 9022 and VPN is port 22. If it's not your own SSH server and it's port 22, then you're still fine if your own VPN is port 22 (on a different host).
Then you have a VPN exposed to the internet in addition to SSH, and if you're not rate limiting connections to that then you should be just as concerned that the VPN "might have a vulnerability that can be exploited with repeated attempts." Whereas if the SSH server is only accessible via the VPN then having the SSH server rate limiting anything is only going to give you the opportunity to lock yourself out through fat fingering or a misconfigured script, since nobody else can access it.
Also notably, the most sensible way to run a VPN over TCP port 22 is generally to use the VPN which is built into OpenSSH. But now this change would have you getting locked out of the VPN too.
The situation is the SSH server is exposed everywhere, and you also have an unrelated VPN, maybe even via a paid service you don't manage. The VPN just provides you with an alternative IP address and privacy when traveling. It matters a lot more if someone hacks the SSH server.
It would also be very rare. The penalties described here start at 30s, I don't know the max, but presumably whatever is issuing the bad behavior from that IP range will give up at some point when the sshd stops responding rather than continuing to brute force at 1 attempt per some amount of hours.
And that's still assuming you end up in a range that is actively attacking your sshd. It's definitely possible but really doesn't seem like a bad tradeoff
lol. depending were you travel the whole continent is already blanket banned anyway. but that only happens because nobody travels there. so it is never a problem.
There is nothing wrong with this approach if enabled as an informed decision. It's the part where they want to enable this by default I have a problem with.
Things that could be done is making password auth harder to configure to encourage key use instead, or invest time into making SSH CAs less of a pain to use. (See the linked paper, it's not a long read.)
So instead of looking, like the author of these new options, for ways to make life for the bad guys harder we do nothing?
Your concerned are addressed in TFA:
> ... and to shield specific clients from penalty
> A PerSourcePenaltyExemptList option allows certain address ranges to be exempt from all penalties.
It's easy for the original owner to find the list of all the IP blocks the three or four ISPs he's legitimately be connecting from to that exemption list.
I don't buy your argument nor all the variation on the same theme: "There's a minuscule risk of X, so we absolutely nothing but saying there's nothing to do and we let bad guys roam free!".
There's nothing more depressing than that approach.
Kudos to the author of that new functionality: there may be issues, it may not be the panacea, but at least he's trying.