I suspect the distributed cracking will move to the same pattern as the SMTP/pop3 brute force guys did and use one IP per x+1 seconds where x=the ssh penalty window. We have seen this on our customer facing smtp server where we have hundreds of remote compromised IPs trying each one password per 30-60min. Still, I welcome this change as there are enough single prick attackers out there where this will help cut down on the size of the logs to process / digest.
Actually this already is the SOTA of cracking. My honeypot can see several different IP is brute forcing concurrently, and they seems irrelevant. But once you let one of them login, it will quit immediately and all those IPs will quiet after ~15sec. Then one of those IPs will login again to deploy miner.
reminds me of using the old MIRROR target in iptables back in the day. before it was removed because its ridiculous. we used to watch script kiddies trying to brute force their own hosts but even then we knew it was ripe for abuse.
Probably for the best, since it sounds like that could be used for DDoS amplification and/or reflection.
For example, if an attack could spoof traffic to get two different reflectors hall-of-mirror-ing each other, or using a botnet that spoofs traffic to get one collection of dupes to slam a single victim in response, etc.
How would you spoof multiple valid packets in a TCP-based protocol requiring a sequence of interactions when you can't receive any of the ACKs (because they'll be sent to not-your-IP)?
This is already the practice in my experience. Fail2ban has become completely useless for ssh about 5~6 years ago. Always just one to three tries per IP address.
So looks like this openssh feature is a decade late.
For what it's worth if you have control over both client and server and don't want to limit access using a strict IP whitelist, an alternative solution that will keep your logs quieter and add additional protection is to use good old fashioned port knocking. knockd on Linux helps with automating this on the server side. Client side you can use anything (although knockd does include a dedicated client) to send your sequence of packets before actually connecting.
I really think this solution is underrated. Port knocking is robust, doesn't use any special technology, and servers using it can't reasonably be scanned for. The only real disadvantage is that any passive observer can see your knock sequence in "plaintext" (so that includes anyone logging netflow).
Even so, I don't know why OpenSSH hasn't implemented it instead of the the silly fail2ban theatre we're discussing in these comments.
One thing to help with the passive observer would be to have the knock sequence be time varying like a TOTP. It's still a very thin addition but more defense in depth the better sometimes.
lol, hadn't read all the comments before posting mine.. Have an upvote! Actually who not do both. Vary the knock code and the resulting ssh port using succesive codes.
I just checked knockd man page and it turns out it can use a one_time_sequences file that contains a sequence of port knock combinations. I wonder if this file is dynamically checked, or loaded and parsed during startup? Or could one simply echo the TOTP code straight into that file and hup the knockd service each time (let's say the TOTP interval was set to something like 5 minutes).
I wonder could you combine command line TOTP tools with port knock for a fully time-based unique knock codes? Or even use the TOTP code for the ssh port?
I think it goes without saying that you would still want to be using keys instead of passwords for the actual authentication. Port knocking should always be an additional layer, not a replacement layer.
I find adding dynamic dns entries to my firewalls much more efficient and to have a more meaningful protection value.
A timed job that checks the up of your clients and updates the firewall every 30 seconds seems a much more secure method than having a magic sequence of ports that can be captured in the wild.
It’s hard to spoof a full tcp connection (with a key) needed to update your ddns.
Best part is you can leave your ddns to a separate box or service which complicates the compromise of a single host
fail2ban works just fine with sshd. I combine this GeoIP blocking ceetain troublesome locations in firewalls. 98% of my scanning / exploiting comes from 11 countries.
fail2ban is great, but only works on the local host.
The post says: "Right now our perimeter firewall is blind to whether a brief SSH connection was successful or not"
(I suspect there's a way to set up centralised logging and fail2ban running looking at those centralised logs and sending updates to a perimeter firewall, but that's not a typical deployment of fail2ban. Or at least is wasn't when I was heavily using it a while back.)
How about a service that lets bruteforcers "in" after some number of failed attempts, but what they get is just a fake command prompt that accepts all of their commands? I'm sure hackers would eventually adapt, but it would annoy them for a while.
I can't tell if that's sarcasm, but that's a thing already, called honey pot. Very often, you set it up to let attackers in so you can analyze what they're doing and protect yourself better.
Hem... There is PerSourcePenaltyExemptList to whitelist specific hosts so while I agree it might create a bit more complexity (not much different than handling fail2ban and co) it's not "blocking" like described.
PerSourcePenalties-like abuse mitigations are not very subtle, but often very effective.
My own experience with this is not as much with SSH as with SMTP. If your particular IP emits a single non-deliverable message every 30 minutes or so, that's fine.
But: more than a handful of SPF failures for the same sender/recipient combo within 10 minutes or so? Yeah, you're now on the general-deny-list. And persisting even then? Automagically tar-pitted on the firewall, and have fun...
> But: more than a handful of SPF failures for the same sender/recipient combo within 10 minutes or so? Yeah, you're now on the general-deny-list. And persisting even then? Automagically tar-pitted on the firewall, and have fun...
What is the advantage of bothering to do this?
To me, it seems like you're risking blocking legitimate traffic (as two obvious examples, in cases where a sending mailserver is temporarily misconfigured, or if a malicious mailserver is using a cloud IP that is later recycled to a completely unrelated user), for basically zero upside.
Like: yeah, I don't care that you're self-DDOSing if you're an actual spammer. But, just in case you're a legitimate sender lacking SPF records, and you just need some help, I still want some kind of signal.
For SSH, I guess you might want something similar, just to catch authorized-yet-misconfigured remotes...
Such blocking was always not about protection, but not having 100500 error messages in your logs every hour. Attackers that are blocked are the kind of attackers that cast a wide net and see what get caught.
They already do. The IPv6 public routable prefix is the first 48 bits, so a /48 is what ~everyone uses as the analogy of the v4 /32, for rate limiting purposes.
We have two perspectives, the normal user and the scraper. The normal user either acquires a /64, /56 or /48, depending on the whim of their ISP. There is typically no cost difference between these options, which means that the scraper (or their upstream proxy) always chooses a provider which offers a /48.
Thus, the default unit of IPv6 blocking must be a /48. This situation will persist for as long as /48s are readily available at the same price as /64s.
Perversely, the reason we don't have this issue in IPv4 space is because the address space is of the same order of magnitude as the number of potential users. That artificial scarcity means that a routable /24 is x256 the cost of a end-user /32, so the unit of blocking can be a /32.
> Perversely, the reason we don't have this issue in IPv4 space is because the address space is of the same order of magnitude as the number of potential users.
Do we not? And is it really?
There are some /32 IPv4 addresses hosting many users, e.g. with CG-NAT, and it's already an issue with regards to blocking/rate-limiting.
Just like there are single-user /48s and multi-user /64s, there can be single-user /32s and /32s hosting tons of users behind a CG-NAT.
Sure, but that's the same argument I'm making: the unit of blocking will be the largest unit that is routinely allocated to a single user. In IPv4 space that's a /32, so people block by /32. In IPv6 space that's a /48, so people block by /48. Check out Let's Encrypt's rate limit policy, for example.
The difference I'm pointing out between IPv4 and IPv6 is that nobody is giving single IPv4 users /24s for their own use. But IPv6 /48s (which are theoretically somewhat equivalent to IPv4 /24s) are freely available. This is a problem because it makes over-blocking even more likely than it already is. And as you point out elsewhere, over-blocking is already an issue in IPv4 space.
If you have two addresses in the same /64, you know almost certainly they are on the same LAN.
If you have two addresses in neighboring /64s (same initial 63 bits), or in general within the same /48 (same initial 48 bits), you know almost certainly that they are somehow within the same organization. They could be within the same building, or in the same company, or using the same ISP or cloud provider; you don't know, but they are somehow related. How do you know? Well, since a /49 isn't individually routable in BGP, they have to _somehow_ originate at the same upstream network. There has to be some sort of cooperation between them (possibly through an ISP as a middleman).
But if they are in _neighboring_ /48s, you don't have this kind of guarantee. They could be from completely different organizations. Most likely, they are on the same continent (since they were given out by the same regional internet registry; RIR), but even that is not really guaranteed.
So when you are bucketing addresses for rate limiting purposes, a /48 is a reasonable place to start doing that, just like /24 is for IPv4. Of course, you may need to get smarter than that (e.g. an attacker could have access to a /32), but it's a reasonable starting spot.
> So when you are bucketing addresses for rate limiting purposes, a /48 is a reasonable place to start doing that, just like /24 is for IPv4.
I've encountered assumptions such as this one as a user, and they're really frustrating.
More than once I've found myself banned from being able to log in, view a site etc. because of somebody else's bad behavior I temporarily share a CG-NAT or large public Wi-Fi with, or more likely because somebody topologically close to me got hacked.
Meanwhile, actual attackers are using pretty much the entire IPv4 space worth of compromised embedded devices spread across the globe...
gnfargbl didn't say the _address_ was unroutable, but that effectively, routing policies mean that /48 is a common minimum unit for administrative purposes (similar to how /24 has a special “minimum size” meaning for IPv4).
Most ISP customers get either a /48 or a /56. The IPv6 equivalent of a random 1.2.3.4/32 consumer IP address would be something like 2001:abcd:1234::/48 or 2001:abcd:1234:ff::/56 in IPv6.
It's certainly an annoying future. I fully expect some sort of ASN-reputation map and service to become de rigeur in the next decade. Pretty sure heavier-duty SIEM/IPS systems already do that.
I’ve been thinking that we need a new ‘WireGuard of SSH’ for a while. SSH is such a complex behemoth now. Before WireGuard came along VPNs were horrible to work with, the cryptography was bad and they were too configurable. Just do one thing and do it well - provide a way to establish an encrypted remote shell. Let others build on top of that if they need more.
Vulnerabilities lurk in overly complex software with a thousand bells and whistles. By reducing the code paths you’re making software that is much easier to audit and fuzz.
I built a HTTP-based shell system on top of a configuration management tool. It uses public key cryptography via JWTs, and generates noise to obfuscate keystroke timing. Since it's all over HTTP, you don't really get any port knocking, and you can expose access using proxies and middleware.
I'm curious about what works and what doesn't: Many experts don't trust OpenBSD's security implementation (I don't want to raise the issue, just stating fact - many don't). Yet many do trust OpenSSH's security implementation, and OpenSSH is of course an OpenBSD project.
What works in OpenSSH that doesn't work in OpenBSD? Maybe it's as simple as, though under the same umbrella OpenSSH uses a different team, methodology, etc.
That's easy: OpenBSD pioneered OS-wide dragnet code audits, and also an ethos of minimally-invasive, parsimonious OS security features (PID randomization and syscall pinning are two emblematic examples). The rest of the world caught up to OpenBSD on dragnet audits, and then surpassed it; meanwhile, the OpenBSD ethos of minimal, modest security features was probably less effective than the Linux approach of features that bend the whole universe around security challenges or that thread deeply through the operating system.
More than anything else though, it's not so much that OpenBSD is less resilient than Linux (I think a case could be made), and more that OpenBSD isn't materially more secure.
They should have killed the "only N K holes in Z time" tagline a long time ago.
I have written poorly worded criticisms of the OpenBSD project on HN before. They boil down to this: from my observation, it looks like the approach to security in the OpenBSD project is adding more code to solve security issues when it should be the opposite. Code is a liability. You should write as little of it as possible to solve the problem at hand and not spare a single line that isn't needed. In this case the problem is getting a shell on a remote host. Why do you need so many configuration options to solve this single problem?
The OP is a prime example of the opposite happening - it's adding code to prevent repeated authentication failures. Why would this be needed in the first place? If you have configured OpenSSH correctly (that is, using public key authentication instead of password auth, which should not even be an option), then repeated authentication failures should not be a problem. At worst, they take up some CPU time.
Much of the code in OpenBSD and the wider OpenBSD projects also address memory safety issues which would not be issues in the first place if they just used a memory safe language. Yet they push ahead using C in the full knowledge that there are better options available. Java, Go, Python, Rust, I literally don't care, anything would be better than C. Developers should not need to spend hours carefully poring over each others patches to find critical mistakes when it comes to memory. They should not need to spend hours reading C development guidelines or rely on mailing list oracles. By eliminating memory errors as a class, mental capacity is freed up to identify logic errors.
I can't speak for OpenBSD specifically but I can speak to some of my thoughts in why an operating system continues to use C. Supporting a language ecosystem is not easy, the less "default" languages needed to bootstrap the core system the better. The nice part about C is that it's one of the few languages suited for both kernel space and user space. Out of the alternatives you listed the only language that could even seriously be considered for kernel space is rust, and even that took a lot of back and forth to get it to that point in the Linux kernel. Higher level languages have a larger range as assumptions and you have to tow those accomodations in to kernel space if you want to use them. There is also the issue of the memory management in kernel space being much more of a complicated environment than user space. How do I teach the borrow checker about my MMU bring up and maintenance?
I am also skeptical to your claim about removing memory bugs freeing up brain space for logic bugs, at least for Rust. Rust has grown quite a number of language features, that in my experience, result in a higher cognitive load compared to C. If you seriously reduce your reliance on the C macro system (as Plan
9 has shown possible), the language itself is quite simple.
In very simple terms, their approach to security is auditing to try and remove all bugs, but offering very little to protect against the case where there is a remote root hole, such as they have had in the past.
Things have gotten better in recent years with things like pledge and doas, but they are quite lacking compared to proper MAC or sandbox implementations. Worse, many of the OpenBSD devs don't seem to know much about those technologies and are kind of dismissive. I remember an OBSD user or enthusiast trying to argue Pledge was equivalent to SELinux which was pretty bad.
I'd take a slimmed down hardened Linux install over OpenBSD any day of the week.
> In very simple terms, their approach to security is auditing to try and remove all bugs, but offering very little to protect against the case where there is a remote root hole, such as they have had in the past.
I'm not sure what is meant by "protect against the case where there is a remote root hole". Do you mean, to mitigate harm from existing holes? They secure things from top to bottom, but maybe you mean some kind of authorization issues? To proactively prevent holes? They do a lot of engineering around the latter (and around other attacks) - to the extent that many question the wisdom of solving 'problems' for which there is no proven exploit. And they have had very few remote holes - whatever the reason, there's not evidence that they fail to prevent them.
> I'm not sure what is meant by "protect against the case where there is a remote root hole".
I don't understand how this is ambiguous. I mean limit the damage that an attacker can do if they get root - this is something RHEL can do and OBSD pretty much refuses to.
> They secure things from top to bottom
Eh. Kind of. The devs are against security technologies that they think add too much complexity to their system regardless of benefits. That's why they don't have any kind of RBAC or MAC, just plain old DAC. You get root, you get everything - pledge and unveil won't help too much there.
It wasn't an attack, but a genuine question, for which I provided two possible interpretations. I was (am) interested in what you were saying.
> That's why they don't have any kind of RBAC or MAC, just plain old DAC. You get root, you get everything - pledge and unveil won't help too much there.
> It wasn't an attack, but a genuine question, for which I provided two possible interpretations. I was (am) interested in what you were saying.
No worries at all! I wasn't taking it as an attack and apologies if my response seemed combative. I just honestly didn't understand where the point of confusion was.
> Thanks for explaining.
My pleasure! If you're still interested in discussing, I am interested in the point you made that 'They secure things from top to bottom' - if I may ask, why do you think this is the case? It's not a statement I would ever make myself.
I don't think OpenSSH's affiliation with OpenBSD really means anything; it's an accident of history that the people most likely to want to build something like OpenBSD happened to have been involved with OpenBSD at the time. Just my take.
> Many experts don't trust OpenBSD's security implementation...
Does this distrust go to the extent where confronted with a public Internet-facing, stock OpenBSD with a TLS-secured SSH connection, and a GSA hardened RHEL 8 [1] with a similar SSH configuration, they’ll pick the RHEL instance?
Many experts will pick RHEL over OpenBSD yes. It's impossible to isolate a single reason; generally you want a server to do something. (I guess if you're talking about SSH only then maybe you want it as a bastion host? In my experience people will generally use the same OS as their other servers though; the risk of misconfiguring an unfamiliar operating system outweighs any security improvement from picking one or the other)
