I wrote about using fail2ban with postfix[0] almost 15 years ago for blocking IPs that were sending to unknown email addresses on my server. It was a godsend for blocking tons of spam back then.
fail2ban is a real pet peeve of mine because anyone security conscious enough to deploy this will have likely already mitigated any actual security risks this could help with either by using a strong password or public key authentication.
That leaves noise in the logs - which sure, it's nice to reduce, but using an alternative port can help here.
I may sound like a spoilsport - but the fact that there have been a number of security vulnerabilities (https://www.cvedetails.com/vulnerability-list/vendor_id-5567...) in this project, make it worse than security theatre, it actually increases risk whilst not at all reducing it.
Yes. At this point, fail2ban has become almost a shibboleth for people following security checklists as opposed to reasoning about a coherent threat model. This is a perennial topic on HN, and almost always devolves to some appeal to grooming logs, because of all the authentication errors fail2ban is presumably preventing.
Don't use fail2ban. (Don't use passwords, either!)
I am one of the people to whom you refer. I read about fail2ban in a "Linux Server Bible" e-book around 2010 and have used it on all of my servers since, even though I am careful with my keys and use password-less login.
Does fail2ban have authorization to write firewall rules? That's a high-impact vector of attack, should fail2ban have a vulnerability. Also, does fail2ban store credentials that provide that authorization?
While I agree fail2ban is a wrong tool to prevent password brute force - better authentication mechanisms should be used instead - it has its uses. For example, it can be used to automatically ban (or alert about) dumb http scanners like gobuster. I am not saying, a determined attacker cannot bypass it, but if it saves me some hassle and raises the bar for them, why should not I do it?
More general, some attacker actions, especially during recon, rely on making many attempts to connect, fetch an URL, resolve FQDN, etc., these could be detected and automatically responded to, making attacker’s job harder and providing extra visibility to defenders.
You shouldn’t use it because fail2ban itself can (and has been) attacked. It doesn’t make the attackers job meaningfully harder but does add complexity to your systems, that complexity is weakness.
Yeah, that’s command injection in mailutils I mentioned, not in fail2ban itself. Did you see how it’s supposed to be exploited? Did you see a real-life exploitation?
While it’s a nice trick, it’s simply not relevant. And the vulnerability before that seems to be 10 years old. I’d say it’s a decent track record.
You have that exactly backwards: if someone is hitting you with a password bruteforce from a single IP address (which is the only threat that fail2ban mitigates) then it is assuredly nothing personal at all.
A personal insult, if you are ever unfortunate enough to receive one, will be much more stealthy and neither fail2ban nor any other magical rock will protect you against it.
If anything, you're doing them a (miniscule) favor by keeping them from wasting more resources on failed login attempts. If you really hated them, you'd set up a honeypot.
More fun: setup a fail2ban actionban script that instead of banning the IP, shapes the traffic coming from it to have abysmal bandwidth so requests/responses takes really long time, so they'll have to timeout instead of getting failures.
As that gaps of inequality of our winner takes all societies ever widen, anyone just trying to do something for themselves and not for global scale SV company is just meaninglessly role playing.
Not really. Only that reading logs by human or even grepping "manually" is super inefficient where you can make a script that will for instance send you a notification when someone actually logs into your VPS.
In world where "login attempts" are basically all day reality reading logs is meaningless. In 2023 no one should be reading logs you should have alerts on events. In 1995 or something if someone was trying to brute-force your user password that was security event to look at and block IP. In 2023 someone brute-forcing is not an event, it is either wrong configuration like not using ssh authentication or not using tools that filter logs automatically and make alerts when something is actually going on.
> That leaves noise in the logs - which sure, it's nice to reduce, but using an alternative port can help here.
No, it cannot. As a sysadmin I do not want to get into user training about telling people about alternative ports and tweaking their CLI habits and any scripts that they have.
If you want to further cut down on the log noise get an IPv6 address (and drop IPv4)—good luck to anyone trying to scan a /64 for open ports.
I can cofirm this, I swapped one of my cloud VMs to ipv6 only ssh and after 11 months I never seen a single ip besides mine attempt to login. This was using the default port 22.
You can try, but a lot of ISPs assign a big subnet to each user. Mine for example assigns a /48 to each home user fiber connection.
Even if I make no effort at all to hide things and just select xxxx:xxxx:xxxx:1:: as the subnet (leaving a factor 65535 options on the table) the devices behind it will randomize the next 64 bits meaning you'll have to scan 18 quintillion (1.8e19) addresses to find one.
> That leaves noise in the logs - which sure, it's nice to reduce, but using an alternative port can help here.
Shifting services to alternate port numbers will stop very stupid scanners but it does not stop the worst offenders IME. Basically it just means you'll only get the really obnoxious sources that try everything ignoring responses.
> I may sound like a spoilsport - but the fact that there have been a number of security vulnerabilities (https://www.cvedetails.com/vulnerability-list/vendor_id-5567...) in this project, make it worse than security theatre, it actually increases risk whilst not at all reducing it.
Given the age of the project and that there's been a whopping NINE vulnerabilities found in its lifetime, this is a great take. By this same logic you better disable OpenSSH everywhere. In the same timeframe as Fail2Ban has has reported vulnerabilities, OpenSSH has had at least 60: https://www.cvedetails.com/vulnerability-list/vendor_id-97/p...
"Worse than security theatre" is quite the statement given they reported and fixed those issues in timely fashions.
If you apply the principles of defense in depth, using the network layer to deny access to misbehaving remote hosts is an obvious win on a lot of fronts and hardly qualifies as security theatre anymore than using a network firewall is security theatre.
It's not 9 vs 60, it's 9 vs 0 if you don't use it, with no loss in functionality. And GPs point is that it's not defense in depth, the vulnerabilities in Fail2Ban can compromise the security of other layers.
If we limit the use case to a single service, fail2ban is just a log cleaner. What it's detecting is merely the service that you're protecting doing its job properly. Now if you analyze the collected data and do something smarter with it, that's another story.
"Don't use fail2ban because you don't need it if you do XYZ"
I'm not so sure that's a good reason to be honest. And if you're worried about CVE's, well, you'll be using handwritten, hand delivered notes before long. Keep your systems patched, keep them tidy, none of this is likely to affect you, fail2ban or not.
But it also helps in reducing the load on your servers when, e.g., instead of 300+ login attempts per minute on your mail ports, you get 20 because the IP gets banned for a day after 2 failures. Or, instead of nginx spending 90% of its time sending out 404s for the various PHP and MySQL holes I do not have installed, it can spend 10% of its time instead.
Particularly on my small server, fail2ban is the difference between "usable" and "on the edge of falling over".
If you’re a hobbyist sysadmin setting up a personal VPS then the security risk is your own competence in correctly configuring things the better more robust less risky way, but you can’t replace yourself with a more competent sysadmin in this scenario, so fail2ban helps to Swiss cheese model this edge case.
Excuse me, if fail2ban is frowned upon, what is the alternative to block crawlers that try to find wordpress or php endpoints on my website, two software that I don't have installed?
The idea is you don't have to block those since there is no attack surface.
I look at the imap login attempts on my server sometimes. The passwords they try are usually pathetic. Nothing close to the 15+ character actual passwords we have in use.
Yup. I see many resources for self-hosting recommend fail2ban for e.g. SSH. But I always disable password-based SSH logins on all of my computers. The one niche use case I can see for fail2ban is possibly reducing the amount of hits to /wp-login.php and /cgi-bin in your web server (or reverse proxy's) access logs.
If someone is performing a denial of service attack from one I.P. address then this will help.
To tptacek's point, you've got to ask yourself is a denial of service attack in your threat model?
The reality is most folk set up fail2ban after seeing auth failures in their logs, not service degradation.
If you're considering a denial of service attack in your threat model, then I'd probably also consider a DDoS attack and there are likely more effective solutions here (a firewall or CDN).
And don't forget you're using some of those precious CPU cycles to parse the auth logs, with python no less :-)
You can also literally have anything pipe rules into it. Want WordPress auth to result in fail2ban-enforced bans? You can do that. Want cheap rate limiting? You can do that too
This thread seems pretty full of people dismissing the project based on the idea that it only protects against ssh credential stuffing, and ignoring the other 99.5% of what F2B does.
Absolutely agree. Fail2ban, sooner or later, bans you from your services becasue something in the configuration went wrong.
It does not protect against anything serious: you must have proper credentials/MFA or certificates and therefore bots can check as much as they want.
There is no protection against DoS either.
And I agree about moving the port - I only see a tiny activity in my logs coming from bots when my ssh port moved away. Obviously 443 is there to stay (this is a public service) so I will get whatever comes.
One nice thing I'll say about fail2ban is that it can fire off reports with decent logs to the networks responsible alerting them to compromised systems and bad actors.
I agree that almost all use cases of fail2ban are little more than feel-good exercises.
Failed login attempts (the noise) are not where bad things happen. What we should be concerned with is if the attempt succeeds but is not from a legitimate user. fail2ban is no help there.
Having said that it might be a decent way to collect IPs. At one point I was distributing the collected IPs from VMs and blocking them for the whole network. fail2ban does provide mechanisms to do this.
At a former employer I hooked up a random Debian box to the corp network. Of course I ran fail2ban, in addition to a host of other lockdown measures. Without fail every morning at the same time some random corp scanner would start hammering on the ports, and fail2ban would kick in and block the scanner. After the default expiration time the rules blocking the scanner would expire, and it would happen again the next morning. This probably continued going on for several weeks or perhaps even months after I left the company.
You would get seriously yelled at by my team if you did this in my company.
The corporate scanner are not there for fun or because security loves to do scans. They are there exactly because they are random wannabe sysadmins who will plug in a box that ends up compromised.
A scanner may help to locate it and possibly warn about the issue. And only maybe.
Do not make life difficult for people who already have they hands full of constant vulnerabilities, fighting corporate crap and being the bad guys because they audit stuff.
I was first exposed to fail2ban by some PBX server suite I was setting up. Totally cannot remember the name of it, but it checked to see if the fail2ban service was running. If it was not the PBX service would not start and throw a are you insane error message. Saved a lot of new kids like me at the time who were just fooling around I bet.
In security circles RDP is called “ransomware deployment protocol” - basically one should never expose it to the internet.
Make yourself a vpn box or some Linux with ssh and do port forwarding and allow RDP from that Linux host. OpenVpn or ssh are much better to be exposed to the internet.
I made ts_block. It solved a particular problem I had at a particular time. Nobody should be doing RDP across the Internet w/o a VPN but, apparently, people still do it.
Also be careful that you've used CSRF and other origin protections. Otherwise competition can DoS your legit users by rapidly cross posting via js from their own site. Not likely in enterprise world but could happen in less than ethical industries.
Every time I spin up a box by hand (which isn't often), I'll set up a sudo user with my SSH key, then drop out of the root shell and log back in as the new user. Only then will I disable root login and password auth over SSH and start setting up fail2ban and the like.
If I lose access by, say, switching to a new computer and losing my SSH key, then I'm more or less dead in the water. But it's a small price to pay and 1password supports SSH keys first-class now.
at that point, why not just do FIDO2 with a YubiKey or something? eg https://www.ajfriesen.com/yubikey-ssh-key/ as long as you don't lose the YubiKey or the backup YubiKeys you're good.
I use it with ssh and password auth disabled, is there a reason not to? Might be overkill but the host is in my home so physical access if I ever get locked out is not an issue.
I think this is still reasonable, attackers may have a database of leaked keys (e.g. if you ever accidentally commited to GitHub, or ever ran a malicious script which uploaded it), which they then try on random servers.
There is a way to use fail2ban together with nginx's rate limiting to implement a temporary ban on addresses that are repeatedly abusive. As far as I know this is one of the least friction ways to achieve that.
The limit_req directive works by dropping the responses that exceed the limit but does nothing more than that. Fail2ban can be scanning the nginx log files for instances of someone exceeding the limit, and add them to e.g. a Redis set. You can then use a Lua plugin to track incoming request addresses and check whether they belong to the "banned" set (then you can drop the connection fully).
For SSH I prefer blocklistd[0], which sadly is only available on FreeBSD and NetBSD AFAIK, it's a much simpler approach though requires a small patch to any daemons that want support.
Another couple fun things for SSH are iptables string module [1] and just adding a really long version addendum. [2]
Silly examples but the bots get confused. I admit these are not efficient rules. Try it out on that domain. Send some SSH bots my way, I honestly have no idea what it looks like on the bot-end-of-things. I would estimate about 95% of bots are libssh. About 4% are Go and the rest are an odd mix. That string can be spoofed but I think most people are too lazy.
I forgot to mention, if the IPtables examples are useful they can make things ever more quiet if instead of "-j DROP" one uses "-j SET options" to add to an ipset, then use that same ipset in an outbound rule to tcp-reset the openssh socket to free up memory faster. Not really a big deal unless one gets flooded but probably useful to someone some day. Maybe I will do a little write-up on this.
Changing the port reduces the traffic, yep, but definitely doesn't get rid of it - all my servers use non-standard ports for ssh and there's still ~10 login attempts a minute.
2244 ? I used to get almost no tries on 2244 which is kind of an obvious choice. Nowadays I have wireguard and no ssh ports exposed as they run on private IPs.
Don’t ban hosts or any service account automatically. Use conditional accesss, mga for hosts with credentials + IP or client cert. Auto raise a snow ticket for the support team and an alert in your SIEM for your Soc. If you want more detail I’ll put my blog in my profile.
Jump hosts are a well-tested and widely-deployed pattern already that basically do the same thing without the downside of lockout problems from connecting from somewhere new.
I was recently thinking on a way to open up my Jellyfin Media Server, so I can stream music from my personal collection. Jumpbox would easily incur a lot of traffic costs... due to all of the media being very large(many users of jellyfin suggest that). I also would like to allow friends access to my library, without them having to connect to my VPN.
Having had a very bad experience with getting my home network once DDoSed and once hacked - I am at a stage where a reliable third party source, that can tell me what IP to whitelist, is the only reasonable solution I can come up with. (I know it's not a common use case, but it is my use case)
I like occasional discussions on these things. People sharing tips, experiences, pitfalls, etc. It gets everyone in one place at around the same time to do it, like a small digital convention.
I would guess the subtext here is related to the 23andme breach.
The problem with fail2ban is that an attacker who has a botnet of significant size at their disposal won't even be slowed down. Not saying it's worthless, but it's not a silver bullet.
Not saying fail2ban is in any way state-of-the-art security or even best-practice at this point, but if they're burning an IP after a few bad attempts, that's going to be more effective than letting them try tens of thousands per IP, no? Security in layers of course, so even a few [dozen, hundred] failed SSH attempts should probably raise an alarm or put things into an elevated security mode automatically...but also don't leave SSH open to the public, especially not on :22.
> but also don't leave SSH open to the public, especially not on :22
100%. These days, at least if you're working out of a cloud provider, there's no excuse for exposing SSH to the world on any port. AWS/GCP/Azure all have different tools to allow you to run bastion-type services without internet-facing SSH.
Firstly, the GDPR doesn't care about PII - that's an American legal term and the GDPR is an EU law. It deals with processing personal data, and the two are not the same (although there is overlap, with PD having a more expansive definition).
Secondly, consent is only one of six legal bases for processing personal data. The other five are that:
- processing is necessary for the performance of a contract to which the data subject is party or in order to take steps at the request of the data subject prior to entering into a contract;
- processing is necessary for compliance with a legal obligation to which the controller is subject;
- processing is necessary in order to protect the vital interests of the data subject or of another natural person;
- processing is necessary for the performance of a task carried out in the public interest or in the exercise of official authority vested in the controller; and
- processing is necessary for the purposes of the legitimate interests pursued by the controller or by a third party, except where such interests are overridden by the interests or fundamental rights and freedoms of the data subject which require protection of personal data.
The last of those - legitimate interests - is clearly appropriate, the rights of the data subjects (i.e. the IP addresses' users) don't override the legitimate interests of securing one's machines, especially given the limited processing performed by tools like Fail2Ban.
Since you're also obligated to secure said data storing the IPs from brute force attempts is not only fine but almost a requirement. You're not making the connection to any individual and the use-case is legitimate.
[0]: https://www.dp.cx/blog/postfix---fail2ban---win.html