The source of the wired piece was posted 6 days ago here[0] along with the bleeping computer analysis (better than wired's, IMO)[1]. It was also posted again 3 days ago [2], when I also posted it until I noticed it was a dupe (the dupe detector doesn't seem to work on the guardicore URL).
I've been watching this specific botnet since April, almost 6k attempts/server so far.. it's slow and wide (rarely repeat IPs, max rate was 2/minute, but it's more like 8-20/day now) so most detection approaches won't work (search your logs of sshd for "Received disconnect from", "Did not receive identification string from" and "Connection closed by"). It's also everywhere.. a lot of digital ocean, but also AWS, google, Azure. Government IPs, consumer IPs, IPs reported as part of backbones
Are you sure? Bleeping Computer (BC) has a publish time of 2020-08-19T06:00:00-04:00, while Ars Technica has a publish time of 2020-08-19T13:00:16Z.. looks like a 3 hour lead.
Interestingly Guardicore reports 2020-08-19T09:50:36+00:00 suggesting BC (a) can produce a quality article in <9 minutes, (b) there was a prior article (c) BC got a heads-up
> search your logs of sshd for "Received disconnect from", "Did not receive identification string from" ...
This is not reliable advice; many SSH port monitoring solutions (including where I work) will do half-connects like this to determine network availability, leaving these log entries by the thousands.
Half connections to determine availability/existence of SSH is exactly what botnets/nefarious actors do. If you know why you have incomplete connections in your logs, then you can filter/ignore them.. those should only be from known IP blocks which you can filter out, but it's unwise to write all off that traffic as "known"
I agree. My comment was clarifying that your blanket statement implied "if this is seen, it is a problem" and I was providing contradictory evidence that no, it does not always indicate a problem. I personally have been known to telnet to ip:22 while debugging "why doesn't this work"; half-connects are just a part of life, used for both good and evil.
Lots of suggestions to place SSH behind other protocols or obfuscate the port number here. Let's be clear about one thing, OpenSSH is the least insecure service you are running. A lot of talented people have audited that code many times over. It may show its age but it is an order of magnitude simpler than SSL. Every pre-auth process is sandboxed.
Most breaches are caused by bad passwords. Use keys, or enforce good passwords, before doing anything else. That VPN might well be less secure, and port knocking is just an unusual authentication with a really short password.
Port knocking is not really “just a bad password”, as long as you are using it in conjunction with another secure service like SSH.
However, you can get the same effect as port knocking with none of the drawbacks using single packet authorization like fwknop if you don’t want to go full VPN.
Running SSH on a non-standard port alone isn't worth much, but it's defense in depth. You don't have to not do all the other things if you run SSH on a non-standard port.
One really valuable thing for me about running SSH on a non-standard port is that it dramatically decreases the noise in my logs. That's not irrelevant!
This is why you should do one or more of the following at the very least.
- Always use keys and disable password based login
- Do not have ssh publicly exposed, or at the very least only via a bastion server, but if that bastion server is publicly available, then you are still quite vulnerable (perhaps even more so if someone gets into your bastion and is able to ssh to all the things if you are using key-based logins from the bastion)
- Always use a passphrase for your keys (helps with the “if someone breaks into your bastion scenario”)... but the best thing you can do is just don’t expose ssh to the public network
I’m curious why people say not to have ssh exposed if you only have key auth enabled. It seems to me that every single other password + some other number of protections is inherently weaker than simple old public key auth.
If you do expose your SSH port, one option is to setup a firewall rule to allow your own public IP for connections to the SSH port and drop all the rest.
If you're using Digital Ocean, you can set a firewall rule via the droplet web interface, that's quick to do. Otherwise just a simple iptables does the trick.
As long as you have a static IP that could never change, otherwise you'll deny yourself access after, for example a power outage where your router gets a new IP (or a random period of time where your ISP decides to change your IP, or you move, or you want to connect from another server).
Using an alternative connection to activate the protocol is probably the only reasonable defense:
* An https service that can enable a port for connections from an IP for a limited time (which has some security implications too since it would need root access, or to trigger something with root access)
* Something built into KLO hardware, or software provided by your provider (including something simple like turn on the firewall a few minutes after boot, and using the reboot trigger and connecting in that brief window.. as long as you don't mind your server down - probably ok as an emergency recovery strategy)
Variable port knocking ie a different sequence tied to rules can help expose compromised networks making up the internet.
Say you port knock from your mobile phone, use one port knocking sequence, if port knocking from a different internet connection use a different port knocking sequence. This can help highlight those networks with taps, but the identity of who is behind the tap can still remain a mystery, unless you set other traps. You also need to trust your devices, which no one can legally do as copyright prevents people from examining the code on their chips and some OS'es like Windows. NSA still provide the tools for examining code for free https://www.nsa.gov/resources/everyone/ghidra/ but as always Resource Burning is something to take seriously when trying to secure your systems. Sometimes its best to view servers as disposable, so automation can be your friend even simple PXE boots can be useful. Its also worth noting you can run at least two different ADSL connections down the same copper wire, they do here in the UK at least, most people dont know this, but the TV & film streaming services use this, some ISP supplied routers can give this away if you examine the backup config files.
I’m not sure I trust myself to set up my own bastion securely. Are there any companies that offer secure bastions as a service?
What I’m envisaging is: I pay somebody, make an account on their site and add 2FA, and then they give me a server with a static IP and handle the auth. Then all I have to do is to whitelist that static IP.
Ideally it would function as both an HTTP proxy and an SSH proxy; like a ‘secure web portal’
Provided you are not behind CGNAT then dynamic DNS is your way forwards here. My parents have a RPi for a phone exchange (RasPBX) and a pair of Yealink DECT handsets and BT Home for an ISP. A static IP is impossible.
I tie the Pi to my home router via OpenVPN and use https://freedns.afraid.org/ for dynamic DNS to send calls to. It has worked fine for four years now. The Pi runs a one liner from a cronjob every 15 mins or so. The TTL for the DNS entry is very short and the DHCP lease is something like 24 hours plus also I think it tends to renew the last one used reasonably reliably. I don't bother monitoring that. I could write the cronjob result to a local log I suppose.
More things can go wrong than the tiny window that might exist when the DNS might be out of whack. Don't let the lack of a static IP address get in the way.
Could you comment on my way of accessing my servers on AWS:
My server are on VPC and no public ssh is allowed (nothing is open besides 443). When I want to login to my server , I run a script to modify security. (I have disabled ssh password logins and ssh is running on non standard port). And when I am done with task, I plug the hole in security group with another batch code. I am worried that I may be not exposing my servers with my practices.
No op, but your practice is probably fine. More ideally, SSH would be locked down to your public IP address only. If you're already scripting this, you could just do a curl request to one of the many "what is my ip" services and set that instead of 0.0.0.0/0 as the security group IP address.
We also use bastion hosts. They're whitelisted for key-based SSH only access for our public IP addresses. We can jump around to other servers once we login to that one, but our externally facing servers never advertise anything other their base service (:443 or whatever is running on them).
I also use them with my personal servers and just leave them shut off when I'm not actively using them. This keeps costs down, but more importantly it permanently blocks off SSH access unless I'm specifically in need of it.
This particular malware doesn't seem to grok when you're using certs or keys, so while it'll never be successful it keeps trying (effectively a resource DOS).
Obscure port does nothing.. it eventually finds it and then you'll see tons of servers try that port (unless you constantly move it).
I'm not sure how you imagine Pluggable Authentication Modules would help? Fail2Ban or some sort of active IPS helps, but because the IPs are very varied (and presumably ever increasing) and infrequently re-attempt it doesn't help much.
Note anyone running exposed SSH without keys or certs, should run the detection script[0] (which is just shell and you should read first before installing) provided by Guardicore
For my hobby servers, I listen on a non standard port. Not once in 20+ years has anything hit that port. Even when I've had domains / sites that ruffled feathers, no hits. It most certainly cuts out noise in logs and stops all the bots.
A few months ago I started noticing failed login attempts on my non standard SSH port (> port 1024) so I moved it up a few port numbers. About 2-3 weeks ago, I noticed that it was getting hit again by failed login attempts. I then changed the port to the original and immediately started getting login attempts again... it kind of spooked me.
I thought about using fail2ban but every login is a different IP (using my eyeballs, I didn't parse the logs).
You can still use fail2ban to block unique IP addresses if you use some supporting scripting. There's an example ([1]), which will check the IP addresses by country code. If they're all the same country then you can just block the whole country.
You do not need to use firewalld, either, although this does. See the second link for something generic ([2]).
I take a different approach than fail2ban. My sftp servers use a standard port 22 and any time people try to log in, I create an account for them automatically via a cron job in the sftp-only group and a null password. The bots will spend years trying to log in repeatedly every few minutes. I have yet to see them upload anything interesting. Many years ago, bots would upload malware, then try to browse to it on port 80. But no more... These bots just want to get a shell and install malware / c&c tools. Some of them try port forwarding, but I have that restricted for the sftp users.
I've also used a port other than 22 for sshd. And I can confirm that the past month I've been getting hit with lots of failed login attempts on ssh. My basic look at the logs shows it trying random dictionary words for usernames. And every connection attempt comes from a different IP.
Yes, I'm seeing the same to my non-standard port and the connection attempts are from a large number of different addresses (and only usually 1 or 2 attempts from each) that Fail2Ban does not trigger.
Guardicore has a shell script (detect_fritzfrog.sh) which you can run on your machine [0], it's short (28 lines) and sweet - you can read the source and see what it's up to.
Every bot has a list of peers and their SSH credentials. This way, peers can reinfect machines that were restarted, thus allowing the bot to be volatile on the infected machine.
The article says the researchers can join the peer-to-peer network. The researchers should be able to get a list of all infected machines including SSH credentials. These credentials could be used to remove the backdoor SSH key, kill the bot & netcat processes and maybe change the SSH password on all infected machines at the same time.
You leave a small enticing ssh listener which is actually a Python or similar daemon process that fakes its output and logs usernames and password attempts. Obviously that thing is on its own DMZ with no outbound access.
You can harvest the latest crop of usernames and passwords that the baddies are using with a little bit of effort, but not much! You can also slow them down a bit with a sprinkle of tarpitting. Take that list and use it as a password blacklist for your own systems. You might allow the list to decay by say six months after the last sighting to avoid it getting too big. Actually you could allow decay with a normally distributed six month spread, centred over one year to avoid predictability. That may be going too far - once a password is on the list, I suggest you keep it.
Usernames that are not simply given names is a very good idea. Service accounts should not be able to login interactively if they don't need it (set your RDP groups up properly, PLEASE!) Give service accounts odd names and sha1sum style passwords. Administrator, Administrateur, Administrador etc should not be able to login at all to anything and nor should root. Those four names come up in my logs so often, it isn't funny. If you've got an account called sql or ldap or mail or similar, then get rid of it. Please.
Is it possible for OpenSSH to block connections based on SSH client version?
I've basically shut down brute force attempts immediately on connection by blocking out 'libssh' and 'SSH-2.0-Go' using Bitvise SSH Server (for Windows, which is free for personal use).
I've been watching this specific botnet since April, almost 6k attempts/server so far.. it's slow and wide (rarely repeat IPs, max rate was 2/minute, but it's more like 8-20/day now) so most detection approaches won't work (search your logs of sshd for "Received disconnect from", "Did not receive identification string from" and "Connection closed by"). It's also everywhere.. a lot of digital ocean, but also AWS, google, Azure. Government IPs, consumer IPs, IPs reported as part of backbones
[0]: https://news.ycombinator.com/item?id=24217592
[1]: https://news.ycombinator.com/item?id=24208873
[2]: https://news.ycombinator.com/item?id=24245824