I've thought about doing something like this several times, but the proposition of using any tools/libraries/pam modules/etc not installed by default, custom pam/sshd configs, and generally anything "outside of the box" sort of scares me.
I've used 2FA for SSH at $lastjobatmegacorp, however all that infrastructure was supported by a team of people dedicated to such things. How finicky would setting this up for myself be in practice? (Am I being paranoid for NOT wanting to go the extra mile to ensure that I actually have access to my systems when I need it the most?)
If all you care about is your SSH key not being stolen, then you can very easily use a YubiKey (or other smartcard..) with SSH via either GPG or PKCS11. Both will mean your key never leaves your YubiKey (or other smartcard...). This doesn't prevent your colleagues from having their key stolen, but does protect yours.
I can use the YubiKey for SSH from Linux, Mac, Android phones without issue, and I keep several YubiKeys with my keys on them so it's extremely unlikely I'll loose them all at once.
Re the paranoia - if you're really the only person who can fix something, and you have lost your closest key, then - the thing in need of fixing can wait till you get home to your second key ;)
I'm less worried about losing a hardware key than I am about something breaking in a mysterious way and locking me out! My initially question was really about personal (unshared) systems. (I don't know - a random auto-update busting a non-standard PAM module and preventing it from loading, or even something just changing the PAM configs on its own, doesn't seem impossible to me). I'd definitely keep a spare if i went the Yubi route.
It's pretty hard to permanently get locked out. In a pinch, you can always mount the disk in another machine, fix whatever is wrong, and boot the original machine back up.
You could run a separate instance of say, dropbear on another port as a failsafe, until you're comfortable. Or maybe firewall it off to a set of fixed source ips.
I wrote a blog post on this recently, using only open-source tools that don't come from big corps. To have TOTP second factor on Debian (like) systems you need only libpam-oath module on the server, and perhaps an open-source app like FreeOTP (RedHat) on a smartphone.
I'm afraid to link it here because the traffic might kill my puny box.
> I'm afraid to link it here because the traffic might kill my puny box.
So what if it does?
Presumably it isn't a "critical" host so the worst that could happen is, what, no one can read what your blog posts for a little while? Eventually, the traffic will go away and things will go back to normal, yeah?
I assume that (part of) the reason for writing your blog post was so that others could read it. Yet, by not posting the link, you are ensuring that we don't -- even though pretty much all of us in this comment thread are the target audience.
Alternatively, as someone else mentioned, there are things like the Internet Archive.
(FWIW, several years ago, a page on my blog was linked by The Atlantic one morning and shortly thereafter it also hit the front page of (IIRC) both Reddit and HN. The only way I figured out that "something was up" was due to the huge number of "new follower" notifications I was getting from Twitter as I was driving to work. At the time, I was running WordPress on a little VPS with 1 CPU core and 768 MB of RAM and it handled the ~115,000 page views just fine that day -- although I did have caching and such in place. YMMV.)
I had considered that, but as someone else else pointed out, what have I got to lose if it's taken down?
At least with links to my blog I can use the access logs for an idea of which articles people find interesting (I don't do other metrics/analytics due to my philosophy).
At the moment I'm trying to build a new box to host this on but it seems Caddy has broken source builds (again) and I need plugins for my Git hooks and Hugo build.
Definitely, all traffic is readable by CloudFlare. Whereas if you have your own SSL certs everything going through the "normal internet infrastructure" is encrypted (in theory your webhost is able to get to it, but definitely not as easily - and this is only an issue if you rent a server).
On top of that there are many arguments against CloudFlare when it comes to an open and decentralized internet.
EDIT: didn't read the Philosophically in your comment. CloudFlare does many things against an open internet, for a long time they blocked (or required excessive captchas) for all TOR users.
Looks great. If it's static, try using netlify and cloudflare; I've used cloudflare with great success for years, and I just tried netlify (works great & free).
I had to do a pam config recently. Having only tweaked a few settings previously, it was kind of intimidating as I was doing some security-type stuff and had to rewrite the config. I locked myself out more times than I care to remember; thank God I was working (mostly) in virtual machines. This is especially a problem since the necessary pam settings vary from distro to distro.
If you really want to, test it on a vm that replicates your machine first and make sure you document exactly what to do to get it working. To be clear, it should be fine if you're just adding a module, but sometimes getting the exact settings can be hard, so make sure to do it in a vm first and be careful. Just trying to keep others from going through what I went through.
Megacorps have their own reasons to select technology. Your best bet is probably to go with a Yubikey (or something like it). It's about $50 and it's what several of the big Linux projects themselves use so it's well tested and documented.
- if you have an opaque enclave on the device (e.g. a black box you can't peer into), you can't know what it's doing
- if you don't, you can read the private key bits out
Your statement was that it has a private enclave you cannot extract key material out of, which resolves the latter half of the question, but not the former, I think?
I'd never done anything much with pam before but found it relatively easy to get PrivacyIDEA working on CentOS, it's even easier if you're using Ubuntu. https://www.privacyidea.org/ and Cornelius is really helpful.
Not much as far as SSH is concerned, but you can buy many smart cards and just use one cardreader. For personal use it's more convenient to use Yubikey as a commercial proprietary solution.
I guess it depends. If I’ve got a fleet of users with commodity laptops, and I want them to do CCID auth, I could buy them each a card reader and a card, or I could buy them each a yubikey and use their USB port.
Things are different if this is the DoD or somewhere that’s already got card readers as a core component, but if this is the DoD, I’m already winning because I get to use their PKI/cards and don’t have to build my own PKI :D
Wait, really? They still make these? People still use them? How secure are they really compared to USB tokens like a Yubikey? How do you interface with one of these?
Just about every Dell and Lenovo laptop from the enterprise lineup still comes with a SC reader option. In the enterprise world, smartcards are the de facto standard. I've heard complaints from $BIGCORP managers that they couldn't have iPads because they don't offer smartcard functionality.
Somehow I managed to snag one of the Thinkpad models that doesn't have an SC reader option (T420p), which is disappointing, because this seemed like a cool thing to look into and experiment with.
Huh, I guess you are technically correct. They are listed on the Yubikey product page, though, and Yubikey is so strongly associated with Yubico that the distinction is easily lost.
I think it’s pretty impossible to expect people to look at these keys and think “ah yes, this is a Yubico Security Key, not a Yubikey”, given they look like a painted yubikey, are sold by the same company, have some feature overlap, and the docs are commingled with yubikey docs.
CACs are just within the Department of Defense (+ Coast Guard). In the wider Federal government, they are called PIVs and use a slightly different standard:
The DoD uses them extensively. All military personnel are issued one and it’s used for everything from accessing workstations and email to buying your meals at the DFAC.
In some countries, every national ID card is a smartcard, so millions of people use them. You can authenticate against our IRS site with yours, for example.
Development infrastructure like Jenkins has no business being on the internet, so my preferred "second factor" is a VPN, secured with machine specific certificates that offer only VPN connectivity, but not SSH or anything else. This means even if a developer's Git or SSH key is floating around your infrastructure no one without VPN access can get at it, and if a developer loses a laptop there's a good chance you'll hear about that in time to revoke that machine's VPN certificate before anyone has a chance to use it.
That approach is a good defense in depth technique.
Network security by itself doesn't work but entirely neglecting it as some misguided articles I read recently makes me wonder if security is actually moving backwards when it comes to fundamentals and principles.
Granted those supporters are frequently cloud vendors, that dont offer mature enterprise integration. But when smaller shops follow those as best practices I'm getting worried.
Exactly. This is what I employ as one of the safeguards as well. All important resources accessible only to via a jumpbox, and jumbox itself is only accessible in a VPN.
My fear is that if the VPN connection is severed, you're locked out of your environment.
I typically have a public bastion and just shut it down. In AWS you aren't paying for it if it's turned off and you have an "escape hatch" if you need it.
If someone manages to launch another instance or start the stopped one then they would have access, but that's another can of worms.
With AWS, in an emergency you can always use the console to provision access of some sort: SSH bastion with a public IP, second VPN, etc. This definitely implies that explicit 2FA is needed for console access, but I’m pretty comfortable with using smartphones for that.
I have two dozen co-located servers to log into. So far I've resisted the idea of setting up a bastion host, because that seems like a way to lock myself out of SSH access if that machine dies. I'm also not sure of the rationale for having a bastion host, other than "big companies do it". How do other non-cloud people handle this access vs. security tradeoff?
I'm also curious what people's preferred fallback method is for preserving access to machines if you lose access to your yubikey(s), assuming you keep your private SSH key stored on one.
Really? Do any compliance standards specifically require this? Or is the specific control defined in the organization itself that requires this?
I've found many compliance standards to be pretty open-ended and function-driven, rather than prescribing specific standards. Client contracts.... they may be a different beast, and often require specific promises by vendors.
I'm pretty sure that at least some of the German financial regulations require that you can audit administrative actions on systems that are relevant for the financial transactions.
Now, you could try to implement that on the systems themselves, but how do you establish a mechanism that logs actions of an administrative user, and at the same time cannot be disabled by the same administrative user?
So instead you use a shell control box as a bastion, and regular administration of the target hosts don't have root access to the shell control box, so they cannot circumvent logging.
I haven't yet any standards that explicitly demand bastions, but it seems to be one of the standard implementations that have proven to pass audits, and so it's a pretty low-risk implementation of the logging requirements.
Easy: on each host you log into, you enter into authorized_keys your Yubikey RSA pubkey and a standard software SSH pubkey. The private key for the software keypair, you stick on a thumb drive and keep in a drawer or a safe or whatever.
Later
Sorry, I see below you're thinking about using Yubikeys in addition to keypairs. This obviously doesn't answer that question.
I'm not sure I understand what you mean. I run a one-person business and my servers are 110 miles away in Sacramento; what do I do if my yubikey breaks?
It sounds like you're trying to do something like Yubikey OTP or U2F with SSH. Both are doable, but I don't think that's the normal way people use hardware tokens with SSH; what I've seen much more is that people use Y4 keys (which are basically pocket HSMs), generate RSA keypairs with them, and configure SSH to accept the public key on the token.
This doesn't require any support on the serverside; as far as the server's concerned, the Y4 is just another RSA key.
It does require the more expensive Yubikey, but there are some security advantages to using it.
No, my thinking is along the lines of what you're describing (generate the key on the Y5 Yubikey so that it never leaves it). The discussion of one-time stuff is just as backup for scenarios where I lose the key, which I invariably will, or need to give other people "break glass in case of emergency" access to the servers.
You should already have a full backup of your (private) RSA key anyway on an encrypted USB drive in a desk drawer somewhere, so you use that until you can provision a new Yubikey.
It's key to setup two or three bastion hosts with similar diversity to your prod fleet (are you in multiple locations? put a bastion in at least two of those; if not, do you have multiple racks? put a bastion in at least two of those, etc) -- you will have network incidents where some hosts are unreachable, you don't want that to hit all your bastions or you'll be very sad.
Bastions are nice, because you can harden them, or put 2fa on them, and not worry as much about the ssh config on the other hosts -- just make sure they don't accept ssh connections from the outside world.
I’ve only ever seen this handled with VPN, and the answer there is that you’d need to launch another vpn server in the right vpc/subnet if one went down (or have failover setup).
It looks like there’s some benefits with bastion hosts around locking down commands & access in a fiber-grained way from vpn, but man what a pain in the ass. I can see a certain perspective that would really care to do things this way but VPN from my cold dead hands.
If you're using a Vault system, it's also relatively easy and well documented to use Vault for this process. There's two components, which is the OTP configuration of Vault itself:
You can get 2FA over ssh by requiring both key based and password based authentication. That is, you need both your private key and your account password to log in (in addition to your private key passphrase).
There are scenarios where 2FA makes no sense for me. These are local password managers and SSH keys.
Rethink this again: If your ssh key is compromised then you have a problem overall. For my point of view there is no real security gain in 2FA ssh key logins because your private key itself is already a secret! Only thing which is important is to set a good passphrase for your ssh key.
However I can think of one exception of 2FA on top of ssh keys which might be useful: 2FA for gaining root access or sudo commands. This might be okay but the attack vector in this scenario would be that someone can see your keystrokes or clipboard (then they most likely also have your ssh private key if they can do that).
As long as you have a passphrase and don't actually circumvent it (by e.g. an agent) you basically already have 2FA (you need access to the key (on your laptop/pc) and you need access to the passphrase (in your head)). But physical 2FA (like smartphone based ones or smartcard based ones) are nice because you can remove them from your computer or destroy them. You can't "destroy" a passphrase, so passphrase and sshkey are not really "independent".
Once just for fun I setup HAProxy in TCP mode and depending on the host name would direct the SSH to the correct host. But I had some nice ACLs so that if they weren’t in the ACL they would be sent to a honeypot.
That is true (and relatively easy to do with OpenSSH's ProxyCommand) but that was not the implication of the statement that HAProxy was being used in TCP mode to direct incoming TCP sockets to the correct backend SSH server based on the hostname using any SSH client that understands the SSH protocol.
That helps against someone stealing your computer, but not if someone hacks it and manages to get a trojan running - it'll just keylog your passphrase.
disclaimer: not a security expert of any kind. Also apologies
in advance for hijacking the thread.
I loosely remember reading on HN that wireguard could be a replacement for ssh. Is that still the case? When do you think we'd be switching away from ssh to wire guard?
SSH is a service to log in securely to get CLI access. WireGuard is an efficient VPN with low attack surface. SSH can be used to run X programs, as proxy, or VPN.
Just going to throw out there that the company I work at (JumpCloud) is a directory service that makes it super easy to setup 2FA with your SSH. Literally a couple checkboxes, and you can mandate that your whole staff has 2FA setup.
I've used 2FA for SSH at $lastjobatmegacorp, however all that infrastructure was supported by a team of people dedicated to such things. How finicky would setting this up for myself be in practice? (Am I being paranoid for NOT wanting to go the extra mile to ensure that I actually have access to my systems when I need it the most?)