Considering that the code in a GitHub organization or in a repo with collaborators is only as safe as the weakest link in the chain of users that have access to it, it would be nice if the admin of that org/repo could set a policy that all members need to have 2-factor authentication activated before they get full access to the codebase. Enforcing strong and unique passwords is hard, but enforcing 2-factor is easy and the bump in added security is high. At least I would sleep better knowing that, for an attacker to access the codebase, brute forcing would not be possible.
Let's start a global registry where everyone registers the password you use at every site. So we can cross reference and check that you (or anyone else) hasn't used that password elsewhere. OF course we will store the password in plain text.
We try to make sure it is by enforcing password expiry, no password reuse and complex passwords. We also have a massive shitlist of passwords from various leaks.
That's a lot safer than using a weak password. I don't have a citation handy but security researchers often support writing down passwords to encourage using stronger and more frequently changed passwords.
What's the problem with writing it down? I periodically generate a strong random new password, write it down and put it in my wallet with my credit cards, until I type it enough times that I learned it.
Writing it down and leaving it exposed in a public place, like taped to your monitor - bad. Writing it down and treating it like any other important asset - not that bad.
Passwords are HARD to remember but it's better to have it under hand than to ask for a new one every two days because we didn't have time to remember it (my school stored students password in clear text in their DB and gave them out printed on paper sheets. Four years later, I still have this sheet in my wallet, my three passwords never got compromised and I memorized each of them after less than two weeks of heavy usage).
> IT professionals usually run screaming for the hills if you talk about writing a password down
I can understand that.
You and I keep our passwords in our wallets. Some employees keep them on post-its taped to their computer screen at work.
If a password I keep in my wallet gets compromised because it's been stolen with the wallet, I have more pressing matters to deal with than "oh crap, a random guy who stole me my wallet in the subway may use it to access our repo and add bad code in my name".
The point about "not that bad" is that security is an illusion: you can use a thousand passwords everywhere with different set of keys for github, server A and server B, change password every week and never write it down, plus a little bit of tin foil hat paranoia,... But in the end, a $5 wrench will make you spit out all the access keys needed by someone who REALLY wants an access.
Actually, that's very nearly the approach I'm taking myself, so fair point (just that IT professionals usually run screaming for the hills if you talk about writing a password down). However, it also sounds like you're using just one password for everything - that seems very unwise.
I think two-factor auth should be the new basic auth. Every site should provide it, either directly or through an integration with a trusted OAuth provider.
I'm waiting for the day that the security people work out a solid federated single-sign-on system that gains widespread adoption.
How is this possible? I would expect repeated failed login attempts would force a timer making a brute force attack impossible? Or at least more difficult. Is that not the case or am I missing a key piece of info? (Serious question, not trying to troll or be smug.)
"While we aggressively rate-limit login attempts and passwords are stored properly, this incident has involved the use of nearly 40K unique IP addresses. These addresses were used to slowly brute force weak passwords or passwords used on multiple sites. We are working on additional rate-limiting measures to address this. "
This is one of the more insidious scraping techniques. At Blekko if we see the same exact same query coming in from a bunch of different addresses, differing only by which page of the results they want we temporarily ban the entire lot of them.
I'm sure a similar situation would work here, especially with a decent geoip database. If you get 3 - 5 failed attempts on an account separated by more than a few miles (and not along an interstate :-)) you can probably assume someone is using a botnet to try to break in.
Before I switched my sshd ports a few weeks ago, I often saw break in attempts on my servers that would imply checking only a single username and a single password.
My guess on that is that bruteforcers, rather than testing thousands of usernames with thousand of passwords on a single host will try a single username and single password on thousands of hosts.
I double checked my log archives, it seems not to be exactly the same, though. In what is reported here, a same username will be tried with several password, allowing something like a minute to pass to make a new attempt.
In my logs, this is really just a single attempt. Other break in attempt may happen something like two days after, but with an other IP and username.
It may just be yet an other evolution of the same botnet knowing its previous pattern has been spotted, though.
(and in case you wonder, yes, I always have the nose in my logs, having a dedicated monitor for that :) )
As with IPv4, DoS attacks which tend to be clustered from specific locations (I've seen them originating from multiple hosts within a hosting center's range, or from a particularly poorly managed ISP's allocation), using broader allocations (CIDR blocks and ASNs) to identify responsible networks can be useful. The challenge in general is translating from IP addresses to the administrative block responsible (there are IP reverse lookups which do this, in some cases the zonefiles are downloadable allowing lookups at local network speeds), and in translating the information to some sort of policy (few network management devices offer AS blocking rules, though CIDR is pretty much native).
It is probable that a single machine gets a /64 ipv6 range, but the first 64 bits are consistent.
When ipv6 gets more common it will probably be more acceptable to block an entire /64 if you suspect a brute force attack.
I've seen this mentioned before - using a tremendous number of IP addresses across a tremendous number of sites and over a massive length of time to perform a very low-grade incessant brute-force attack. It means that any weak password anywhere is always vulnerable because everybody is always under low-grade brute force attack.
Actually not. I'm the maintainer of a popular Open Source project that is hosted on Github. If someone stole my credentials, they could replace the current release with a binary containing a Trojan.
Looking at the security history page I see a lot of failed login attempts. Makes me glad I enabled 2-factor-authentication!
It's a great way to distribute Open Source software. (Previously Mattt hosted it on a personal Amazon S3 account which he paid out of his own pocket; now bandwidth is generously paid for by Github)
I don't know about you, but I seldom read through all the source code of the open source/free software I use. Yeah, even when I actually compile it myself.
If someone would slip in rogue code - it's quite likely some to many would actually run it and deploy it. Especially if it's a fast moving piece of software - like being so rapidly developed that distribution packages can't keep up for either time or stability reasons, leading people to compiling/deploying from source themselves.
Many large companies use them as a SaaS dvcs provider for private, closed source repositories. They provide a great tool for an agile workflow at a good price, what more is there to say?
Obviously I have nothing to go on for this, but just guessing, it would seem to make sense that:
- Of all users, only a small subset would have private repos
- Of those users, only a small subset would have private repos that would be of interest to third parties
- Of those users, only a small subset would have a weak enough password to allow brute force
To reverse it, of these accounts that were hacked, I can't see many of them having private repos that would be of interest.
And if so, then this would seem a bit excessive.
I know that's a lot of ifs, but it seems reasonable. I would be interested to see the number of total accounts vs. the number of accounts w/ private repos.
- That private repos are the only thing worth targeting. What if you could inject a trojan into a popular open source project? You could do a lot of damage that way, probably way more than on private repo, because so many people incorporate them in their products. Imagine they hacked the Rails repo, for example. Worse, some repos host binaries, for which a meddling would be harder to detect (a bad idea, but doesn't mean it doesn't happen).
- That the users being attacked are random and not specifically targeted based on who the user is and what the work on. Not sure if that's the case or not, but I see no reason to assume it.
People often reuse user/password combinations. So if attackers can find working combinations here, those are combinations that they can try elsewhere as well.
Rate-limiting based purely on username/password is a great way for me to lock other people out of their accounts (Or force them to do a password reset, or wait for a timeout) by just failing to login to theirs a few times.
The way around this problem is to add source IP into the rate-limiting algorithm so that I cannot mess someone else up.
Unfortunately that can be worked-around somewhat by using 40k different source IP addresses.
No perfect answer. Personally I think GH did ok here.
TL;DR - compute the average number of system-wide failed password attempts, and if it's over the norm, impose small delay on all users (except those that login via a persistent login cookie).
Rate limiting is not the only thing they could have done. They could also notify users of failed attempts and ask them to confirm it was their attempt (once they were logged in of course) - the offending IPs could then be blacklisted.
Of course, this works only if the number of attacks is small enough.
I think the point is that with so many source IPs, blacklisting techniques are pointless, and given the duration of the attack, timeouts are useless too. What is better is a IP whitelist coupled with a notification if a certain number of logins fail from IPs outside of that whitelist, regardless of the time inbetween. Users with strong passwords could disable this notification but it makes others aware that they are under this attack (possibly).
An even more secure approach is what Linode does. If you log in from a non-whitelisted IP, they require you to whitelist the IP via an emailed link, then you can login.
But that's overkill and the first one is a lot better for casual websites. You store the user's last login IP and you wait for maybe 1024 failed attempts not from that IP, which ought to be enough to suggest a bruteforce attack. You're protecting "bluedog4", not "password" or "1234" which are going to get cracked anyway.
Two-Factor Authentication is great but even without, it would seem that they could more aggressively lock out any new IP address which is failing password retries several times. (but continue allowing log-in from trusted IP's.)
IPs aren't stable. For instance, the legacy SSID broadcast all across my university assigns you a new (usually different) fully routable public IP every time you authenticate through the captive portal. Some random person enjoys the "trustedness" of your IP at the time of login; you never see it again.
There is a new SSID with WPA2-EAP replacing the captive portal and the usual DHCP/NAT business replacing our massive waste of IPv4 space, but that was implemented last year. This introduces the opposite problem: one person out of thousands can't remember their GitHub password so GitHub mysteriously drops all traffic from everyone on campus.
IP addresses don't identify people, computers, or usefully discrete units of any kind.
While that seems like a logical solution arrived at through a pragmatic approach, it's toeing the line of a really bad practice. IP addresses are not, and should not be, identifiers like this, especially when you're talking about a whitelist. What happens if the ip address changes (happens all the time for the majority of US internet users who have dynamic IP's)? What about if they want to work at a coffee shop?
While it might be possible to do all this correctly, it's just so easy to do wrong that it's hardly worth it. Given that there are existing better ways to do it, it's the kind of thing you probably don't even want to mess with.
I suppose if you see repeated failures across a range of IP addresses you could temporarily block the entire account and send a reactivation email to the registered email address.
That would prevent a practical joker at work from locking you out unless he went to a lot of extra trouble.
At some point I'm starting to see the appeal of completely abandoning passwords in favour of email or SMS-based token creation.
You attempt a login, they send an out-of-band request for confirmation. You do the confirm, and they let you in. Like a superfast streamlined version of the "reset password" email.
The existing 2 factor auth basically does this, without the requirement of SMS, internet, or any connectivity really in general. The OTP tokens are based on time or a counter, and a secret key.
Considering Github is frequently the target of DDoS attacks, I would assume there are people willing to go to a fair bit of trouble to lock you out of your account.
The article did mention the use of 40k unique IP addresses, so I'm guessing their rate limiting is based on IP address and not the username or IP+username combination.
Maybe they're doing this now that it's a real problem?
At any rate, don't store really sensitive stuff on Github. It's a bad idea for many reasons, security flaws being one. In particular, keep in mind things like AWS server credentials which might go in your repository.
One could attempt to log in from multiple IPs over a long span of time, let's say weeks. This scenario is difficult to trace using timers. Alternatively, sending email alerts on each failed log in attempt may not very optimal and generate a lot of false positives - person A (not an intruder) randomly tries logging in by impersonating multiple github users.
IMO something similar to SSH security policies like allowing logging in from a set of IPs and/or without password i.e. using public key, may be a good idea.
I have a single failed login from an IP address in Brazil 16 hours ago. And another single failed attempt from an address in Thailand three days ago. No suspicious activity had been logged before that.
The attacker probably tried a password like 'github' on the first pass and 'password' on the second pass.
You may not be trying to troll, but either you didn't bother reading the article or they added details since you read it (doesn't seem likely). They answer this near the end of the not-very-long article.
Hrm, so they do! I swear I read the article a second time and even re-skimmed looking for that detail before making my post. My brain really, really wanted to not read that next to last paragraph. Stupid brain.
You can also force a timer on a per-account basis - e.g. twenty failed logins for one account from five different IP addresses -> set a timer on those IP addresses and a timer on that account. It's not simple, but it would help mitigate a mass-brute-force effort.
But then it's trivial to lock someone out of their account by trying to login as them a bunch of times. That's why they'd key it based on ip and username, not ip alone.
Interesting, my account's Security History shows a few failed login attempts in the last 3 days from Eastern Europe and Southeast Asia, and my account is not even popular.
I had 13 login attempts from South America, Asia and Middle East. My login details have been leaked in several leaks, latest being the Adobe leak. And even though I've had strong password, my fault's been that I've used the same password in lots of places.
Luckily the Adobe leak was the final straw and I finally started to use a good password manager and super strong, distinct passwords in all the services I'm using. It wasn't that bad at all, just selecting a good password manager[1], strong encryption and an easy way to sync your passwords between devices (git), I've had no trouble at all with strong passwords.
Although it would be nice to have an Android app for the password manager. I think it's a nice excersize for writing my first Android app at some point...
Your comment prodded me to check my own Security History, and I have five failed login attempts within the past 3 days from unrecognized IPs also on a not-popular account.
I've seen a small amount of suspicious activity (possibly 3 failed logins that weren't mine); I'm wondering how relevant various attributes of the username are. I'm starting to move to unique usernames for various services: not anything vastly secure, but something that's at least harder to automatically cross-reference. Does anyone know if using much longer usernames is a worthwhile investment?
They did the OAuth token with the Ripple give away 16 days ago, and before that, I had no failed attempts, just a single successful attempt. I too used the same password for a few sites.
I wonder why Github didn't publicly post the details behind the attack -- namely, that is was an attempt to game a Ripple (cryptocurrency) giveaway. They mentioned it in their "heads up Github user, your account was compromised" emails, apparently: https://news.ycombinator.com/item?id=6766293
The attack must have been done over Tor. My account was flagged and locked but it's not reasonable to say the password was cracked. However, I do my day-to-day browsing over Tor and the matching IP addresses caused a false positive.
Its worth mentioning, though, that the rate limiting is at the account level and not the IP level. When I was trying to get access to my account before the email was sent out, my account was locked and remained locked even after developing a new Tor circuit. How were the attackers able to circumvent the aggressive rate limiting? Three password attempts a minute is a generous estimate and it would take longer than the expected life of the universe to crack even a moderately secure password at that rate (my old password had 704 decillion permutations.) Perhaps some part of GitHub isn't being rate limited?
I only have Tor logins recorded and nothing indicates that my account was compromised, so one of the exit nodes I used must also have been used by the attackers. I guess it doesn't necessarily mean that the entire attack was done over Tor. Perhaps a zombie in the botnet was running it. :)
If these attackers had tried targeted brute-force on some limited accounts, they might have been successful without tripping over the GitHub security team. Nonetheless, kudos to GitHub for handling this perfectly.
I'm very impressed by GitHub's response - what a prime example of how you should handle an intrusion attempt! 11/10 for style from an IT governance and disclosure point of view. They've successfully: identified the nature of the attack, notified customers in a polite manner, publicly disclosed the goings on, and exercised added paranoia (which in this case == security) by disabling related accounts. Kudos! The only thing they missed is disclosing that they're working with local law enforcement - something I'd presume they're doing.
More interesting is the attempt. I frequently scan customer systems on their behalf, by spinning up clouds of resources to hammer their login APIs. By using Digital Ocean $0.05/hour boxes, and massive threading, tests tend to cost me ~$50/hour (yes, that's 1000 boxes orchestrated).
Given the real world cost of cloud computing, and the ease of libraries such as libCloud (or ruby's: FOG), the cost of doing this in terms of time and real money has dropped dramatically.
I wish the weak password blacklist was public, so we can all implement password check like this easily, and maybe get rid of them forever (even if banning "chicken" as password is only a way to have "egg" popping up).
Looking at my security log, it looks like they gained access over two weeks ago. No failed login attempts, just a successful login. No brute-forcing about it.
oauth_access.create: GitHub XRP Giveaway - 16 days ago
user.login: Originated from 23.29.121.166 - 16 days ago
With a feature like that, if an IP address from country C attempts to log in to accounts of users in separate countries A and B, they are quickly spotted.
Gmail does something a bit more sophisticated than this but it's a real pain trying to check your mail from the airport in another country when your phone doesn't work either to confirm that way. Also annoying for VPN users who may appear to be in another country.
What was the purpose? Were the repositories private? Was the attack targeted toward known accounts working on new projects -- i.e., industrial espionage?
I'm going to guess it was simple spray-and-pray. (My account took its fair share of compromise attempts, too, probably because I use the same username everywhere so it's going to be in many modern dev-heavy password dumps.)
Github is, for better or worse, a high-value target these days. Compromise of a Github account will often get you credentials which you can use to compromise high-value targets, for example production machines.
(Example from the Rails world: if someone gets read access to your git repository, expect them to figure out your secret token which you use for HMACing sessions to prevent forgery. If an attacker can forge sessions, they have remote code execution on your production servers. You can assume they will go directly from code execution to root on those servers, and from root on those servers to a systemic compromise of your entire network (if they want to do that).)
Use your imagination on what a mass compromise of accounts gets you. The simplest possible example is grepping for /bitcoin/, finding the full source code and credentials for a Bitcoin exchange, and rooting them then emptying the hot wallet. That results in fairly obvious economic advantage, right? You could also use scriptable attacks to root thousands of big-n-beefy production machines on fairly trusted IP addresses, then add them to a botnet, which you'd use for spamming or various other nefarious activities.
Then, even farther down the list, you have a dedicated adversary actually go through every repository they got access to and look for something uniquely fun to do with that particular target.
If you were looking to do industrial espionage on a particular target, you'd probably pick a way that was less likely to be detected and cause a company-level security response. (Use passive recon to identify one or a few likely target email addresses, find one or a few likely passwords for them, and try them first. Heads you win, tails you just left evidence in a log of perfectly normal user behavior. You might follow up this perfectly normal user behavior with a social engineering attack.)
On the Rails note, you should be setting an ENV variable rather than hardcoding it as is the default. Same goes for any other keys/tokens/credentials that you're interacting with in your code. With these sorts of things, it's best to rely on pessimistic assumptions and take proactive steps accordingly.
You could, hypothetically, move from CookieStore to a different session store, but I am not positive that this will mitigate the vulnerability (by making it more difficult for the attacker to get write access to the session, not by making it more difficult to turn a session into code execution), and do not have a few hours to reacquaint myself with the code to give you a firmer answer there.
I am weakly confident, on the basis of some spelunking in the Rails internals in February, that if you get attacker chosen data into the session you're uniformly hosed on (at least) Rails 2 and Rails 3.
And Rails 4. The difference with Rails 4 is that the session is encrypted as well as HMAC'd, but it's still the case that as along as you have the token (or can otherwise craft a session) you're hosed.
Long story short: If a cookie is valid (HMAC signature matches what you'd expect given the cookie contents and your secret key), Rails (Rails 2) or Rack middleware (Rails 3) will attempt to use Marshal.load(txt) on the cookie contents to populate your session hash. You might naively assume that this creates a map of strings into other strings, but surprise, Marshal.load(txt) can create arbitrary object payloads and, through a code path which the margins of this comment are just slightly too thin to contain, allow you to do remote code execution.
As patio11 mentioned, moving off the CookieStore is a step in the right direction, with the understanding of course that you guard whatever you move to with your life. That means the db with the ActiveRecord store, or memcached if you're putting it in the cache store with the cache set to memcached. With memcached, be careful not to leave the port open to the internet if you're managing it on your own (it's designed to be used inside a trusted network, just like redis).
In theory, an additional precaution one could take: change the default session serializer to `JSON`. By using `JSON` instead of `Marshal`, you're limited to storing strings, hashes and arrays in the session, but that's a good thing. The remote code execution vulnerability takes advantage of `Marshal` deserializing the session and loading objects deep within the Rails process in order to run arbitrary code. Take away that ability by using `JSON` instead of `Marshal` and you should no longer have to worry about this particular attack vector.
After a brief spelunk of the Rails codebase (and not getting very far, my laptop bricked itself today) there doesn't seem to be an easy way to do this, apart from finding the instance of `ActiveSupport::MessageVerifier` in your rails process and then running something like `verifier.instance_variable_set(:@serializer, JSON)`.
From a Github email to a friend to whom this happened:
"We have reviewed our logs and it doesn't appear that any actions were taken by the attacker other than to authorize the 'GitHub XRP Giveaway' application against your account.
You should be able to find the OAuth events for that application in your account's security history:
Wow... a botnet-orchestrated massive brute force attack to try and game a cryptocurrency giveaway? o.O The future was yesterday.
As of this comment, 2020 RXP is worth ~ 0.03 BTC. Multiply by ~$500USD/BTC and you get ~$16 USD (over $20 when BTC was peaking $800+USD/BTC in the last couple days). Multiply that by the number of compromised accounts that meet the cutoff date criteria, and you get the take.
Potentially some good money depending on your success rate, but maybe not worth the cost of renting a botnet?
Ah, I misread that -- I thought it was if you had contributed to their repo, not simply starred it. I had left it out of the calculation for simplicity's sake as I figured most people would not have done that.
Where did you get the 1,698 figure -- was that the number of accounts compromised?
I just checked my security section (Didn't know about that!) and there were 12 attempts on my private, totally non-corporate, account.
My password has not been reset by github but I have updated it anyway.
On a sidenote I think that is a very clear and well written overview with clear details and a nice reminder about a section of the site that I was not familiar with.
I just checked too (https://github.com/settings/security) and I had 6 attempts, all from unique IPs. Asked a friend of mine and he had 6 attempts (all from unique IPs) as well.
If you get a user-password combo, you can try it in more sensitive sites that may have stronger protection against brute force attacks. People reuse passwords a lot, specially the kind of users that use weak passwords on the first place.
If the account has an email address connected to it, you can also try the password against the e-mail account. That can be worth quite a lot if it works.
Ah yes, what with the usual "email address being the skeleton key to your online identity" and all. I keep forgetting about this, and find myself wondering why the heck people are phishing me for my LinkedIn account login.
This reminds me the #1 tip of web development: Never trust the user. I can't imagine there is a person out there who has all the time in the world to brute force accounts with common phrases with 40,000 proxies (or whatever). Seriously, is there anything worth gaining if you get hold of even say 100s of accounts.
Well, first of all, it's very likely they used a botnet and some automated attack software. Secondly, there was a financial incentive[1] -- my back-of-the-napkin calculation was ~$15-25 per compromised account. That of course is ignoring secondary values like API keys, reused passwords allowing access to accounts on other sites, etc.
I see similar a thing; I'd guess that they just tried a few common passwords against millions of accounts, rather than targeting some specific accounts.
Facebook detects this when you move countries and makes you do a quiz about your friends pictures - though it can be pretty annoying [though I haven't seen it recently so maybe they've stopped].
Logging into my account, I can see quite a few failed attempts from a variety of IP addresses. All of them are from the same ISP in Venezuela. Quite scary, but my password is 40+ characters long and I have 2FA enabled. How weird!
Single failed login attempt from china. Though could this be any way related to the adobe leak? If you have the list of emails and password, you could try a bunch of commonly used services and see if that works.
> These addresses were used to slowly brute force weak passwords or passwords used on multiple sites.
..."multiple sites" could possibly refer to a compromised list, like Adobe? Of course, it doesn't have to be Adobe - Macrumors was compromised recently, etc etc.
Not really. Some ISP provide same external IP to multiple users and you have to take that into account. I think otherwise a good part of github's users would never be able to access it.
