> Here’s how the hack went down: Two attackers accessed a private GitHub coding site used by Uber software engineers and then used login credentials they obtained there to access data stored on an Amazon Web Services account that handled computing tasks for the company. From there, the hackers discovered an archive of rider and driver information. Later, they emailed Uber asking for money, according to the company.
I'm surprised Uber doesn't have their engineers set up 2FA for GitHub. Super simple to implement and require organization-wide[1] and would have prevented this. Then again, not storing credentials in GitHub would also have prevented this . . .
Github 2FA has been part of the first-day training/laptop setup for a while now (I joined in may) and there's security-related training in place as well. I was told there are also scanners in place now that check repos, gists, etc for secrets for exactly this type of mistake.
One snippet of the email the article didn't mention was that Sullivan's firing happened pretty much right after Dara learned of the breach and an investigation was conducted. It definitely inspires more confidence in leadership seeing that the CEO will not tolerate unethical behavior.
I think many people don't realize this, but the majority of the leadership team from like a year or two ago is now gone, including Travis.
Also, Uber has been hiring a lot of new people - the ratio of new people vs old timers is really high. I'm obviously just one anecdata point, but I believe new hires (and a lot of old timers) want Uber to be an ethical company, and many have joined the company specifically to tackle that challenge. One great example that comes to mind was when one board member made a sexist remark on an all-hands meeting a few months ago and by the end of that same day, Liane Hornsey (who had just joined as the new head of HR) had him give up his seat.
There's a big push towards trying to make things right, with the holden report, the 180 days of change campaign, the implementation of new training courses, anonymous complaint hotline for employees, etc. And the unspoken message right now is pretty clear: inappropriate conduct _will_ get you fired, even if you are the head of your org.
Obviously there's still a lot of work to be done, but I think we're at least in the right track now.
I really like how your description gets at these policies creating a tipping point in the culture. Hearing about any one individually always sounded like a bandaid, but hearing about them together and then how you and other employees react to them is very encouraging. Good luck to you and the rest of the company.
This is good for Uber and their employees in the short term, but I can't help but think it's bad for their ideals in the long run. There are a lot of scenarios that look very bad for Uber economically and it would be a shame for a culture shift to coincide with the realization of one of them.
Honestly, I find that a lot of economic discussions on the media are highly speculative (and dissonant to what I've seen circulated internally), and things get just downright sensationalist on some topics, so I've been taking news about Uber with a large grain of salt.
> it would be a shame for a culture shift to coincide with the realization of one of them
I think everyone at Uber has at least some idea about the P&L situation, but there's no doubt in people's minds that we need to drop the go-fast-and-dubiously culture and embrace a do-things-properly culture. If anything, I think it's more likely that a major crisis would continue to drive home that idea.
The downvotes are likely because you're taking an Uber thread veering it off to GM's management and your children, neither of which have any relevance here.
Except for the CEO being changed and having a toxic corporate culture that didn't change and produced the same deadly car across CEO's after promising change but did nothing different--including not stopping production of a deadly vehicle.
I probably should have spoonfed the readers more. They grew up in a world that doesn't need critical thinking anymore so it's probably too much to ask for their brains to activate while reading on a website and have them put distinct ideas together to form a grander one.
Must. Downvote. Comments full of facts but from people I dislike. Must.... errooorrrrroooorrrrrr. 505.
It's okay. Every time I see downvotes here, I know I said something great but I just pissed someone in power off. I'm used to being a minority oppressed by a majority in power. It's no big deal. The system just builds people like that these days.
Still not making the connection from Uber to GM that you are trying to make. Because the GM CEO could not prevent teens across America from joyriding, the new CEO is going to be unable to reign in the behavior of his own supports?
Either make a valid point or let your comments stand. Leave the /r/iamverysmart tandems at the door
We primarily use private phabricator and gitolite instances for internal stuff, but we also have OSS things in regular public GH repos. We do have a few private GH repos, but AFAIK, you're not supposed to version control internal stuff on GH, and there's no real reason to use a private GH repo, except for legal review prior to open sourcing.
I don't have any context on why someone would have put production secrets in a GH repo. If it had happened in my team, I would definitely have sounded the alarm at code review.
Well, I am one, but the things I say here are my individual opinions and observations. I just think that as an insider I get some insights that you'd normally not get from the media, and I figured I'd share them.
I'll believe it when they stop having stories like this every few months. They had over a year to report the breach, and they paid hush money instead. Typical Uber
Yep, but think of all of the private keys and tokens used in automation servers (think CI) for pulling down source. Those don't have 2FA - because they don't login - but they have full access to most source.
In an organization of about 200 engineers across various products, 1000+ github repos, and 10 or so different CI systems. We enforce 2FA at github. I can still easily see how someone could easily gain access to source code with secrets in it.
> In an organization of about 200 engineers across various products, 1000+ github repos
Wait, what? That's 5+ repos per engineer. What on earth would warrant that level of granularity? I've only worked once in my career in a place that used more than 2-3 repositories total, and that was a "MegaTechGiant" with thousands of engineers.
Depends on the company you work at, but most tech companies I've been at have gone the "micro" services approach.
Example:
- 1 repo for the frontend
- 1 for each api
- 1 for the infrastructure terraform scripts
It's good for CI / CD and general code base organization. Also easier to track changes and handle security. You give devs access only to the repos they need to do their job.
Our team has a product with multiple integrations and internal apis, so we easily have 40+ repos.
I know that mentioning downvotes usually invites more downvotes, but...
I'm surprised you're being so heavily downvoted for your question. Engineering teams (and software companies) come in all shapes and sizes. It is absolutely reasonable for even an experienced engineer to have only worked at companies with a handful of repos.
Rather than downvoting, it would have been helpful to explain why your company has opted for such granularity (perhaps engineers or teams have a high level of autonomy, or your software is highly componentised and built from a great many, separately managed, parts).
Some CI setups benefit from a one-repo-per-service approach, as it makes it easier to figure out when an individual app has changed. In orgs where everything is in one giant repo, it can be difficult to establish what subset of your applications needs to be rebuilt when a commit is pushed.
I personally don't have a strong opinion about either way - they both have tradeoffs.
There are just three of us in my company and after 10 years I worked on close to 80 projects for 30 different clients. Each project has its own repo. So +3 per engineer is really not that much;)
It's normal and expected. I have a few dozen. git makes it great to create little repos for lots of different things. They don't have to be production apps. They can be libraries, utilities, documentation, scripts, or just random crap I may want to refer to someday.
Could also be a company using clone - pull request workflow.
10-20 project repo an then each developer has a bunch of projects clones, including a few shared one - like the common infrastructure stuff, ...
I can see that with a company that has grown day 1 around Github, especially during early startup stages with a variety of contributors but no formalised "organization".
How about, if Uber stores all data across those git repositories (1000+)? Perhaps they use git as a multi-versioned data storage? Perhaps better than Kafka (event sourcing thing?). Just a thought :)
You couldn't enforce 2FA on GHE for the longest time. GHE version 2.8.0 lists [0] "Enforce two-factor authentication" as a feature. 2.8.0 was released November 2016. According to the article,
> Kalanick, Uber’s co-founder and former CEO, learned of the hack in November 2016, a month after it took place, the company said.
I don't know if they were using GHE. If they were, at the time it did not come with a good way for them to enforce 2FA for users.
Yeah this was such a PITA several years ago... To solve the problem we ended up building a small proxy in Perl for the express purpose of adding 2FA to Github Enterprise.
> I don't know if they were using GHE. If they were, at the time it did not come with a good way for them to enforce 2FA for users.
Well, sort of - at the application level, that's true, but GHE is typically run behind a VPN. Certainly that should be the case for a company the size of Uber.
Even before GHE added 2FA, it shouldn't have been possible for a leaked set of login credentials to be used to access GHE, without some other sort of compromise (VPN cert, physical compromise of hardware, etc.).
At my company (mostly a Windows and Microsoft shop), my domain credentials are used to log into the VPN, and TFS, and Octopus. Compromising just that one set of credentials could effectively "own" our company. And I'm just a senior-ish developer.
Lateral movement by an attacker is a real thing. And while credential reuse is something most security focused web companies are trying to mitigate, a push for "sso"-like account management is seemingly undoing most of that effort inside the network if not done properly (specifically, auditing and monitoring of behavior).
> my domain credentials are used to log into the VPN, and TFS, and Octopus. Compromising just that one set of credentials could effectively "own" our company.
This is why 2FA is important! I worked for a company that had a very similar setup: I essentially had a single "LDAP" password. But: everything web-browser went through a single sign-on site, and it required 2FA (and so, you were never entering your password into even random internal applications: there was exactly one page where you should log in). Terminal stuff had a similar flow that also required 2FA (e.g., for SSH). As a user, the experience was not painful at all.
It does seem like, however, from an operations standpoint, getting such a setup in the first place is not trivial.
If they are/were using GHE, I would expect (hope?) that they require some sort of VPN to get access to it, so my guess would be this was stored on github.com.
This is so gob-smackingly uncommon I started asking "do you require 2fa for your github accounts" as part of my interview questions when I was looking for jobs (i.e. I'd ask my interviewers).
I don't know how to feel knowing that there is even one software-focused company out there that doesn't enforce 2fa on its github accounts. Like... how?! Why?!
I really don't think using 2FA and the direct hacking of an individual developer's machine are all that comparable here.
Who cares about access to individual dev's machines if the credentials to access code on github are obtained - 2FA at least offers some degree of protection in this scenario. The scope for attack is extremely different.
> A backdoor can sit around and wait for the user to press the button.
There exist 2FA protocols[1] that permit tying the 2FA challenge to a particular context: you can't just take the response from the 2FA hardware and use it anywhere. In this regard, the malware doesn't get anything more than what they already have, and the 2FA still adds protection: if the malware is able to compromise your password (e.g., through keylogging) it doesn't immediately get access to everything you have access to. Now, of course, if you 2FA for some resource, then yes, at that point, you're probably doomed, but I don't believe that gets the malware anything new (e.g., once the auth is complete, if that results in a "user is logged in" cookie, the malware could just read that, and go to town.)
Compromise of a local machine is definitely bad, and not what you want, but 2FA tokens are not useless, even in that situation.
The hackers wanted access to the code to look for Amazon keys. For them it doesn't matter if they get the code from the internal GitHub or from a developer machine.
If you have an ultra-secure door, the thiefs will just enter through your regular window.
How do you know they "wanted" access to look for Amazon keys? Do you know it wasn't from a blanket scan of github?
Sure, there are only 13 projects on https://uber.github.io/, but there are 169 on https://github.com/uber, and it only takes a short while to scan for access keys. There are plenty of open tools that will scan github for keys.
This may not have been targeted at Uber but a net for all of github with Uber being just one company that was hit up for cash. Unless you're saying that you know the motivations of the attackers.
You only need the ability to generate TOTP or U2F tokens. This is often done using a smartphone app, but can also be done by a desktop app like 1Password or a hardware device like a Yubikey:
https://github.com/blog/2071-github-supports-universal-2nd-f...
You can also record the TOTP secret in your automated login script, next to your password, and generate the token on the fly right there.
It's things like that that make me wonder why TOTP tokens are supposed to be conceptually different from passwords. A TOTP scheme involves knowing a master password, and nothing else.
Recording a TOTP secret next to your password would make 2FA worthless, true. That’s why you should use hardware generators whenever possible. However, Github supports Fido/u2f which is conceptually superior to TOTP: The authentication secret is bound to the domain and the token generator verifies this. So even a software u2f implementation protects against phishing for example, while TOTP does not.
> use their personal phones seems like a very bad solution
Why? You're not any less secure by using a personal phone. What are the odds that an employee is going to be phished and have their phone compromised by the same entity.
IANAL, but here is my thinking: The problem with personal phones is they are hard to audit. When a phone belongs to the corp, corp owns the phone, and "probably" can audit it as it wished.
In order to install my work Gmail account on my phone, I had to install a program on my personal phone that let admins wipe it remotely. This is not something that bothers me, because I expect to lose the phone almost anytime, so the contents on it are backed up continously on a system I control.
Whereas that bothered me so much I refused to put email on my phone and told my employer they needed to provide me with a phone if they wanted me to always be on email.
I'm already answering emails out of office hours which is for my employers benefit and they want to functionaly own my phone because of it?
Unless you're talking about a 3 person start-up, wouldn't the use of github itself be a red flag? If you're a software company, you live and die by your source code. Why on earth would you rely on some other company to hold it for you? This seems as ridiculous as doing your bookkeeping on Google Docs.
I've never once worked in a company that permitted source code to leave the company network.
Because you trust their security better than your own, which at any organisation without a dedicated security team seems like a reasonable decision. I live and die by my money, too, and I give that to a private company to hold rather than protect it myself.
It's not just about who knows more about security. It's a trade-off, and you need to account for other factors like cost, availability/uptime, data integrity, total attack surface area and others. Honestly, I'm surprised this is such a controversial point of view, but judging by the downvotes it appears it is. You learn something new every day, I guess.
The point is that the trade-offs usually come down in favor of using GitHub Enterprise (or whatever other well-regarded, trusted enterprise system). The availabilty and uptime are your own, because it’s self-hosted, like git. The data integrity is also your own. The security is better than probably any other VCS interface over git, with the possible exception of GitLab, and almost certainly better than what an organization could come up with on their own if it’s not their core competency. Unless you’re literally using straight git, GitHub Enterprise (or again, whatever other competitor) usually enhances team productivity. The attack surface is larger than git, sure, but the rational solution to that would really be to use no interface over git, because GitHub Enterprise is as safe as they come.
I think you’ve misinterpreted people’s reactions. It’s not at all controversial to use other companies’ services for your most sensitive assets, it’s your opinion that appears controversial to them. If you’re in control of your own servers, what remains is to trust GitHub Enterprise not to literally phone home your source code or to enable remote code execution on your own server. There are myriad information security policies and compliance methodologies for compartmentalizing, quantifying sharing that risk.
For what it’s worth, having personally performed security assessments for over 50 different companies across the gamut of size/maturity, nearly all of them use a centralized VCS hosted or produced by GitHub or Bitbucket (and nowadays, occasionally GitLab too).
GitHub Enterprise is a different beast, as it's self-hosted. My comment was in response to the parent's mention of companies storing their source code on GitHub, which might imply external hosting. I suppose it was ambiguous.
Right, but none of those things is necessarily a home run for self-hosting your central git repository. Particularly in today's world, where you likely have remote workers and don't necessarily have any other servers you're managing, anything you could call a "local" network or even a VPN.
I've been surprised how many commercial, closed-source projects have opted for Github in recent years. While I would probably prefer to self-host (Gitlab, or similar) in order to reduce dependencies, I do see the benefits. Having recently worked at an organisation hosting exclusively on Github, it made collaboration with remote contractors and third parties very straightforward and helped eliminate much of the maintenance burden on our small team.
You have a full checkout on your laptop and probably a whole bunch of other developers laptops. With git you can also have random backup computers do the same thing! You don't have to rely on github alone, for this.
uber engineer here, we have 2fa set up for everything. Starting my day takes about 5 different 2fa checks (ssh access, aws, phabricator, team chat, etc)
I know Uber has a strong engineering culture, which is why I was so surprised. I think philsnow's assessment that organization-wide required 2FA wasn't available for GitHub Enterprise at the time of the hack is probably correct.
Although more and more applications support SAML for SSO, much of the SaaS world is disparate and siloed. There's definitely something to be said for centralised user management on a homogeneous system. User leaves your organisation? Just retire them in LDAP.
2FA wouldn't have necessarily solved this, if the hackers had access to an engineer's ssh keypair (e.g stolen laptop) they could clone repos as they pleased. 2FA isn't a silver bullet.
Maybe it's just me, could "private GitHub coding site" have meant a private GitHub repo with GitHub pages turned on?
If that were the case, there would be no authentication whatsoever to access the closed-source site; the hacker would have just needed to guess the right url.
The most I've ever personally seen a company do is require a VPN for their privately-hosted repos. For others using GitHub or Bitbucket? Never anything beyond a standard login.
Two factor won't protect you from a spear-fishing attack.
The attacker can submit your info to GitHub the moment you submit to the malicious site. You receive the token via SMS as expected, enter it on the second page of the malicious site, granting them access.
I assume it was password reuse from one of their engineers or something similar. If you could compromise GitHub itself there would probably be higher value targets (source code for upcoming AAA games, Coinbase, government organizations, etc.)
AAA games have budgets in the millions. Threatening full release would likely net you much more than a few hundred thousands, and without requiring any secondary attack.
We use a tool under a Linux Foundation project called anteater https://github.com/opnfv/releng-anteater, which does the same thing (but is for a jenkins / gerrit workflow). A key difference from looking at talisman, is anteater uses standard RegEx rather then code to seek out strings, so anyone can add their own strings / file names easily into a simple yaml file. Like wise they can use regex to provide a waiver, should something be incorrectly reported.
I am thinking now would be a good time to port it to working with webhooks as well.
There are a few SaaS offerings that will let you do that. LastPass or onepassword are two commonly used.
One you can use something like keypass to store a database in a shared location if you don't trust the SaaS offerings.
If you are looking at storing credentials for automation purposes, and don't have a secret store built in, you could look at something like Hashicorp Vault to help provide this for you
We're using Keepass / MacPass password protected vault shared with the team using Dropbox. It's really good and essentially free to use if you use a free Dropbox account.
They key part is "Warning: Once you have pushed a commit to GitHub, you should consider any data it contains to be compromised. If you committed a password, change it! If you committed a key, generate a new one."
Removing the secrets from the repository is nice to have, but not that necessary - what is mandatory is to ensure that the compromised secrets are no longer useful, since they aren't secret any more and won't be ever again.
I am rather disappointed in github for publishing this guide. The portion at the top stating
> Warning: Once you have pushed a commit to GitHub, you should consider any data it contains to be compromised. If you committed a password, change it! If you committed a key, generate a new one.
Is a good argument as to why you shouldn't let users erase this data from history, it's already out there so no matter how painful or convoluted your process is for regenerating auth credentials is, you need to do it if you've published them into your SCM. If the process is painful you might want to simplify it because you'll probably need to do it sometime in the future again... yes even you large corporate workers who have no control over credential regeneration, an arduous process leads to credential sharing between projects which is another horrible thing.
Yeah, but hopefully they can't do much if they just have your code base. If the secrecy of your code is the only thing stopping hackers from exploiting you, you're missing some gaping holes in your infrastructure. With that said, nothing wrong with using secrecy as a additional barrier, but shouldn't be the only, and if it's not the only, you're not "so owned at that point".
“Just” leaking full source could be enough to destroy a lot of IP-based companies. A lot of companies stay wealthy because their IP is so huge than nobody can afford to develop competitive alternatives anymore (Adobe, Microsoft Office, Salesforce etc). Some of them have actual “secret sauce” that they cannot afford to share (suggestion engines, biotech processes etc). Even a service like Github, which relies on others entrusting their work to them, would take a humongous reputation hit from a leak like that.
I don't think either of those companies would cease to exist if their code bases leaked online today. Sure, someone might get something to build, but there is surely A LOT of things around the code bases to support all of this, which means the code bases would mostly serve as a study for software in general (and finding holes obviously).
Github is a bit unfair comparision, as their business is literally to make your code private, so if it leaks then of course it would be a hard hit. For the general company, I think leaking access credentials is a much bigger (but easier to fix) problem than leaking the source code itself.
> I don't think either of those companies would cease to exist if their code bases leaked online today.
A serious Photoshop clone that can match PS feature for feature would wipe Adobe, people cannot wait to get rid of them. 25% of MS revenues comes directly from Office and another 25% from Windows or other commercial offerings that are basically driven by Office, so yeah, MS would survive a working Office clone, but they would be deeply wounded; they pulled all the dirty tricks in the book to keep competitors from integrating seamlessly... having the real code responsible for their formats available in the open, would hurt them massively.
These companies are as big as they are because they did the right moves at the right time, and now they have spent so many man-decades on their codebases that nobody can realistically hope to catch up starting from scratch; but having a good look at their codebases would likely kickstart oozes of competitors with very good chances to replace them in a very short time.
> For the general company, I think leaking access credentials is a much bigger (but easier to fix) problem than leaking the source code itself.
Credentials are a mean to an end: protecting something. If you are Ashley Madison, your valuable IP is your database of users and their preferences; but if you are Microsoft or Adobe, what credentials are protecting is your source code. Adobe survived their user credentials being leaked, like so many other companies. They would have hurt much more had they leaked the entire PS codebase.
But a competing company can't just give a copy of the leaked source code to their developers and tell them to go to town. Even by employing clean room design, you can't get around all the patents that likely protect many of the features that Photoshop users consider crucial.
"If the secrecy of your code is the only thing stopping hackers from exploiting you"
I hate these types of arguments. Yeah no one said that ever.
Losing your code base is terrible. I view it as losing a journal. What your company tries, tests you run, funny comments, or funny mistakes. I mean they post it on the net, blackmail team members, imposter team members, forge for leaks, sell it, pushes to prod from compromised accounts, CI systems, -- seems bad to me. Sure don't have aws keys in there.
Glad to be talking with you too! :) I didn't mean to imply you said something you didn't, only that I would consider access keys to various services be of much more importance the code base itself. I read you comment as "Doesn't matter about the access keys, if they have your source code, you're screwed no matter what", which in that case would seem a bit strong.
Also "pushes to prod from compromised accounts, CI systems" seems more related to access keys and account security rather than the actual code base.
But hey, in the end I'm no security expert so what do I know.
If they have access to the code inside Github, would they have been able to push their own changes to the code without anyone noticing?
Maybe pushing something that was labeled as a "security patch" but was actually a disguised vulnerability? I could see not even checking into that, and just downloading it. But I'm on a small team. Do big companies have procedures to protect against this?
Depends on how they get access. If they got control of one of the user accounts with push access, they could surely push code (but unsure about "without anyone noticing", depends on their own development processes I guess). However, if they got access to the code by reading some part of the memory/storage holding the code, without actually gaining access through authentication, they wouldn't be able to change it.
Really surprising to see that sensitive credentials were checked in to VCS. Apart from peer code review, how can a company avoid developers checking in sensitive data to VCS?
I really wish AWS would stop enabling master API keys by default. As soon as you create an AWS account you are given API keys which basically have SUDO permissions to your entire account. That is super dangerous and is probably the same key set that these hackers got ahold of. AWS needs to disable these full access API keys by default and instead should encourage users to generate keys for specific access to limit what they can do.
Store credential information where it is used. It is not used by the repository, so it is an improper location for it.
If someone gains access to a system that uses the credentials, then there is, in principle, no difference between puppeteering that system versus stealing its credentials.
Every day we see more evidence that boards of directors and senior management should be personally accountable financially and with respect to their liberty for the company they are managing or overseeing doing foul things that they ought to have known.
The "I didn't know, I just took a vast salary to play golf" argument should not be any kind of defence. If there is the real prospect of going to jail, golfers will resign, those who take the job would actually take an interest and have the ability to do so.
I'm in charge of security at a large e-commerce company. I do not play golf. I mostly live in fear.
No sensible person would sign up for the CSO position if they risked jail time when their company gets hacked. You can't really control it. A random engineer could make a mistake that gets hackers a step closer. Or it could be a zero-day vulnerability that nobody knows how to protect against.
There are millions of motivated adversaries out there and a finite number of employees at your company to outsmart them. It's a game you can't win. The larger your company becomes, the broader your attack surface becomes, and the higher value a target you become.
You just have to hope that when you get hacked, it is a "forgiveable" hack like a zero-day or highly targeted attack.
If CSO's are to be personally accountable for the malicious actions of others, it needs to be due to clear negligence on their part and the responsibilities need to be clearly defined.
Not because you got hacked. No. Hell no. I never suggested that and reject it totally.
We're talking about cover up, if you cover up the fact someone stole private data belonging to other people you took responsibility for. If you try and pretend it didn't happen because you might get away with it then claim you didn't know when it comes out? Then yes, absolutely, you deserve to risk jail time for that. As does your board of directors.
CSOs, senior management, boards of directors should be personally responsible for their own actions. They need to have something at stake that they really dread losing when making the decision "perhaps we can get away with this?"
The problem with jail time is that the courts need to assess how much damage was done to determine what a fair sentence is. How do you assess the damage done during a data leak? Do you get one hour of jail time for each person's data you leaked? Do you get a day per gigabyte leaked? What if nobody does anything with the stolen data?
And how do you make that scale? If I miss a semicolon and leak 5 people's data, then I'd hardly get any jail time. If I miss a semicolon and leak 150,000,000 people's data, I will die in prison. In both scenarios, I made the same error, but the outcomes were insanely different!
IANAL but isn't it already the case that if you knowingly break a law, you can be convicted and sentenced to jail? In this case, I'm assuming that it's not against the law to cover up a hacking incident, in which case there's no basis to suggest putting someone in jail.
Covering up isn't done by the CSO. It'll be driven by lawyers, or worst-case by a rogue CEO. Mind you the CSO would be expected to resign under those circumstances, if they were kept in the loop on the cover up plan.
It's not really much about that Uber got hacked, which is bad enough. It's that they paid off the hackers and didn't tell anyone they got hacked until now.
I’m not sure the previous comment is saying in all circumstances. I agree with what you’re saying - because higher officials obviously aren’t able to catch every mistake made by every engineer but I think the parent comment really meant that if there is gross negligence or intent to cover up breaches then that seems like a crime that the individual should be punished for.
You raise a difficult issue - how you would honestly resolve it. On one hand, CSOs cannot be personally liable for every hack. On the other, they shouldn't be given a pass on everything either.
So how does one draw the lines between bad luck, reasonable security problems, everyday poor performance, civil liability, and criminal negligence?
> A random engineer could make a mistake that gets hackers a step closer
That could be prevented, to a large extent, with much tighter controls. Of course, those controls would greatly increases the cost of operations and other things.
Is it possible we're all accustomed to the wrong model, that our standard of IT security is like the standard of car safety in the early auto industry (and maybe until the 1970s) - far too lenient? Maybe we should be facing the potential fact that the normal cost of IT should include those controls and other security expenses.
(In the EU) companies are already required to tell where my personal data goes to. There is no specific fine for violations as far as I know though.
Essentially we need a price tag on personal data. Let's say 1$ for each email and password leaked to an unknown number of entities. That would be a 114M$ incentive for Uber to keep their data secure.
> There is no specific fine for violations as far as I know though.
It's a shame this happened pre-GDPR because that has steep fines - 4% of worldwide revenue - which would be north of $260M going off their 2015 numbers. And that's assuming they get off with a single fine.
GDPR is pretty much the thing that will - if properly executed - mean the end of these things.
As CEO, former engineer and customer I really hope this gets some serious traction. IMHO if you are making money from customers, it should be mandatory to follow compliance regulations and protect all data.
GDPR will come into effect in about half a year. Everyone is sitting duck about exactly how to implement things. When this gets into effect, companies will take it seriously - the fine is astronomical if you fail.
> You raise a difficult issue - how you would honestly resolve it. On one hand, CSOs cannot be personally liable for every hack. On the other, they shouldn't be given a pass on everything either
Sure they can. It is called "insurance". Sort of like malpractice. CSO wants to get paid millions of dollars? Excellent, either be personally on the hook or have an insurance company that would be willing to underwrite your method of dealing with it, be that having your own crack team of people who get to oversee everything, or relying on Jr system admins from your company or whatever else.
If you can demonstrate due process and reasonable effort to secure against breaches, your doing your job. For instance with described procedures, sane defaults, locked down environments ect. But if you're a CSO and have not described threatmodels, workflows, and security practices, then you've not done your job and should be held accountable for data breaches.
I think your perspective is either immature or unrealistic.
OP's a realistic. His perspective is nothing to do with how a company values security.
No one in security assumes they won't get hacked, we assume we will and when we do get compromised. Our metrics aren't measured on if, our success metrics are:
* How quickly we find out
* How much damage we can mitigate
* How quickly we mitigate the risks and controls for X vulnerability and
* How we incorporate our reporting to find trends to find the event quicker next time
Now we report on many compromises. I'm not talking just about data breaches here, there's a whole spectrum of compromises that we manage and mitigate.
I don't know anyone who operates in Security who has a different mindset to OP.
> OP's a realistic. His perspective is nothing to do with how a company values security.
Of course it does. The stick is not big enough so CSOs just do not care enough. Increase a size of the stick and it would split the group of CSOs into two:
1. Like OP will run away saying "I'm not going to put myself in a line of fire if crap gets hacked". We need broomsticks for those.
2. The ones that will say "OK, two years", do their best and probably succeed.
Having practices in case for the event of a hack is obviously good, but it doesn't imply believing that you can't control getting hacked and can't win against the hackers (previous poster's exact words).
It's because you can't control it. There are limitless attackers and vectors. Security is mostly a game of being hardened enough to where most of those attackers will give up and go off looking for easier targets. Against a zero-day that nobody knows about yet, or an extremely determined attacker with a lot of patience? You will eventually lose, and you have to do your best to detect when it happens and act accordingly, as stated previously.
Our company cares more about security than anyone in our space, if you look at how much we invest relative to the others. We have full time penetration testers on staff. We contract out to countless third party security vendors. We take their advice.
This has nothing to do with not valuing security, it's just about being realistic. Can you guarantee that your company is hacker-proof? No? Then we're on the same page.
I'm not sure why we have to accept a dichotomy between guaranteeing hacker-proofness and throwing up your hands and saying you're bound to get hacked no matter what you do.
It's great that you take all those steps and investment. The fact that you still don't believe you can control whether or not you get hacked is a sad reflection of modern software practices, which are akin to throwing together a house out of plywood, newspaper, and gasoline, then asking the security team to place fire extinguishers.
I believe it's more like getting into a car accident. You can be the best driver in the world, you can always drive under the speed limit and take all precautions but you are bound to be in an accident at one point or another.
You may go decades without incident but it's almost a certainty that you will find yourself in a situation where another driver collides with you in a way that couldn't have been forseen. This driver could have hit you accidentally or on purpose, it doesn't matter. You could be teaching another how to drive during the incident, you could have had a momentary lapse in judgment...it doesn't matter. What matters is how you handle the situation after the fact and the steps you took to mitigate the damage.
If you spend enough time on the road the likelihood of an incident approaches 100%.
More a sad commentary of how many people think there's some magic bullet of security practices and if they just follow those, then they won't be hacked.
If you don't assume that you will be hacked, then you won't design in auditing, alerting and containment that will tell you when you've been hacked, let you determine what data was compromised, and prevent the attacker from having free reign over all of your systems.
Otherwise, you'll be like a former coworker that refused to secure internal systems because "We paid a lot of money for our firewall, it's going to block any hackers". It took me less than 30 minutes on my first day to hack the login passwords of senior executives because they logged into a non-SSL reporting server (and I did through a simple MAC overflow attack on a network switch from a network port in the break room)
> If you don't assume that you will be hacked, then you won't design in auditing, alerting and containment that will tell you when you've been hacked, let you determine what data was compromised, and prevent the attacker from having free reign over all of your systems.
I see a big difference between preparing for the event of a hack, and believing that a hack is inevitable no matter what practices are in place.
How do you get your CEO to pay for the monitoring and other breach preparation if you've just told him that "We have air-tight security, we cannot get hacked"?
CSO: We have airtight security, we cannot get hacked.
CEO: Great!
CSO: Please approve and fund this plan to handle a breach in case we are hacked.
CEO: But you just told me we can't get hacked.
CSO: Right, it's impossible.
CEO: So why do we need to spend money preparing for it?
CSO: Just in case.
CEO: Just in case what? You just told me it can't happen.
That seems a little like asking for money to prepare for an alien invasion or a zombie attack.
Probably the CEO would have read the document their Insurance Carrier made him sign that details the measures they need to keep cyber cover valid and therefore this conversation wouldn't happen.
There is personal liability for board members and management boards (CEO etc.) in many (most?) jurisdictions, usually including for some things that they should have known but didn't know, typically if the "not knowing" part can be characterized as gross negligent.
Also, the personal liability for board members and managers is something that is exceedingly pursued by shareholders and creditors (for the financial liability) and prosecutors (for the criminal liability) compared to how it used to be.
You cant just give Jail-time for data breeches. It would encourage cover ups and scape goats. Also never underestimate just how disorganised large organisations are, incompetence at addressing issues is systemic and goes far beyond data protection. What seems like malice is sometimes just plain stupidity.
It has to be backed by some sort of regulatory framework. Just like a fire code or employment rights. But crafted in a way that it doesnt end up like PCI, ratings agencies or financial auditors. IE creating an industry that sells compliance and not actual security.
Perhaps something light, like mandetory minimum bug-bounty schemes for all companies, where fines (or more) are imposed for not addressing issues and an independant regulator works with larger companies to resolve issues (or penalise the company severely if they deliberately wont).
The reasonable company director should have known X and when found out was bound to report it. Person Y did not report it, should have known as it was their job to know and there aren't extenuating circumstances. Guilty. 6 months. Next case.
"I don't know anything about this company I accept 7 figure sums to oversee as a director." Should never be any kind of legal defence. If senior management and directors have something personally at risk you'll see vastly improved behavior. Right now we're selecting for the opposite and seeing the inevitable results.
There is a story like this about directors and management cover ups everysingleday
So person Y read an email late at night and forgot about it. So you send a director to jail. Tomorrow many of the "good" directors feel scared and they simply do not accept any new appointments.
Who will fill the void ? People who are overconfident and people who are not scared of going to jail.
It's much better to impose financial penalties. Should the directors or the shareholders pay ? Let them figure it out between themselves!
Doesn't PCI work well in general? It has a large amount of overhead but we see very few credit card breaches from within the "PCI vault," while we see many Social Security number breaches and email/password combinations.
For example, the Target credit card breach occurred because malware intercepted the credit card information at the Point of Sale appliances before the information was encrypted and transmitted.
Prison time seems extreme, but Congress should should absolutely establish statutory fines (for companies) for breaches of PII. Then any company officer can save the company money by simply spending more on prevention because it will lower breach insurance premiums.
You cant just give Jail-time for data breeches. It would encourage cover ups and scape goats.
WEll that's already happening without jail time so maybe give it a whirl. LEt's get real here, the idea of suits going to jail is just scary to some people but it'll be fine.
Man, I don't know if Uber is evil or if most tech companies are evil and Uber just doesn't drop the kind of money on PR strategery that an evil company need to drop in order to seem normal. But either way, holy cow does that company come off as toxic. They've completely revolutionized the drive-for-hire industry and all anyone ever hears about it what a D-bag their CEO is or how toxic and mysogonist their work environment is or how hard they work to spy on their employees and customers (to the point that the CEO of Apple had to have a Come To Jesus talk with the CEO of Uber) or how their employees feel like they are getting screwed or, now, how they are concealing massive data breaches. It's like how I imagine Uber would be if it was run by Magneto instead of Tony Stark.
>and all anyone ever hears about it what a D-bag their CEO is or how toxic and mysogonist their work environment is or how hard they work to spy on their employees and customers
I don't think the average Joe is up to date with this news, or even care about.
Nope, they don't know. My non-tech friends are all using Uber / UberEats with no clue about the company, and the CEO. They just see an easy to use app to get what they want quickly.
> My non-tech friends are all using Uber / UberEats with no clue about the company, and the CEO
Just like we don't know anything about the CEO of the product making your detergents, the CEO of the brand of clothes you purchase, the CEO of your oven at home... Not knowing about CEOs is rather the norm, not the exception, and ultimately if the product/service is good, the CEO does not matter for most people, or they are only going to care about it in passing and then return to their old habits. GoDaddy is still in business.
Speaking of politics, media influence and tech, a fair number of average joes have a terrible opinion of Musk/Tesla/SpaceX; but these same people are unaware, or maybe vaguely aware the US Gov spent >10billion, just in direct financial losses, bailing out GM.
If everyone in the country was told "write a check to GM for $50 or go to jail," and conservative media wasn't berating Tesla/Musk, public opinion would be a lot different... Take it all with some healthy skepticism.
This is news to me, I've never come across an average person having a negative opinion of Musk. On the contrary they think of him as some iron-man like figure.
Most "average people" I know (admittedly on the other side of the world) have some name recognition for Tesla the company (they make expensive cars and batteries right?) and would probably struggle to tell you who elon musk is, let alone have an opinion on him. The tech bubble has thick walls sometimes...
How many people do you know that listen to Rush Limbaugh or read Breitbart? Apparently none.
Personally, I like him quite a bit, but to be fair I know that outside of my own echo chamber of my news and social media feeds, that there are a lot of people who don't like him, and where that negativity is coming from.
HN is a community. If users don't have some consistent identity for others to relate to, we may as well have no usernames and no community at all. That would be a different kind of forum.
Anonymity is fine, and throwaways for a specific purpose are ok. Just not routinely.
I think it's more the "anti-fact" wing of the media (which does mostly overlap "conservative" on the Venn diagram). Unabashed alt-right agitprop outlets like Breitbart news, for instance, or climate change deniers.
There are a couple different things at play.
First, one plank in their infowar strategy is to combat anything that even indirectly propagates any understanding of climate change among the proles. They take positions even against more-efficient-than-incandescent light bulbs, so this line of attack certainly includes targeting electric cars and solar. Musk is obviously a celebrity of sorts in these areas. Any government help to build solar plants or subsidize non-fossil-fuel alternatives (e.g. electric vehicles) is portrayed as deeply corrupt, a betrayal of American values and working families, etc. Ergo, Musk is bad.
Two, Elon Musk and John McCain have a strong association. Musk has supported McCain and in turn McCain has supported Musk and his business ventures. This is the kind of invest-in-politicians-who-can-help-you relationship that is pretty much a fundamental building block of how the American government works, but it always looks bad to somebody inclined to see it that way. (It's probably also objectively bad that this is how the system works, but anyway it is.) So I think a lot of conservative media that doesn't like McCain (because he is too "establishment" or whatever the reason) have repeatedly brought Musk into it, implying corruption on the part of McCain to help Musk use Russian rocket engines at SpaceX, for example. McCain is bad, ergo his sleazy buddy Musk is also bad.
On your first point, I don't think most (fiscal, small-gov't) conservatives would have much problem with tax incentive for electric vehicles, or anything that generally reduces taxpayer burden. But they are typically vocally against subsidies and grants that favor particular individuals or companies over others -- which is not only unfair, but also adds to more spending. It does little, but to help justify a bigger gov't.
Secondly, SpaceX have been spending millions in political lobbying and McCain's political campaign is among many who benefited from such largess (and his own McCain institute) from Musk. Most Americans don't see this kind of lobbying activities with millions dollars spent on politicians as a "fundamental building block" of a well-functioning gov't, but a corrosive force that serves interests of a few at the expense of the majority, however well-meaning in the eyes of Musk supporters. I personally don't see any problem with organizing an interest group to better represent their views -- or lobbyists -- but when it involves so much money and the final outcome ends in lopsided legislation favoring one particular individual or company over others, it's probably a good time to question their "invest-in-politicians-who-can-help-you" relationship.
Ideologically, McCain's views are aligned with those of the "neoconservative" wing of the republican party -- he's mostly known for aggressive foreign policies, American democracy everywhere, and subsequently pro-Military Industry Complex (MIC) which inevitably all leads to a bigger gov't. While most conservatives are also for strong national defense, not everyone is necessarily on board with permanent warfare and welfare (and police) state and that's why "other" conservatives are so annoyed with McCain.
So, once you put these together, it's not too difficult to see why the holy alliance between Must and McCain is criticized by those on the right. They are not necessarily grounded on "anti-facts" or alt-right views as you mischaracterized here. It's just too bad that your pathetic, uninformed comment had to start with the poisoning the well logical fallacy.
I personally mostly don't agree with conservative media either, and I even mostly agree with you here, but to be fair the left also has their anti-fact narratives & outlets, and wrongthink, just the same as the right--just on different issues.
I completely agree with you, and didn't mean to imply otherwise.
Although I do think there tends to be a broader overlap on the "conservative" side, for reasons for that are complicated and don't necessarily have a lot to do with being conservative, the "liberal" side does indeed have its vaccine deniers, MSG paranoiacs, and so on. (However, they don't have TV networks dedicated to these things, available in every hotel and airport in the country...)
I try to judge media organizations (and people) based on their commitment to truth and openness to empirical evidence and new information. Their political leanings may be interesting, but are a (much) less significant data point.
That used to be the tendency, yes, but according to Jonathan Haidt at https://heterodoxacademy.org, "the left" has caught up to "the right"/conservatives in the number of scientific topics they deny outright, and it has happened in the last 5 years.
So very recently, and unless you've been to college in those years, you won't be aware of it.
I forget where he said it, or I would link to it. It might have been in a recent conversation he had with Jordan Peterson.
I don't know. I've received a lot of flak for even using Uber from non-tech friends/dates recently. I think the continual tide of negative publicity is definitely having a material effect on their brand image.
Funny, they opened a satellite office right near my apartment and I'd considered applying. Then I heard pretty disconcerting stuff about the environment, and now this. Dodged a bullet, I guess.
And testing their self driving cars without getting the proper permits. And hiding shit like this from investors so they’ll lose a lot of money later.
I would never work as an engineer for a company like that. How can I trust that it will honor any deal I make and not screw me? I have to think about that with every company but this one in particular can’t even spell ”integrity”.
Nobody's getting screwed yet. Nobody had their data stolen then covered up either, until they did. No investors were lied right into their face about this either, until they were. No women were harassed and had the events covered up either, until they were.
If the company views engineers as better than other people and someone they wouldn't want to screw with, I'm not working there either on principle.
Google and Facebook are publicly traded companies and stock can be sold immediately; I'm not sure we can say Uber is not screwing employees until there is a liquidation event (not counting the internal buy back program)
I never quite know how to think about them. On the one hand, they’d changed an entire industry in a way that people wanted but was getting serious resistance from the entrenched players. They had to break a lot of rules and go around a lot of people with a whole lot of connections to get where they are and in the process made a lot of enemies.
I expect blowback. I expect negative news. They essentially pulled it off by looking at every day as combat where fighting dirty was rewarded.
Yea, probably a good idea. I only even need them when I travel but all the cars I saw at the last airport had Uber and Lyft stickers anyway.
The biggest part of the comment was seeing the taxi driver protest in Seattle when I was there on business. My hotel room window had a view of city hall and I watched a bunch of cabs with a news crew pull up for about 45 seconds and start honking their horns. Then they all left and went back to taking fares.
When I watched the local news that night, the broadcast made it look as if they'd blockaded city hall for the day in protest.
It's the things like that that give me pause when I see bad press around a company that has upset entrenched interests.
AirBnb had to fight a very similar path and the only bad press I can remember about them was that tone deaf/ offensive political marketing campaign they had.
AirBnb have their own issues, and plenty of bad press about cities popular with tourists crumbling under the load AirBnB are causing on their communities.
There's been several articles about them shafting apartment owners, offering very little from their "insurance" for trashed apartments. In most cases the renters are sane people, but if you get the drug-fueled orgy, you can probably safe in betting that AirBnB will not compensate you for the cleanup.
Plenty of companies have disrupted plenty of industries by now, but without this kind of behavior. I don't see how the way they dealt with the data breach was required to disrupt the local ground transportation industry, nor was the way they've treated employees (including drivers) and customers.
As much as I hate blaming the victim, but giving your personal info just to get a Taxi ride is utterly moronic. I don't want to have any relation of any sorts with a Taxi, I ride I pay and bye.
Uber is like the Donald Trump of corporations. It sets new ground in how openly vile you can be without any consequences. Just like Donald Trump the real danger is not in Uber itself but in whatever it will be that uses it as a role model.
You can't say anything approaching positive about Trump. You will be down voted. The vehement supporters of freedom of speech doesn't support this. Well, the irony.
User starik36's comment was in a downvoted state. Which is what prompted me to write that comment. I didn't think what he said deemed a down vote because from general observation what he stated seems true.
There isn't any irony. Freedom of speech is about preventing government censorship of citizens and has absolutely nothing to do with shielding people from the social consequences of saying unpopular things. It's about being legally allowed to say unpopular things, not about stopping people from disliking what you say.
I agree. That's all I meant too. Stating the obvious - As in, if you say anything in support of this Trump fellow, you will be down voted. And this is true as exemplified by the "down votes".
Just yesterday, I got some serious flak for suggesting that changing one person in leadership wasn't enough to make them not evil. I'm sad to see just how bad this is/was, but I'm not one bit surprised.
I think it's progress. Culture does tend to come from the top down - and just like dogs companies do have a habit of resembling their upper management.
Looks like they fired two people over this, pretty immediately at that. Uncertain if the new CEO was aware of the cover-up until (presumably) contacted for comment by a news org.
The fact that the cover-up persisted this long is bad, but on the other hand the Kalanick-era Uber probably would've gone to war with the journalists breaking the stories rather than admit fault, so there's that.
I'm not sure if that's an accurate analogy. Few politicians would want to find themselves working in the climate Trump has found in Washington, (if anything, he has proven that a groundswell of popular support can't unseat a party establishment). Also, it simply would be inaccurate to describe Uber's actions as impotent.
Edit: allow me to replace the word "found" with "created." I was just using a figure of speech.
If you think the current sitting POTUS is an innocent victim of politics, then I have a bridge to sell you. Uber has used similar PR tactics in the past to deflect/detract from their actions.
> If you think the current sitting POTUS is an innocent victim of politics, then I have a bridge to sell you. Uber has used similar PR tactics in the past to deflect/detract from their actions.
I think the point that the great great? grandparent top post was making is that whoever is in charge of dealing with the media at Uber is doing a horrible job.
Also, I am sad that we don't talk about the policies and rather focus on the personal flaws. I think there would be a chance of a compromise if we debated on policy. I mean if we talk about just personality, what makes our Honorable Governor of New Jersey eligible for office? Not a fan of 45 but really I think politics has become too polarized.
Don't check secrets into VCS, folks!