The really worrying thing about this is that Okta’s announcement fails to mention several factors which paint them in a bad light, like the fact they sat on a report about this compromise for 16 days.
Surely you’re not suggesting that the company whose CEO denied being compromised for a week or so (before relenting and playing semantics games about whether it was “really” a leak or a hack) _might_ not be real trustworthy about disclosure…
I'm left a bit perplexed here. It sounds to me like Cloudflare did the right thing, but I don't understand why they still want to use Okta moving forward.
Surely instead of "recommending" a better security posture, they should tell Okta that they won't continue to use them as a vendor unless they follow a stricter security policy?
First, you are assuming Okta is worse than any other vender. It is very very very possible that other authentication venders are either just as bad or even worse. Remember, no one knows how to build an unhackable computer, service, or system.
Second, moving venders is VERY hard. It can take years to do it. I have worked at another cloud provider and moving from an old system to a new one is a huge undertaking. You do not just say, “See you, I am out of here.”. It can take years of work and planning.
Third, you do not know what CloudFlare is doing internally. My guess is they have already spoken to Okta several times about the problems they have had. At this point, CloudFlare may actually be moving off Okta or thinking about moving off of Okta.
Often I see news like this and expect the stock price to take a hit, and then am surprised to discover that "the market" doesn't care. Not so in this case: OKTA down 12% today
Edit: Looks like a fairly strong down day in general so maybe it was just a coincidence. When you look at 1Y of data it's clear that OKTA swings around a lot. There's precedent for moves like this, in other words.
With most security breaches the product is serving some primary purpose other than security: security is an ancillary goal, and security breaches may not immediately cause companies to reevaluate their investment in the product.
Okta is different—the problem it solves is squarely in the domain of their customers' infosec teams, and its one job is to be safer than alternative authentication methods for large enterprises. It makes sense that investors would worry about them taking a larger hit than ${random_SaaS_product} would.
The title of the blog post starts with "How" but the only actual how that is in the post seems to be basically: "we used Cloudflare products X Y and Z which mitigated the issue". Unfortunate.
Cloudflare 'dog-foods' its own products, and that saved the day; what's the problem?
There might be/come a longer form technical write-up with more detail of how those products helped, if that's what you're after, CF often seems to follow that approach, sort of announcement + detail, or press + technical.
The problem is, if you asked me how did you bake that cake, and I said: I went to my kitchen and used up my ingredients and my methods, then I would not have answered your question.
>There might be/come a longer form technical write-up with more detail of how those products helped
There won't be, they just used their blog to pressure Okta to improve certain things, which is their right I suppose but it makes for very boring reading.
I get what you’re saying, but Cloudflare isn’t trying to keep their methods a secret. They’ve openly written before about the key takeaway: how phishing-resistant MFA (particularly security keys) have saved them in the past. Everything else is just product documentation that’s online to read.
I think in this case they _are_ keeping their methods a secret; they don't say how their detection methods work. The BeyondTrust blog post doesn't mention their methods either. In this case that is probably right because knowing what they do to detect intruders would make it easier to stay invisible.
I think we might be talking about two different layers. I was misinterpreting your statement as “why won’t they share their safeguards”, but it sounds like you meant you would have preferred something more analytical in terms of the underlying compromise and detection methods.
So I was thinking “but they did share their safeguards: they’re using the features of their product that can let you do X, Y, or Z.” Which is an entirely orthogonal point.
I expect that their proprietary techniques underlying all of these capabilities are mostly derived from their insane eyeballs on Internet traffic and their ability to fingerprint and correlate devices and network traffic at scale. I agree it would be nice to hear more about that.
Okta messed this up big time. Companies get breached all the time but how they handle the breach shows the maturity/mettle of the leadership team. This is the second time their comms strategy failed miserably. They need to immediately re-evaluate their incident response operating structure. Cloudfare has some good suggestions.
How do I report a breach to OKTA? Our company has been clearly hacked, and they are ignoring me. This happened through OKTA portal-ServiceNow app.
I also think my iPhone is screwed because I used my iPhone to do the 2FA. It gave me full admin pass, backdoor access, and customer information. Nobody is listening.
For context, this is apparently a regular part of Okta's customer support process [0][1], not something Cloudflare just decided to do on their own. It's the kind of obscure process that would be hard to catch by enumerating specific rules like "don't upload HAR files to customer support even if they ask for them", and it's a technical enough process that you wouldn't expect any random employee to realize that an HAR file contains the keys to the kingdom (much less why that even matters when you're dealing with customer support for your authentication provider who seems to already have the keys).
I think it's pretty fair for Cloudflare to place this in Okta's court. Okta customer support knew what they were asking for and should have had greater controls in place for dealing with those files safely.
> Okta explained that when it is troubleshooting issues with customers it will often ask for a recording of a Web browser session (a.k.a. an HTTP Archive or HAR file). These are sensitive files because in this case they include the customer’s cookies and session tokens, which intruders can then use to impersonate valid users. [2]
In fairness, they posted Okta tokens to Okta support. Cloudflare already trust Okta with their auth anyway.
Sure, it’s still not a good idea for them to give Okta their tokens. But the above, combined with the fact that forgetting to redact a HAR is an incredibly easy mistake to make, makes me tend towards giving them a pass on this one.
How do I report a breach to OKTA, my company is clearly ignoring my efforts to alert them that we’ve been hacked..
I also used the 2FA app because I thought it was the newest thing to do. But, it gave me admin rights, and back door access. Ect. And nobody believes me.
If I am reading this correctly, Cloudflare is just as culpable as Okta. It looks like a Cloudflare employee uploaded the har file with the open session.
Given that Okta’s response in the past to being hacked amounted to a weeks-worth of “nuh uhhh we didn’t, you’re lying” before then quibbling about whether it was really an issue, I think a public shaming is definitely in order.
- Cloudflare shared/detected? it earlier. How long till more Okta customers were breached because they were silent?
- Cloudflare considers authentication as very dangerous. There was a post recently: "hackers login" ( can't find the post anymore?) which means they are very consistent about these types of attacks.
Okta asks for a HAR file of your okta session. By default this includes okta session tokens. The attacker who compromised oktas support systems (again) then used this session token to access customer's okta accounts as the first step in trying to gain access to the customer's systems.
At one former gig “security architect” and ciso didnt even flinch when okta got breached the first time. They still happily migrated to it with much fanfare since it helped them tick a compliance box
This is a naive take. Okta solves both security and non-security problems.
If they keep undermining the 'security' bits they will 100% get ditched. Assuming there is a credible alternative. But I'd argue there isn't a credible alternative at the moment (on either the security or non-security front).
If you listed all of the 'security forward' SaaS companies you can think of. I guarantee 3/4 use Okta. I also bet all would be keen to switch as soon as an alternative was reasonably 2x better on either the 'security' or 'non-security' fronts. Even given the massive pain in the bum it would be to migrate. No one loves Okta or their sales team.
The dumbest guidance your security architect could have given was "Okta got hacked by actors that don't care about us. We should move to ${SOLUTION} that is objectively worse for users and probably worse for security"
> If they keep undermining the 'security' bits they will 100% get ditched.
No they won’t. This is my point - the ‘security’ bit isnt even on the radar of 3/4 of those companies. Compliance is
We were using a different solution at that point and planning a move TO okta. It wasn’t just about the fact it was hacked (happens to the best of us) but how they got hacked, how they found out and how they responded all of which made immediately clear what a nightmare it was/is
First, Okta should get credit for publicly acknowledging the compromise.
Second, expecting any company or organization to never get hacked is unrealistic. Organizations which are transparent and acknowledge their security breaches should get credit for it. Organizations which cover things up should be avoided.
Third, no one knows how to build secure software or services. The closest anyone has gotten is Open BSD but it can still be hacked once you install software on it.
Forth, the number of publicly acknowledged breaches does not tell us anything about Okta’s security. Here is why:
- We do not know how many breaches each authentication provider has had
- We do not know how many breaches were never detected by the authentication provider
- We do not know why the breaches occurred. Breach causes can range from gross incompetence to “WOW, that attacker was really clever and found a new class of security bugs”.
- We do not know how Okta responded to the breach. We also don’t know how its competitors respond to their breaches.
My main point is security is hard and measuring security is also hard. We cannot use simple metrics to determine if an organization is a good organization or a bad one.
>First, Okta should get credit for publicly acknowledging the compromise.
"In early October 2023, Okta was notified of a breach resulting in hackers stealing HTTP access tokens from Okra's support platform by BeyondTrust. Okta CTO Charlotte Wylie denied the incident for a number of weeks, but later recognized that a breach had occurred"
If you cover up security incidents you are facing some actual jail time not just civic suits. Look at what happened with former uber ciso for example of this [0]
I think this was likely a case of a TA getting in with legitimate creds that they obtained from an outside source. How can that be stopped? Happens every day. As someone said earlier - scrub your HAR files and don't leave sensitive data out there. I don't see that this was much of a compromise of a system in that the TA likely got in with legit creds. Where these creds came from is the bigger question.
I get it, I think what is really the egregious part is that they had the power to stop this two weeks prior and instead sat on the disclosure, didn't deal with the security issue, and then their inaction forced their client to act. Luckily CF has resources to detect and deal with this. Most of us probably do not.
This contains a lot of assumptions. Think of this in another context. How many reports do you think HR gets about a breach of company policy? Tons. They have to investigate each one to determine if a policy was actually broken, if the reporter is telling the truth, and what the scope of impact the policy breach had. Meanwhile the offender can continue to keep breaking policy. Now imagine you are an HR Services company managing HR for thousands of companies....Point is it takes time to investigate and validate. Imagine the disruption if they took each report as true on face value.
I would also suggest that each of the companies that publicly posted have something to gain from doing so. We also don't know if they are telling the full story. Using the "we told okta on X date" as assuming that starts the clock on okta not disclosing a breach to the public is a pretty ridiculous take.
We still don't know the full circumstances of how this happened and Okta has not yet publicly commented on their side of this story. Presumably because the investigation is ongoing. But reading the details in the cloudfare post, access for the breach stopped on the 18th and okta told customers on the 19th. Is okta supposed to alert customers of every single report of a breach, every report that might have some credibility, etc...? Maybe there are process improvements to be made in the review process, but we have no visibility into the current level of effort they are making.
Meanwhile, other companies have had known issues for months/years (keyword is KNOWN) before disclosing. I don't want to be a victim of this more than anyone else, but I think we need to be more reasonable in our "hot takes" to these situations even if we are calling for continued improvement.
It’s true. Look at the number of security bugs in Android, iOS, Windows, Linux, Oracle, DB2, SQL Server, PostGre, MySQL, Nginx, Apache, Cisco routers, Intel chips, AMD chips, etc.
The security bugs are not there because of incompetence or stupidity. They are there because security is really hard and it is even harder to get every software engineer to care about security.
If the best organizations in the industry make security mistakes, what makes you think the rest don’t either?
Reality is often unpleasant. It is better to acknowledge it than to pretend it does not exist.
Again, security is important for all of those products, but unlike all of them, Okta has exactly one job.
Perfect security is impossible, but better than 50% odds of never getting critically compromised are reasonable to expect.
If their security is not good enough, their value quickly becomes negative. Not only are they already a gigantic target and a single point of failure, but by being visibly bad at security they are standing in the spotlight with "hack me" on their back.
What kind of argument is this? Sometimes, there are serious bugs in widely used products. So therefore shit vendors should get a pass when their 10 year old Apache instance gets owned? Because "security is hard, mkay"? No! It depends on the compromise.
(If Okta was zero-dayed, IMO we'd have heard about it. Great way to shift blame.)
You can't say they all make mistakes and equate them. How frequent and severe are the incidents? How sophisticated does the attacker have to be to exploit these bugs? After reading all this I'm not inclined to consider Okta.
"Take any report of compromise seriously and act immediately to limit damage; in this case Okta was first notified on October 2, 2023 by BeyondTrust but the attacker still had access to their support systems at least until October 18, 2023."
It is good to call Okta out here as it impacts Cloudflare's business as well and if you can't fix a critical issue for 16 days, that is bad. Remember we are talking about Auth here. A breach impacts everything.
SEC requires public disclosure basically immediately (within a few days. Less than a week for sure) for public companies if a hack could harm your bottom line or trade value.
Hopefully they sink their teeth and give out a nice fine for this insane negligence, but I suspect okta is in for a strongly worded letter.
> We raised our concerns of a breach to Okta on October 2nd. Having received no acknowledgement from Okta of a possible breach, we persisted with escalations within Okta until October 19th when Okta security leadership notified us that they had indeed experienced a breach and we were one of their affected customers.
Do we have anything to suggest CloudFlare is factually wrong? — or was that just random conversational chaff from a brand new account distracting from the stunning incompetence of Okta in ignoring a breach for two weeks?
CloudFlare has more than enough reputation to make such an allegation — and Okta should be cut from any production usage.
Two weeks of failing to address auth compromise is unprofessional conduct by both Okta leadership and engineers.
To be fair, it's also the second time this has happened in 2 years - I don't mean okta breaches in general, I mean it's the second time the support system has been compromized to get access to customer accounts.
First, Okta got hacked and that hack allowed CloudFlare to get hacked. That is bad. Second, one of Okta’s other customers reported the hack and Okta either ignored the report, or investigated the report and did not find the hack. That is not good. Third, Cloud Flare’s response was professional. They asked a company providing a very important service to improve because that company’s product and practices endangered CloudFlare.
If Okta does not want its customers to publically complain about its actions, Okta needs to improve and do better. In particular, if someone says they have been hacked, listen to them and keep digging until you find the problem.
Yes. No one likes a sore winner. Providing your customers with assurances? Good. Providing tips to Okta customers? Sure. Publicly chastising another company you do business with? Unnecessary. That should be kept private. Just my opinion
I am responsible for spending several hundred thousand dollars a year with Cloudflare (out of my budget). I like this style. Don’t want to get called out, get your org fixed. This is somewhere between the third and fifth breach, depending on how you’re counting.
This is the _second_ time this has happened, and it's clear Octa hasn't learned any lessons. So Cloudflare is right to call them out, and Okta should be embarrassed. What surprised me about this post is that they didn't say they were dropping them. Okta is a vulnerability to any organization.
They do win some points on having better security than a popular security product, considering Cloudflare's own security posture is also quite important to their customers.
Agrees - CloudFlare and its employees did outstanding work. My main point was calling CloudFlare a sore winner did not make sense because they did not win anything.
Also, I think CloudFlare’s blog post was very good.
Agree. CF wont have the inside scoop and they use another company's statement to bolster own thoughts. I wonder about the BeyondTrust statement too. This just doesn't sound right....and, so far, although it could happen this week, there have been no SEC filings by Okta - which would have to happen if this was a bad situation for them.
First, CloudFlare is not responsible for people misusing its service. It is not feasible for ANY cloud provider (or company) to determine which customers are legitimate and which are crooks.
Second, reports of fraud do not mean there is any fraud going on. Remember, not every report is legitimate or correct. Lots of people file false reports because they dislike a person or group (sad but true).
AFAIK, it happens when the site claims cloudfare endorse them, which is not true... They were just yet another pay customer, cloudfare endorsed nobody..
Thanks for correcting me. I re-read the article and I stand corrected. Okta is at fault here.
That said, security is a really hard thing to get right 100% of the time. And there is no guarantee that another security vendor won’t have other issues.
But Okta should’ve been more forthcoming communicating this.
Okta has had a bunch of security issues fairly recently and the last one I recall reading was enough for me to not want to have anything to do with the company. This pretty much seals the deal for me.
Compared to other businesses I've worked in, they seem to be having too many preventable security issues, and they're in the security business.
Cloudflare's incident report here is what I'd consider forgivable and sounds like they take security very seriously, but they can't be perfect. Okta looks pretty bad.
