(Context: part of my job is breaking into stuff all day)
This is a technique commonly referred to as "living off the land" where an attacker makes use of a tool like cloudflared to conduct an action that would otherwise be blocked by security tools. It makes the defenders job so much harder because you now need to differentiate between your devops team being cool and a legitimate threat inside your network by looking at the exact same indicators generated by the two. Looking for things like unsigned applications making outbound network connections are removed from the defenders toolbox.
Yes, cloudflared does the same thing as ngrok. You'll also find that ngrok is blocked in most corporate environments as well for posing an equal risk. As an attacker, you have a good chance of setting off alarms that (should) specifically detect ngrok.
I think the point of this post it to highlight that cloudflare tunnels need to be block by default as well and only allowed when there are specific approved use cases.
> If an attacker is capable of installing apps on your server... you've already lost.
That is a deep misunderstanding of how security works.
The defender’s dilemma states that breaches are inevitable because defenders have to be right 100% of the time whereas attackers only have to be right once. This is why you have to defend in as many layers as possible.
A better way to approach it is to assume that at any given time an attacker can execute code on your systems (because they have knowledge and access to unreleased exploits) and work on ways to detect anomalies and behaviors indicative of compromise as well as limiting the blast radius.
> If an attacker is capable of installing apps on your server... you've already lost.
You're right, but cloudflared is a whole lot easier to get running than previous RATs, and attackers don't need to operate their own c&c infrastructure - you can't defend yourself from such attacks by blanket banning Chinese, Russian, Iranian and North Korean IP addresses (which I honestly advise everyone to do if they don't have business in that country), and you can't easily block outbound to Cloudflare either as half the Internet is hiding behind them.
Basically, cloudflared lowers the price and effort for attackers dramatically, while the effort to defend against this threat model has now risen significantly.
If Cloudflare were to actually be willing to do something against being used by scammers, they'd put the ingress IPs for the C&C infrastructure on dedicated IP ranges and publish these in a machine-readable format so every reasonable person that does not use Cloudflare tunnels can ban them.
This kinda surprises me, what is so "dramatic" about cloudfared specifically? This seems just like another reverse tunnel tool, and there are plenty of them.
I am not working with malware specifically, but in the past I've used ssh tunnels, random one-off websocket thingy we wrote, wireguard tunnel, frp proxy, and even AWS SSM agent to get access to machines with all incoming connections blocked. They are pretty simple to setup and generally cannot be blocked with whitelist block already.
(and I bet that for malware, they are worse than cloudfared. Based on CF's reputation, they take security reports seriously, so I would not be surprised if they take down malicious tunnels fast. While random VM on low-cost hosters will probably takes days to respond.)
> This seems just like another reverse tunnel tool, and there are plenty of them.
These are known though, you can block them without causing issues.
> and even AWS SSM agent to get access to machines with all incoming connections blocked
SSM is awesome, but the ways it works... I have no idea how. I think though that it uses outbound connections, but unfortunately AWS SGs can't do deny rules.
> If Cloudflare were to actually be willing to do something against being used by scammers, they'd put the ingress IPs for the C&C infrastructure on dedicated IP ranges and publish these in a machine-readable format so every reasonable person that does not use Cloudflare tunnels can ban them.
I don't like the idea of making it easier to block certain services, because it goes both ways: it'd also be easier for bad guys to block good guys from using said services.
I'm not seeing how the GP's idea could be used by bad guys to block access to services - it'd be something you could add to your firewall, if you don't use this service, to prevent it's use in gaining persistence.
If the bad guys can modify your firewall config, you already have a problem.
"Winning" and "Losing" are not a great framework for this sort of analysis. In that binary framework, there's no difference between "1 bit of sensitive data was leaked" and "the medical and financial records of every person ever, nuclear launch codes, and the backdoor that causes every nuclear reactor to go critical simultaneously were leaked" - both are just "losing".
This is why various compartmentalization techniques exist throughout the stack - principle of least privilege, network segmentation, acls, and so on. If something is compromised, limit the "blast radius" of what can be done with that compromise.
The other part is containment. By analogy alarm systems and guards. If the alarm system goes off and says an intruder is in the building, well by the "winning and losing" framework, the defender has already lost. (Of course by the binary framework, you don't even need an internal alarm system - once the breach occured you've lost so why bother?). Or you could try to contain the problem and send guards to stop the intruder before they do too much damage.
This is where the cloudflared usage described in the article is problematic. It's a tool that is less likely to trip alarm systems, and further can provide a wide range of access while doing so[1]. If it doesn't trip alarms, a small breach leaking a little data can turn into a big breach leaking lots of data. It absolutely should be considered when designing networks, security methods, etc.
I don't know what the solution here is, that probably depends on the individual/org doing the threat assessment, and probably varies by environment within that domain. Should the tool be blanket banned? Probably not it's also a really useful tool for a lot of people in a a lot of situations. Maybe cloudflare can split the tool in to individual binaries for functionality and have the tool called cloudflared just be a wrapper. Maybe they can change it some other way.
[1] this tool is used for a lot of stuff and may already be on the systems in question, so it's not even installing a program -- just execing it. Since it is used for a lot of different things, it may just be on a blanket "allow this app" list. And so on.
The big thing that I found from my research on this tooling was the ease with which an attacker could a) learn things about a target environment, particularly by enabling RDP and accessing a device directly without generating any red flags and b) more importantly, being able to exfiltrated data over SMB that would appear to be local activity.
Sure, flow data would eventually show the outbound transfer rates suggestive of data exfiltration, but without any other correlative alerts, it's likely to get through before defenders have a chance to respond.
I find that more and more CloudFlare is so ubiquitous that we have to use CloudFlare tools to protect ourselves from other people attacking us via CloudFlare. Here's an example of a rule we had to put in place to block people using CloudFlare workers to scrape pages and bypass security. CloudFlare doesn't seem to care about this kind of abuse (or maybe they do but aren't talking about it publicly).
(!cf.bot_management.verified_bot)
and (cf.bot_management.score lt 10 )
and len(cf.worker.upstream_zone) gt 0 and not cf.worker.upstream_zone in {"<zone>"}
and (not ip.geoip.asnum eq <exception as>)
We definitely care about abuse generated by Workers.
A request coming from someone else's Worker should not "bypass" any security -- it should be treated the same as if they were running their bot on any other cloud provider and making requests across the internet. If you find otherwise, please file it as a security vulnerability: https://hackerone.com/cloudflare
(Actually, it should be slightly harder to run this kind of abuse from Workers, because any request coming from a Worker has the cf-worker header identifying the domain that owns the worker. It looks like your block rule is actually taking advantage of this. Other cloud providers usually cannot inject headers like this so it's harder to trace and block abuse from them.)
Thank you for your response. Since I have your ear, I'll provide some direct feedback.
"We definitely care about abuse generated by Workers."
That may be the case internally but it's not as evident through support. I base this on their ability to answer security-related questions, diagnose odd behavior, or mitigate problems. FYI, we are paying for your highest tier of support.
"it should be treated as if they were running their bot on any other cloud provider and making requests across the internet."
Part of this is how Site Analytics and WAF analytics represent worker data (or don't). Even though the worker modifies the contents, the IP is passed through as the end-user IP (at least the way we use them). These analytics systems need to do a better job of identifying anything worker or tunnels related. If this were being done through AWS it would be evident. We can use various mechanisms to mitigate this, but I wanted to clarify why abuse through Workers is different for our implementation.
Also, the process to block non-zone workers was undocumented, and it took a couple of weeks for someone to dig it up and only after several trials and errors. Support wasn't sure it would work, and there was no other documentation (at the time).
We have yet to get an answer about the high volume of tunnel requests from specific goes.
Combine this with the opaque stack you have that causes odd situations. For example, I added a worker that automatically retries 500 errors from the origin. Simple enough. This breaks Zero Trust, though, and causes a redirect loop. There may be documentation somewhere, but there isn't a good breakdown of how Zero Trust fits into a customer-defined worker, and a redirect loop is undoubtedly a poor failure mode.
There is a world of difference between the information you have and may provide as the Technical Lead of Workers, and what I'm getting via support or raw documentation.
I could provide you more information offline if you want.
Edit: Let me just add the other recent thread [1] regarding redirect loops in verification is par for the course. We had the same issue and tried for weeks for progress or resolution. The issue was never resolved, nor could we get information on why the failure mode was so poor, etc. That ambivalence adds up to expecting general ambivalence or inability to make meaningful progress. At some point, you say to yourself, "Why waste my time talking to support and putting in much effort and getting the run-around."
Sorry to hear about your difficulties getting good answers from support. I will pass along your feedback.
> Even though the worker modifies the contents, the IP is passed through as the end-user IP (at least the way we use them).
That should only be true when your own worker makes a request to your own origin. It should never be true for requests coming from someone else's worker. Again, please file a security report if you found a case where that is happening.
> CloudFlare doesn't seem to care about this kind of abuse
I'm sure it's just a coincidence that they turn a blind eye towards obvious bad actors that just happen to increase the demand for cloudflare's services
It's a bit more pernicious than ngrok just because it's more difficult to isolate and block only malicious usage of the feature. But yeah I agree it's not breaking news that implants will find ways to tunnel out of your infrastructure when exfiltrating data...
Doesn't ngrok require a local config? This technique allows attackers to maintain a nonfunctional tunnel and enable functionality only when they want to engage with the victim machine, similar to TA569's SocGholish campaigns across news sites last year.
As an enterprise cloudflare customer, we were interested in using this product for legitimate purposes but also block internal non-legitimate access and asked them for advice how to do this.. Support wasn't able to offer any guidance.
Hi from the Cloudflare Product team - blank stares shouldn't be the response there. Can you send me a note, srhea AT cloudflare, and I'll make sure you get a response?
Hi Sam, thank you for the constructive reply. I've toned down what I said. I'm not able to talk about the specific example, but are you able to provide guidance for this generic issue of how organisations could block entirely this service at the egress edge and/or make it selective for approved/unapproved usage?
Hmm, risking sounding like the dropbox-is-only-ftp-and-svn guy but shouldn't any threat actor worth their salt be able to use any array of open source tools to do this? Especially if you already have access on the machine and the possibility to place binaries there?
The biggest risk I see is if the target is already using this for legit use cases, since I guess it would be really difficult to discern between the two.
If an attacker is capable of installing apps on your server... than anything is possible. Don't know why mention Cloudflared other than clickbait. O
Overall I'm huge fan of CF Zero Trust and tunnels. I wish documenation and examples were clear, but form a security stand point CF is one of the best security solutions we use.
Two reasons I conducted this research. Firstly, because we saw its use in the wild after exploitation occurred. Second, because it poses a significant risk for data exfiltration.
Many tools can be used for this effect, but the fact that the configuration can be easily modified attacker-side through the cloudflare dashboard, and only the token for the tunnel is exposed "client" - side poses a serious challenge for defenders, especially IR teams conducting post-breach forensics.
Yes, their docs, everywhere, leave something to be desired. For example, don't put worker code snippets with little context. Always post an entire file that the user can expand and collapse. Snippets assume the user knows things that they might not.
I'd like to just say that I (the blog author) am not saying Cloudflared is bad. In fact my research on it makes me really want to test out its usefulness for some of my personal projects.
But it is important to demonstrate the ways in which this seemingly benign tool can be (and has been) used to conduct nefarious activity if not properly detected and defended against.
This is a technique commonly referred to as "living off the land" where an attacker makes use of a tool like cloudflared to conduct an action that would otherwise be blocked by security tools. It makes the defenders job so much harder because you now need to differentiate between your devops team being cool and a legitimate threat inside your network by looking at the exact same indicators generated by the two. Looking for things like unsigned applications making outbound network connections are removed from the defenders toolbox.
Yes, cloudflared does the same thing as ngrok. You'll also find that ngrok is blocked in most corporate environments as well for posing an equal risk. As an attacker, you have a good chance of setting off alarms that (should) specifically detect ngrok.
I think the point of this post it to highlight that cloudflare tunnels need to be block by default as well and only allowed when there are specific approved use cases.