Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: How to survive indefinite DDoS attack?
40 points by hknmtt 12 months ago | hide | past | favorite | 59 comments
Say I have a website that is under DDOS attack. Assume it will be going on indefinitely. Assume my hosting provider has DDOS protection that will protect my servers but at the same time users will be unable to access the website.

How can I survive being targeted like this? Cycling IP addresses and updating DNS ad infinitum does not seem like viable solution... because in the end, it's the network attacks that get you, not the application level attacks.




I work for a company in the DDoS mitigation space and there is not nearly enough information in your question for anyone to offer any kind of sensible response for your particular situation.

What is your business? How much traffic "normal" do you get? What is the size of the attack? What is the bandwidth of your upstream connection? Who are your customers? Where is it hosted? What are your acceptable thresholds for false negatives and positives? Do you know who is attacking you and why?

Most every hosting provider will have some sort of DDoS monitoring and mitigation on their networks already. Their response to sustained or repeated attacks might range from scrubbing the bad traffic before it gets to you and not notifying you at all, to reaching out to you to work with you on both ends of the issue, to cancelling your account.

If you just have a fairly simple website that you host yourself, Cloudflare likely a fine option. If you have more advanced needs, you should talk to a more comprehensive DDoS solution vendor.


Who would be more advanced than Cloudflare?


DDoS-Guard, it's basically cloudflare russia edition that is used by controversial websites like Parler, 8chan etc that will get ddosed indefinitely.


I think it's reasonable to call them "less ethical" rather than "more advanced"

Technology wise, cloudflare is probably the leader.


Assuming this is an API and not a website, my approach would be (other than just paying cloudflare to handle it).

Application servers that aren't listed on DNS and their IPs kept secret. A _lot_ of proxy servers that take requests, check the auth, and forward the valid requests to the application servers. For invalid requests, just let them time out.

If checking the auth is expensive enough to cause problems, require some proof-of-work in the request too, and only check the auth if the PoW is there.

In other words, keep all the DDOS traffic off the application servers. Have sacrificial proxies. Make it cheap to handle the DDOS traffic and expensive to generate the DDOS traffic. And then just process the traffic. Spread the proxies across multiple cloud providers if you need to - but ideally ones with free ingress.


That's "building your own cloudflare".

> Make it cheap to handle the DDOS traffic and expensive to generate the DDOS traffic

The traffic is from compromised devices, so it's very cheap.


Even so, the suggestion is a good one. Modern proxies have gotten much better at handling auth and processing the traffic as the GP describes. Service meshes have these features built in now when a few years ago we had to roll our own solutions with haproxy or nginx.


If you can make it take 10x as long to send a request, the DDOS attack becomes 1/10 the size.


Only for some categories - SYN floods aren't limited by responses at all.


That sounds fun for an engineer but if it’s a business then you should focus on your core business and not trying to become a anti-ddos service.


sacrificial proxies make sense although they would all have to be essentially located in the same local network as the target server to avoid routing traffic over different geozones and essentially function like a more advanced firewall. still, it is possible solution and not even expensive.


As others have stated, Cloudflare is the best solution here. Other services like AWS WAF will charge you just for validating requests, which means even if the DDOS is mitigated, you're still paying for it to inspect each request and then block it, which now turns it to a denial of wallet attack.


Pay couple of $ for Cloudflare be done with it.

https://www.cloudflare.com/ddos/

Don't forget to have a fresh IP before doing the move so that your origin is unknown.


I hate that this is the right answer. I hate that I use it too. I hate that I don't have time/cash to explore alternatives.


because alternatives usually cost several thousands $/months..

what has the internet become?

Why not create blacklists (even on a timed basis) with the IPs to block at BGP level? and these blacklists managed by bodies,

- okay they are bodies..., but at least they can be open foundations to which it is not mandatory to join (as in the case of cloudflare)-

available to BGP backbones and routed on a network, castrate the blacklisted traffic?

maybe even at host level be able to make one IP available for normal traffic and possibly one for blacklist traffic using BGP routings to which the blacklist tables are applied?


That basically exists, look at BGP blackholes. Fairly well supported, and it works at what it does, but it can be too blunt of a tool, because it blackholes by destination, not by source.

Border routers are wired to make forwarding decisions based on destination addresses, and not source addresses, so options are limited. Even if you could blackhole by source/dest pair, the distributed nature of DDoS means thousands of sources, which means thousands of rules/routes, which isn't ideal. Some providers might have some capacity to do smarter filters, but it's limited and not very standardized.

If you're dealing with volumetric DDoS, the simple reality is you need big pipes if you want to accept the traffic. Otherwise, cycle IPs and hope legit traffic finds new IPs faster than abusers do. Run your backend communication over a separate network or at least totally separate IP space, so at least you're not losing management capability while under attack.

DDoS mitigation should be a complimentary business with CDN, becuase the traffic flows are opposite of usual, and CDNs generally connect with symetric connections, so what were they going to do with the inbound bandwidth anyway? But that doesn't mean all CDNs run that line of business.


So if someone becomes part of the bonnet, they are denied all access to the internet


Isn't that good though? Unblock them when they fix their host.


(I am not a lawyer) I don't know how legal it actually is, how enforceable it is, and if the whole marker applies those, but my ISP contract has provisions that says I'm not authorized to invade or damage third parties using their internet service.


Doesn't it bring you a tiny bit of concern that this is only current answer people can give?


There are others e.g. ddos-guard, akamai.


How much do they cost? That first one starts at $100/month


Even better, use the excellent Cloudflare Tunnel product to completely shield your origin:

https://www.cloudflare.com/products/tunnel/

I haven’t had a port open to the Internet in well over a year. It’s fast, easy, and even easily supports things like failover (just launch the daemon on multiple machines on the network). Bonus points - setup Zero Trust with their Warp client and get direct access to Cloudflare Tunnel networks and hosts without having to expose them.

Integrates perfectly and smoothly with all of their other products, of course. Going from register/transfer a domain name to full blown CF tunnel, CDN, DDoS, etc comes down to a few minutes of total configuration once you get the hang of it. After that exposing a new service to the internet via Cloudflare tunnel takes 10 seconds - the Zero Trust dashboard takes care of everything, including creating the domain DNS record for the tunnel endpoint.

Free for up to fifty users and what I’ve described here is only scratching the surface.

Cloudflare is really crushing it these days and they’re moving in on everything from AWS to Tailscale. The scale, scope, and tight integration of their products is nearly unmatched. Bonus points for being the cheapest and most performant platform from object storage to functions (Workers). And of course no bandwidth charges on any of this, which can save a tremendous amount of money.


Heck, even Cloudflare free tier is very impressive. That's why it's so widespread.


You can also whitelist the cloudflare IP's and block all other traffic, that way the only traffic hitting your origin is through the proxy.


If under heavy DDOS, it's very likely that your available upstream/downstream bandwidth is already starved, so dropping requests at your public IP level is too late.


If you're under attack now, you should get a new public IP first. Then you can hide behind Cloudflare and drop all unproxied connections. Unless and until you get a new IP, no amount of Cloudflare wizardry can stop all the direct traffic that you're already getting.

If your host's bandwidth is so completely saturated by the attack that no amount of IP switching will help, well, you should find a new host.


good point


but is it far-sighted that to keep the internet as free as possible there are no ways to mitigate a ddos,

and at the same time this freedom is completely lost

when the companies that protect against ddos can be counted on the fingers of one hand?


Yeah. Cloudflare sucks but it's also by far the most effective solution against this type of stuff.


ever since cloudflare went political and started kicking off customers for their opinions, i never touched that service and never will. and i have a very long memory. but alternatives are welcome.


I’m curious which customers you had in mind here. Is it KF?


I can't think of a single service that would have held off against censorship anywhere near as long as CF did, that also doesn't suck.


How to survive indefinite DDoS attack?

That probably depends on who your customers are. I can chime in one one scenario but maybe this does not help your case. If you are B2B then you could just shut off all public IP's and set up some Anycast VPN WAN accelerated meshes with your customers and use a wildcard sub-domain for each customer to access. They won't like having to set that up on their end but then they will have more resilience, privacy and security than they had before in that they can route around internet problems that would have otherwise caused outages or degraded services. Your company would have to provide professional services to help them with the setup and the customers would need discounts to offset the costs of the hardware assuming they do not already have it. The customers end-user devices would join their network over their VPN, not yours. For some B2B arrangements this isn't even a rounding error and removes scrubbing center costs and removes some SLA breach credits. Give this text to your principal network engineers, approve the spend and they will know what to do.

For the customers that do not accept this option, update the contracts and SLA's, then give them a public end-point that has a "best effort" SLA or use a cheap CDN if your contracts permit sending your sensitive data through them. Incentivize them to move to your Anycast VPN WAN accelerated mesh with tighter SLA's and improved security posture by keeping sensitive data off third party data processors like CDN's.

[Edit] This would also be a good time to use one of your companies sub-divisions or set up a new legal entity and get a new AS number with some dedicated IPv6 space and let your attackers just keep attacking the old address space. Announce the old IPv4 addresses on a throw away circuit to gather stats for law enforcement and network mailing lists. Try to do as much mesh peering with that new IPv6 space as you can. This is not security through obscurity so to speak, but rather a small piece of The Art of War.

The marketing team will not be happy but they can distribute their sites to a myriad of CDN's on marketing specific domains for your company. Gotta keep 'em separated, set some smart cache control headers and hope for the best.

An alternative to this semi-complex setup is to pay DDoS scrubbing centers to handle the routing of your datacenter traffic but that can hurt latency sensitive applications. This may be useful for the occasional extortion attack but would be very expensive and sub-optimial(c) in a never-ending indefinite DDoS attack.


There are lots of posts here about using Cloudflare. There's also X4B[1] and Cohmbaton[2] in case you're looking for a non-Cloudflare alternative.

You've mentioned that your hosting provider has DDOS protection (which is usually Voxility[3]/Path[4]) will usually send traffic to your VPS through their scrubbing centers, taking care of the L4 attacks. Ratelimiting and aggressive caching can take care of the rest.

If you're completely into this on your own, consider setting up a number of PoPs which relay traffic to your backend servers, and these PoPs should drop UDP traffic dropped the prerouting level to drop packets without high CPU consumption. There's also eBPF/XDP, but those require much more engineering work and a compatible network card that can run a eBPF program, otherwise you're still using the CPU to drop packets.

[1] https://www.x4b.net

[2] https://www.combahton.net

[3] https://www.voxility.com

[4] https://path.net


I'm on Linode since ~2012 and some time ago they added free ddos prottection to their offering, it was advertised on HN too [1]. Hetzner is also offering something to mitigate ddos.

There was HN thread asking the same question [2]

Having said this, none of my services were targetted by ddos, it made me thinking that maybe maintaining a list of 'healthy' clients IP addresses during good time and then banning anything from outside of that list in bad times? Sounds lame I know, I wait for ideas emerging in this thread :)

[1] https://news.ycombinator.com/item?id=22144369

[2] https://news.ycombinator.com/item?id=12376596


Yeah essentially this. Then have something crawl your database and find IP's that are crawling your dummy pages, and block those. Most of it is EC2/GCP instances and Azure VMs that people spin up with stolen cards, so you have to block a lot of 3rd party vendors. OVH and some others came up often. Lot's of crawling companies were using end user VPNs, so those are harder to block.

The best thing I found was dummy pages to block IPs of bad actors. Also, serving different urls with JS enabled versus disabled, but showing your page as something that works without JS.

Unfortunately, as good as CloudFlare is, their layer-7 isn't going to help you if someone is targeting you.


Cloudflare's layer-7 protection is crap, but it's still orders of magnitude more effective than anything Linode or Hetzner can pull off.

Any major cloud or datacenter can block an old-fashioned UDP flood these days, but botnets have evolved too. Now they speak TLS and HTTP/2, and can send (relatively) small amounts of traffic to select endpoints to generate a large load.

In addition to blocking layer-3 and layer-4 floods, the DDoS mitigation service needs to MITM all your layer-7 traffic in order to determine which requests are legit. Cloudflare can do this (to some extent). AWS WAF can do this. Regular hosting companies can't, unless you use their load balancer and let them manage your TLS keys for you.


why would i care about udp when i run only tcp?


If I saturate your uplink with UDP, none of your TCP is going to get through. Before you have a chance to drop it at your firewall. You have to get your ISP to do that for you, and hope there isn't too much traffic for their uplink.


Cloudflare is the simplest solution, but for the sake of treating this like a system design exercise, can you reasonably assume that there is not an infinite supply of devices with which to conduct the attack, fingerprint the devices, and block them?


Depending on the weight of DDoS, you may simply find it saturates your upstream router so very little valid traffic even reaches you. Or your ISP kicks in their own protection and drops all your traffic.

Edit: https://www.cloudflare.com/learning/ddos/famous-ddos-attacks... largest ever seems to have been about 180k devices. Getting your upstream to accept that many firewall entries is a big problem.


The first "D" in "DDos" means "distributed," which means there _effectively is_ an infinite supply of devices with which to conduct the attack. It's a hard problem, which is why DDoS attacks are so prevalent and difficult to defend against without sophisticated mitigation devices and services. You have to block the "bad" traffic by some characteristic and many DDoS attackers have gotten _very_ good at making their attack traffic look almost identical to legitimate traffic.


My question is, what did people do before Cloudflare? There are different types of DDoS attacks. I vaguely remember what we did over a decade ago, which was blackhole IPs. I'm not sure if people still do this directly with iptables, but at the time, this was the bread-and-butter approach for transport DDoS. If your issue was web application related, then you'd move up to fail2ban.

DDoS protection for web applications is a different beast than raw transport.

If you're a vicim of an amplified DDoS attack, your filtering may need to occur off the server.


what did people do before Cloudflare?

DDoS scrubbing centers. They are still used today but add some latency and are very expensive. There are also some expensive appliances that learn legit traffic and can be put into a defense mode that drops anything it did not learn at a certain volume. CDN's have incentives to be more cost effective as they want to have your traffic 100% of the time. DDoS scrubbing centers are on-demand via BGP changes.


I think about this sometimes. Shouldn't there be a way for your app server to report to your DoS protection service which requests are abusive and which are legit? Like a fail2ban but distributed: if someone connects and sends invalid credentials, block them at the ISP or proxy level, not on your host's firewall.

It's possible that this doesn't exist for good reasons though, I am not sure how the numbers work out in terms of relative capacity.


If your provider has an API for the firewall/protection, just use fail2ban with cURL to block certain traffic.

This can be done with Cloudflare using this action: https://github.com/fail2ban/fail2ban/blob/master/config/acti...


People mention Cloudflare. Google Cloud Armor is another option (we looked into both).

A colleague was DDoS attacked and was able to move the service behind Cloud Armor in a few hours, effectively ending a ridiculous barrage of network requests. I think CA charges a minuscule amount per request, and can be capped with a fix monthly cost which limits the DDoS liability.


but all how are all these services able to let in valid traffic? won't they simply cut me off altogether to protect my infrastructure at the cost of no access for my users?


The entire business model is in not cutting you off, but cutting off the requests. E.g. by forwarding them to the cloudflare captcha first.

This does mean occasional false positives.


Unfortunately, the captcha is a deal breaker for anyone who is running APIs instead of regular, boring, human-accessible websites.

We can barely tell humans and robots apart, and have no idea how to tell good robots from bad robots.


The idea is that a IP under heavy DDOS already has effectively 0% availability anyways. More that 0% is a infinite availability increase.


Given the frequent mentions of Cloudflare in this post, I'd like to offer an alternative suggestion.

Consider migrating your website to web hosting providers renowned for their DDoS protection, such as OVH or Path. By doing so, you can ensure that your server performance remains unaffected even during ongoing attacks.


Lookup edge service providers like Cloudflare or Akamia. If you’re hosted by a large cloud provider then look into their free and premium edge services offerings. Traffic can be filtered to allow legit users, and consider scaling up servers too if cost of downtime outweighs cost of edge services and scaling up.


What kind of DDOS first, then you need to try to isolate the issue, is it from specific region/country? Can you block an IP range? Sometimes you might have a vulnerable service that allow such attacks, and then look into some providers that might help.


Network level or application level DoS? Those are two vastly different beasts.


Have you considered placing your website behind Cloudflare? This solution resolved my DDOS issues. Previously, I was regularly dealing dealing with over 20 Gbps and 5M pps.


Cloudflare is a great and relatively easy to implement solution.

You can enable bot-mitigation and reCaptchas which will help a lot


Information would be nice. You can't properly stop a DDoS if you don't know anything about it.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: