We made a big mistake in our server setup. We bought beefy as app and db servers but went for ultra reliable sun ultrasparks and openbsd for firewalls. This was fine until the DDOS started. After 3 days of many different attacks we now how much better firewall hardware installed thanks to our fantastic colocation which went the extra mile and helped us when we needed their help most but we are now surviving on total brute force.
Our firewalls are faster then the SYN Floods that hit us. This is an arms race that we cannot win in the long run. Yes we can buy more hardware but it's much easier to infect more machines with bots over time.
How do people protect themselves against extortion and malicious ddos attacks? What software / hardware protects the bigger sites on the net?
Every ISP does something a little bit different for large-scale attacks. Some of them off-ramp traffic to scrubbers, some of them have inline devices that can characterize and block SYN floods. My previous employer, Arbor Networks, now has gear deployed at something like 90% of the worldwide tier 1 and tier 2 ISPs that will detect any major flooding event and generate a report which might be helpful to you.
What I'd watch out for is the dozen odd fly-by-night operations that are promising you that, for a monthly fee or a one-time purchase of some $100k box, you can block these attacks on your own network.