I dispute the notion that Slowloris's behavior should be considered an "attack". If your server is hammered by tens of thousands of visitors who are on slow links - say modems or 2G wireless links - then you will have the same problem. The real problem if you ask me is the limited amount of I/O concurrency that multi-threaded and multi-process servers can have. Pretty much the only practical solution for this right now is by using evented I/O.
The problem can also be solved by putting the web server behind an evented, buffering reverse proxy and fully buffers both the request and the response. This shields Apache from slow requests.
Maybe some time in the future operating systems will be able to handle millions of operating system threads easily but we're not there yet.
The default thread count for Apache is 256 is it not? I know we can't handle millions (and I can't speak for how Apache itself with threads) but servers running thousands of threads are not only possible, but exist and work well. Whichever way you do it (threads or events) at some point you will hit a limit.
I see Slowloris as an "attack amplifier" - it's a way to do a lot more damage with N connections than a straightforward DDOS. In some situations, it's possible to deny access to a server with just just one.
> The real problem if you ask me is the limited amount of I/O concurrency that multi-threaded and multi-process servers can have. Pretty much the only practical solution for this right now is by using evented I/O.
Why do you think that? What constraints do you believe limit the number of threads you can use?
- An idle thread is essentially free.
- The memory used by an ongoing request should not be inherently different between an evented server and a threaded server.
- The number of sockets an evented server can use is the same as the number of sockets available to a threaded server.
If evented servers are handling this better, all it shows in my opinion is that the threaded servers have been written incorrectly or else configured incorrectly. But that is not an inherent benefit of evented servers. Please correct me if there is a constraint I am missing.
> Why do you think that? What constraints do you believe limit the number of threads you can use?
Virtual memory address and context switching overhead. On 32-bit platforms, if each thread has an 8 MB stack then after creating a few hundred threads you run out of VM address even if you don't actually use that much memory. Most OS schedulers also don't like dealing with tens of thousands of threads. Furthermore each kernel thread takes a small amount of space but kernel memory typically isn't swappable, unlike userspace memory.
By context switching overhead, I assume you mean the overhead of faulting data into the cache from a thread that had been sleeping. Evented systems have this overhead too when switching between events related to different requests.
Stack size I've discussed in another response. It can and should be decreased if you plan on working with a lot of threads.
I can't comment on the Mac OS X scheduler. The Linux scheduler handles it just fine, and my understanding is that the Windows one does OK with it too.
The small amount of kernel memory that isn't swappable likely isn't your system's overall throughput constraint, but if it is I stand corrected.
One constraint I believe you may have missed is that an idle thread does actually consume a significant amount of memory. While most resources a thread consumes are fairly small, the stack is actually quite large and is allocated up front. The usual default ulimit stack size for Linux is 8192Kb. This means for every thread created, 8 megabytes is allocated whether or not it's used. On Linux the OOM killer will start to kick in, by default, once you've overcommited your virtual memory by 50%. You can adjust the ulimit stack size and overcommit ratio, of course, but either way you're playing with fire.
- 8MB of virtual memory is allocated. In practice, only a page or two will be allocated to hold the base of the stack, and of course any memory required for the request's data.
- You can decrease the default stack size, and would if you were trying to work with a lot of threads.
(Disclaimer: my assertions apply only to Linux, I'm not familiar with the vmm in other systems).
I think you may be misunderstanding how the vmm works. When you allocate memory by calling mmap(), sbrk() etc., that amount of memory is committed whether or not you actually use it. Because many processes will allocate more than what they actually use, for example excess stack space, the vmm allows you to overcommit by a certain percentage, defaulting to 50%. This will allow processes to allocate 50% more memory than the system actually has. If that is exceeded, however, Linux has an Out Of Memory Killer which will start killing processes.
Say many Slowloris clients attack at once. There reaches a point where threads are much less free than events. Newer concurrency systems will handle thousands or millions of lightweight processes without firing up large numbers of native threads, and there's a reason for that design decision.
> There reaches a point where threads are much less free than events.
Why?
> Newer concurrency systems will handle thousands or millions of lightweight processes without firing up large numbers of native threads, and there's a reason for that design decision.
I would say that is based on the historically accurate but currently false notion that OS threads are expensive to start up or keep idle.
It's still true -- I recommend looking at some comparisons. Worker-thread based http servers consistently have worse performance at handling a large number of connections, and the error rate for those (simply) using threads is also higher at large number of connections,
It'd be really nice if karma could be earned on comments from autokilled users, and eventually undo that. The permaban seems excessive in maybe about half the users I see.
That's because either Paul or one of the mods hellbanned him. When you're banned on Hacker News, you don't get told, because that would make you get a new account and start posting without having learnt your lesson. So instead they just auto-kill your comments to prevent other people from seeing what you're doing. (Sometimes they're polite and email you to let you know you've been banned, which is nice of them. Other times they just let you keep posting away to nobody.)
This is a really disturbing characteristic of Hacker News moderation. The people it happens to must feel like Bruce Willis in The Sixth Sense, when they realize it. Why not let them know??
I believe the idea is that it's for people who are legit trolls. If you let them know, they'll just create another account. But if they think people have just grown tired of their antics, they're more likely to move on to another site.
I'll be honest, and hope that my comments and honesty don't earn me the silent treatment. I am iteration 2 of a formerly silenced commenter. I learned a lesson or two and consider my comments more carefully before I post them, but it was infuriating to find out that 2 weeks worth of interesting comments weren't receiving responses because no one was reading them.
I am surprised you stuck around, to be honest. I'm not sure I would have, and I'm sure many don't. Could be argued that's the point, but it doesn't feel quite right.
They (nginx, lighttpd) are more difficult to exploit, especially when used to buffer requests to a heavier upstream server like Apache.
EDIT (Clarification):
They (nginx, lighttpd) are more difficult — although not impossible — to exploit, especially when used to buffer requests to a heavier upstream server like Apache.
Specifically, they are typically able to handle many more connections than your application server would be able to (as long as they are properly configured), without the incurring the resource overhead of your application server by bufferring the HTTP request/response.
Nice job editing your comment without saying so (" -- although not impossible --" and your last paragraph).
Anyway, your nginx/lighttpd server is more likely to be exploited and compromised via an actual vulnerability rather than your Apache server via a slowloris-style attack. It's akin to putting a wide receiver in front of your runningbacks...
I updated the original comment w/ the original and updated text. Sorry if you saw this as inappropriate — I had updated it before it was upvoted and thought it only clarified the points, not changed it.
Regardless, many deployments use this same technique for just this reason (to avoid spoonfeeding slower clients responses). How is this any more risky than running Apache up front, barring configuration errors (which could just as easily happen with any other server software)?
I wish they could have picked a better name - 'Slowlaris' (note minor spelling difference) has been a colloquial name for the older, but more common versions of Solaris, due to the badly performing IP stack for some time.
Apache has many differing mpms - you can use an evented mpm similar to how nginx handles connections. Anyways since this 'attack' is making a full tcp connection a simple fix would be to limit the amount of full tcp connections from one source.. or if you feel particularly evil add it to a blacklist of tarpitted connections and then reverse DOS the attacker.
See also "A. Kuzmanovic and E. W. Knightly. Low-Rate TCP-Targeted Denial of Service Attacks (The Shrew vs. the Mice and Elephants). In Proc. ACM SIGCOMM, 2003." and related papers -- http://www.cs.northwestern.edu/~akuzma/rice/shrew/
It's important to note that you can protect against Slowloris itself not only by the various apache modules available specifically for Slowloris and specifically for this kind of HTTP attack, but also through Apache's native configuration settings that (among others) govern the number of simultaneous connections any single IP is allowed to have. Slowloris itself is not much different in terms of the effect it has on the HTTPd than a script pulling data from the server using curl or wget.
I don't know how the Apache modules to guard against Slowloris work, but I can think of a modified attack that would still work if connections per IP were limited. You can't limit connections per IP to just one, as browsers will pipeline requests, they may have multiple tabs open, asynchronous requests, &c . . . 256 is obviously very high, let's say you set the limit to 10 simultaneous requests per IP.
It's now impossible for a user to Slowloris your webserver, but they only need to get hold of 26 separate IP addresses to be able to once more. Depending on your setup, this may be far less than they would need for a naive DDOS attack.
I think a way to mitigate both attacks would be to limit how long the client can send headers for (and perhaps refuse connections for X amount of time if a client repeatedly acts in a way that appears malicious, but that's possibly beyond the scope of "native configuration").
First off, as others have pointed out you can successfully run thousands of threads, so the default 256 does not mean much. Also, HTTP pipelining is not what you think it is: http://en.wikipedia.org/wiki/HTTP_pipelining
I know, I was one of those people - I just wasn't sure if Apache can do that well, I know of other servers that can. And yes, you are correct, that's what I get for posting when tired - I meant simultaneous requests!
http://www.darkreading.com/vulnerability-management/16790102... http://www.owasp.org/images/4/43/Layer_7_DDOS.pdf