Shouldn't the child pool simply be reduced and the queue increased which would have the same effect as the slow MySQL database?
Requests should spend their time queued rather than letting the operating system thrash between processes, queues are stupid anyway in the context of webserving, they should use stacks as the best case scenarios are largely the same, and the worst case scenario for a stack are much better than a queue. (Yes, stacks are unfair, but it's better that a few people are served than no one in the event that the server gets bogged down.)
The issue is that the child pool now has too many children and the children spend their time thrashing, before it was acceptable because the children were mostly doing nothing. This sounds much more like an apache configuration issue than anything else. I wouldn't point the blame at 'fast' MySQL, but easy to misconfigure apache architecture.
Or simply frontend apache with nginx, and limit the number of backend connections. If you're using a fork/join apache which it sounds like are you, you'll get much better performance from this anyway. Setup this way apache can dump the output directly to nginx which can then spend time spoonfeeding it to the client.
I submitted this a few days ago, but he could use a very simple Object Pool to handle this. Set up a pool with 3 or so Ajax request objects and make requests to that pool.
Here is an example where I setup four MooTools Fx.Morph objects. Then I set up 400 divs that whose width will be modified using the next available Fx.Morph object in the queue. It is pretty simple, powerful stuff. Your objects must be reusable though so I dont think jQuery's built-in ajax/animations would work (I use this in "production" with MooTools' Request objects)
you simply get an instance of Pool, add your actions, by calling pool.act(fn). In your callback you'll receive one of the objects in the pool when it is ready. You use it, and when youre done, put it by by calling this.add(obj)
pool = new Pool();
pool.add(new ObjectInstance());
pool.act(function(obj){
//use obj and put it back
this.add(obj);
});
Since you got rid of the MySQL bottleneck, it seems your server architecture now simply isn't able to handle higher concurrency. If you suddenly got 5-10x the number of users, you'd have the same problems as before adding your AJAX queueing, right? That's, of course, assuming nothing else "breaks" first due to the increased traffic.
a legitimate concern from someone who's hellbanned:
No offense intended here... but it sounds like you could easily become the victim of a DoS attack. All that's needed is for someone to create their own front end and you'll be back to normal.
Have you investigated some sort of solution on the server side, and not the client side?
It isn't nearly that simple- they'd have to create the custom front end and get everyone to use it. I doubt one person constantly hitting the site would have any issue, it's the combination of all of his users doing it that caused the problem.
Yeah, but if you want to do a DDOS attack you can just use one of the already configured tools out there that will bombard the server with as many requests as possible. A few chained ajax requests are never going to replicate that sort of load.
I guess your next step should be to profile the apache reqs?
Unless you know that you're doing CPU-heavy work in your apache procs (and it's unavoidable or already optimised), it seems that you could get back the responsiveness of your site?
It seems surprising that you're CPU-limited on the web head.
I've had a vaguely similar occurance once that took forever to track down.
When I would persist a particular model back to mysql, I would fire off a job in Gearman, that would then grab the saved object and asyncronously send it off to lots of other sources.
After moving servers it would always send the previous state of the object. Having previously worked flawlessly.
Eventually I trackked it down to the fact that my Gearman job was being kicked off before the actual write to MySQL, so then when I did the select in my job process, it would grab the previous snapshot.
I once had a similar issue with a Silverlight program I wrote. In my case I ended up adding worker threads to handle data processing with an event to signal when it was done processing and followed up that tweak by consolidating my SOAP requests as much as I could (ironically making it even faster helped fix it). Timing issues can be rather tricky sometimes but the upshot is that if your going to have a problem, running too quickly isn't a bad one to have to deal with.
When I load a dynamic page with content that isn't ready, I push the missing content requests it into an array. When the page is done loading, it polls the server with the data it needs. Server replies with data and client keeps polling every second or so for more stuff until it's done.
I second this type of approach - when I developed an iOS app that needed to display the results of multiple long-running API requests made on the server-side, I had the app make a request and then wait for data to be pushed to the device on a persistent "back-channel" connection the app kept open. This way data could be displayed as it was available, but the app didn't need to poll or open multiple TCP connections (expensive on EDGE, which was common at the time).
The client makes a lot of ajax requests since it's so ajax heavy and there is no throttling on the concurrency. When 5-10 requests hit the server from the same client at the same time it is a "little" bit like a dos attack.
Any idea why that's pegging the CPU? My naive analysis would be that once the database has sent/stored the data, the server is just doing some light filtering/serialization and sending it off to the client. But obviously there's more going on if these requests (en masse) can get you to 100 load.
Shot in the dark: Is there some kind of sorting going on?
Essentially, mySQL was the bottleneck. He fixed that, then Apache became the bottleneck. He just didn't realize mySQL was his bottleneck (or that fixing that would increase the load on the webserver) and therefore "blames" the functioning of the app on mySQL's slowness. Now, whether or not there are changes to Apache he could make to avoid having to queue ajax requests is not something I can speculate on.
Requests should spend their time queued rather than letting the operating system thrash between processes, queues are stupid anyway in the context of webserving, they should use stacks as the best case scenarios are largely the same, and the worst case scenario for a stack are much better than a queue. (Yes, stacks are unfair, but it's better that a few people are served than no one in the event that the server gets bogged down.)
The issue is that the child pool now has too many children and the children spend their time thrashing, before it was acceptable because the children were mostly doing nothing. This sounds much more like an apache configuration issue than anything else. I wouldn't point the blame at 'fast' MySQL, but easy to misconfigure apache architecture.
Or simply frontend apache with nginx, and limit the number of backend connections. If you're using a fork/join apache which it sounds like are you, you'll get much better performance from this anyway. Setup this way apache can dump the output directly to nginx which can then spend time spoonfeeding it to the client.