And god help you if that starts your autoscaler to flapping.
Too much CPU load, start some servers. Too little, kill some servers. Oops now CPU is too high, bring them back. No, it's too low...
It reminds me of the old UI bug where you get an off-by-one error for scrollbar logic and so it appears and disappears in a cycle until you resize.
Edit: I think we want something akin to a learning thermostat, but for servers. Figure out that I get a CPU spike at 5:20 on week days and spin some servers up at 5:10. Then spin em down at 8:00 when EST starts to go to bed.
> .. Oops now CPU is too high, bring them back. No, it's too low...
You just described closed-loop control system oscillation. Cause is wrong gain and/or delay parameters. AWS ASG has some knobs for tuning like Cooldown, MaxSize. What you described above is most probably long boot time (delay) problem. Service should be ready in less than 30 secs after boot. To get there, one should bake AMIs, not install afresh all the stuff on boot.
You can get an oscillation without long boot or ramp-up time but they sure don't improve the situation.
Lazy loading of modules is a common cause, but any other resources that are loaded on demand or in the background.
But that's more of a situation of thinking you need two additional nodes, getting three, settling back to two after warmup, and then killing them all off again when the spike ends.
I may have said elsewhere that I'm more comfortable with scaling load based on daily and weekly cycle patterns, with perhaps a few exceptions for special events (sales, new initiatives, article publications) and making it easy for someone to dial up or down.
To use a car analogy, get really good at power steering and power brakes before you attempt cruise control, get really good at cruise control before you attempt adaptive cruise control. And don't even talk about self-driving unless you're FAANG.
Too much CPU load, start some servers. Too little, kill some servers. Oops now CPU is too high, bring them back. No, it's too low...
It reminds me of the old UI bug where you get an off-by-one error for scrollbar logic and so it appears and disappears in a cycle until you resize.
Edit: I think we want something akin to a learning thermostat, but for servers. Figure out that I get a CPU spike at 5:20 on week days and spin some servers up at 5:10. Then spin em down at 8:00 when EST starts to go to bed.