Actually the load balancer approach can be further simplified. Just have a shorter health check on the server, like 3 seconds. When the server is in long GC pause, its health check request from the load balancer will fail. The load balancer automatically takes it off the pool and no traffic reaches it. After GC completed, the server's health check succeeds. Load balancer adds it back to pool automatically.
It's just a simpler solution to have a group of servers with enough capacity accounting for the random GC pauses. Having short health check enables failing fast which is generally better for the users overall - their requests don't stuck in overloaded servers for a long time (GC or otherwise caused).
It's just a simpler solution to have a group of servers with enough capacity accounting for the random GC pauses. Having short health check enables failing fast which is generally better for the users overall - their requests don't stuck in overloaded servers for a long time (GC or otherwise caused).