I have the same worry when using cruise control. What I've trained myself to do is to cover the accelerator pedal when overtaking other cars or in any situation where I might have to brake. That way the car is still maintaining speed for me but I'm ready to intervene, to get back the safety margin. With adaptive cruise it may even increase the safety as both me and the cruise control can both brake.
