If im understanding GP correctly, I think what's happening for them is that the extended quiet daily makes their hearing a bit more sensitive (as their brain and ears work in tandem to amplify the sound and be more receptive to it, respectively). Now when they take off the headphones, since their hearing is more sensitive, they pick up more of the actual background noise that exists around them.
