I have a similar experience. It is the emotional detachment from otherwise emotionally salient things that catches people off guard. Sometimes it is interpreted as "dark humor" or "deep cynicism" when the person offering the insight means it only in a realist/stoic sense.
