As a life long anxiety sufferer, I identify with your interpretation.
My anxiety devolves into panic attacks if I try to deal with it consciously - self talk, rumination, avoidance. But it goes into a dormant state when I don't engage with it consciously and just let it be.
Yeah I had a couple rough years with panic attacks, high anxiety state (always been anxious but not to that level). What you start to realize is that you are actually scared of the anxiety itself, not the situations so much. So, you sort of get caught in loop of being scared of your own emotions. The treatment is simple, but takes work and a lot of habit forming through time... but essentially you just stop thinking of the anxiety as an emotion you shouldn't have or should fear.
Like you said, just let it be and stop caring about it. Anxiety can be a good thing, it's a normal emotion. Any attempt to stop it or control it will end in disaster. Let your body/mind do it's thing. Emotions are fleeting and they will always come and go. Over time you sort "re-wire" your brain to stop panicking every-time it feels a fleeting anxious feeling.
My anxiety devolves into panic attacks if I try to deal with it consciously - self talk, rumination, avoidance. But it goes into a dormant state when I don't engage with it consciously and just let it be.