+1 to this. I switched to a Kinesis ergo and typed qwerty for a while, then switched to Dvorak. When I got my Moonlander I kept it Dvorak and haven't had any issues with going back to qwerty on normal keyboards.
I switched to Dvorak on regular staggered keyboards and got to 80-90 WPM or so, but pretty much lost my ability to QWERTY.
Then I switched back to QWERTY full time for 5+ years.
Then I switched to Norman on an ErgoDox, getting to 80-90 WPM or so again. This time around, I could still QWERTY on a staggered keyboard just fine. Night and day vs. the first attempt.
(Then I switched back to QWERTY again. Now I can't Dvorak or Norman or ErgoDox!)
I switched off of Dvorak because of vim, and because keyboard shortcuts in general were so awkward. Shortcuts are better in Norman, but vim is still awkward.
For me, while I did learn colemak on a different physical layout, my typing speed on qwertz still plummeted. Maybe the difference wasn't large enough because it was just a columnar layout and not a split design.
I spent a couple of months using both regularly though, and now I can comfortably switch between them.
I switched to an ergodox when they came out with the Norman layout. (I started using a Lily 58 earlier this year with the same layout.)
I can still do decently on staggered layouts (with qwerty) when I need to.
I'm not a musician but I figure this is similar to playing different instruments.