Even music without lyrics is distracting to me, but lyrics (if in a language I understand) definitely make it much worse.
Oddly enough, non-musical sounds (like birds or cars) don't affect me at all. I think it's the complex pattern-matching in music that is a problem for me. Of course, that's also what makes it so delightful to listen to.
For me it's more that the language processing centre in my brain is strongly single-threaded. Some non-verbal sounds do engage it, but there's a complexity threshold. Magpies chattering to each other do. Some classical music does, but not all by any means.
Oddly enough, non-musical sounds (like birds or cars) don't affect me at all. I think it's the complex pattern-matching in music that is a problem for me. Of course, that's also what makes it so delightful to listen to.