Yes it does, so there's no reason to mark it up. The failure of the semantic web was trying to lower human text to the level the machine could operate on it. The success of LLMs is to raise the machine to level it can operate on ordinary human text.
You mark it up with the LLM so you can do something with it. A LLM like GPT-4 is still just emitting text tokens, it's not also generating pixels to push to your monitor. If you want to do something involving semantics like part-of-speech tagging, you'd obviously use the LLM to parse & annotate it, and then do stuff like use CSS to style nouns differently from verbs, or whatever. Maybe you don't distribute that, but run it locally, but you still have to do that at some point!
