The source mentions in a reply that they were able to reproduce this exact text multiple times through different prompt injection approaches, right down to the typo "you name" rather than "your name", which seems unlikely to happen if it were making it up out of thin air.
I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.
That seems to be a fundamental misunderstanding of what LLM hallucinations are?
A hallucination, when it comes to LLMs, just means "the algorithm picking most likely next tokens put together a string of tokens that contains false information". It doesn't mean the LLM is having a novel false idea each time. If the first time it hallucinates it thinks that that misspelling is the best next-token to use, why wouldn't it keep thinking that time and time again (if randomness settings are low)?
Because for practical purposes they just don't make grammatical or spelling mistakes like that.
Obviously they're a black box so it's possible there could be some very rare edge cases where it happens anyway, but it'd be a complete fluke. Changing the prompt even superficially would essentially cause a butterfly effect in the model that would prevent it from going down the exact same path and making the same mistake again.
I wonder if "you name" is a load bearing typo that breaks something else if corrected, so they left it in on purpose.