Yea, so I might have believed you except I can ask GPT-4 to step by step explain its reasoning. It is really weird to say it doesn't understand but "I do" when the response it can give is better than the average human would give to prove they understand.
You might say it is just predicting based off of old data it has, to which I say this sounds like a semantic jostling. What is "understanding" then in human beings if not us doing some form of predicting off of old data we have in our brain?