How about the fact that llm's don't work unless humans generate all that data in the first place. I'd say the llm's energy usage is the amount it takes to train plus the amount to generate all that data. Humans are more efficient at learning with less data.
Humans also learn from other humans (we stand on the shoulders of giants), so we would need to account for all the energy that has gone into generating all of human knowledge in the 'human' scenario too.
i.e. not many humans invent calculus or relativity from scratch.
I think OP's point stands - these comparisons end up being overly hand-wavey and very dependent on your assumptions and view.
