As a non-vi user, I feel very excluded :( ^H means backspace guys. And probably the same type of CEO that wants to make money by not paying out affiliates
^H is the ASCII control character for backspace. It has nothing to do with vi, and everything to do with a terminal. It used to be common knowledge that when you press Backspace, sometimes ^H appears on your screen (if the program in question couldn't handle control characters natively). Naturally, in forum posts it started to have the meaning of "delete" for humorous purpose.
Its modern equivalent is HTML tag <del></del>, which also means "delete" due to the common knowledge of its meaning. One doesn't have to be a front-end developer to understand <del>, and one didn't have to be a vi user to understand ^H.
caret notation just maps all ASCII control characters to ^ + letter, many of which do not always have keys. "Backspace" is 0x08 = 8th letter is H = ^H
it's also not just used as an input, but also what a terminal that can't do or doesn't understand a backspace as "delete character, step to the left" prints instead.
But like I said, it's not just letters, if you use `cat -v` to show all non-printable characters in this notation, you'll also see things like "^@" for the 0x00 null byte and "^[" for the ESC. All of the terminal color-code control sequences start with ESC, so if you pipe colorized output to `cat -v`, you'll see a lot of "^[".
I don't think that's the answer to the backspace=^h question, but HJKL is precisely an artifact of the keyboard that vi was first used on: https://en.wikipedia.org/wiki/File:KB_Terminal_ADM3A.svg / https://en.wikipedia.org/wiki/Vi#Creation - in addition to HJKL being labeled as the arrow keys on that keyboard, also note the far superior placement of escape and control where modern keyboards typically place tab and caps lock; this also made vi a lot more sensible.
(And of course that last bit is a whole topic unto itself; to this day, caps lock is the dumbest key on the whole keyboard to me - mostly useless, takes up prime real estate, and 100% worth remapping on every machine I control. But that's a rant for another day:])
Because the h-key is in the middle row of the keyboard, meaning relatively shorter keyboard distance needed to execute that common command, thereby saving typing time.