The difference with bits is that it's a convention realised in the hardware.
0 and 1 are physically different from each other. A 64-bit float could easily be seen as an array of bytes without any change to the information it contains.
Bottom line: types are external to the information something contains.
Whether a particular voltage represents an 0 or a 1 is an interpretation external to the device. Internally, there are just voltages. Whether a particular voltage represents an 0 or 1 will not in general be consistent, even within a device, and whether a particular piece of the device represents a bit or not is a matter of interpretation.
What tells you that a particular location in a chip represents a bit and what voltages correspond to 0 and 1? You have to bring external knowledge of the particular device.
Entropy. I can measure the number of steady states of the device and discover the actual, real, physical information contained within.
Also through this I can discover which voltage corresponds to the high and low state.
Meanwhile a series of bytes can be a float, a string, a table of numbers or even a function pointer - you can't tell which without external information.
It's a bit frustrating to see a comment that starts with a single, incomprehensible, verbless statement like "entropy." I know what entropy is, and I could give you three different senses for the word, but it's not a particularly illuminating thing to think about here and I'm clueless about how this connects to the rest of the comment.
I think what you're doing is taking your existing understanding of how a processor works, and saying that you could figure it out, given an actual processor. But that's begging the question. You have a definition for what a "bit" is, and what a "byte" is, and since you know what they are you can go on a hunt within a physical system and label things as "bit" and "byte". But I could come in and start labeling certain things "function pointer" or "float", and I don't see a particularly compelling reason why that's not allowed, but labeling things as "bit" or "byte" is allowed.
Or in other words, why is, "I know how synchronous digital circuits work and I can figure out a processor" acceptable, but "I know how compilers work and I can reverse engineer a program" prohibited?
And I'll still say that the statement that, "Typed programming languages don’t exist," is not even wrong. It's just a cute thing that you can say, to demonstrate that you can choose a definition for "exist" or "typed programming languages" that makes it needlessly difficult to have a conversation.
0 and 1 are physically different from each other. A 64-bit float could easily be seen as an array of bytes without any change to the information it contains.
Bottom line: types are external to the information something contains.