Why stop at 16-bit? I'd be curious to see a study that tries every number of bits from 32-bit down. I see https://en.wikipedia.org/wiki/Minifloat says there is an 8-bit float which uses 4-bits for exponent and 3-bits for significand. Maybe there is a sweet spot between 8-to-16 bits for a good-enough accuracy tradeoff. Of course the hardware for that isn't standard, but maybe low-bit float hardware would be useful.
Posits seems to be better for 8-bit or even 6-bit. There is only one not-a-number place, the NaR (not a real). This means for 6-bit posits you have 63 points in the number space.
The way posits focus on numbers near 1.0 is probably going to have a bigger effect. A 6 bit float with 4 exponent bits is the best competitor to a 6 bit posit, and it would only have four non-finite numbers.
What would it mean though? If the information is embedded in the network, and we find some performance figure of merit, wouldn't it probably be about the same performance when normalized to power utilization?
Why would you expect performance to be constant when normalized to power utilisation?
If your 16 bit floats perform about the same as 32 bit floats in an absolute sense, then they will probably perform even better when normalised for power utilisation.
And if 16 bit work, 15 bit floats might perform well, too, for all we know. That's what the original commenter was getting at, I think.
Yes, that is what I was getting at. Floating point hardware with fewer bits have less complexity and so have smaller transistor die area and power consumption.
I'm guessing that "about the same" will be hard to measure, and that at some point, thermodynamics will dictate the maximum performance per power output (assuming fixed transistor architecture).
Thermodynamics will dictate the performance in some sense of bit-operations per Joule sense.
The more important performance metric is not the number of bit-operations, but the quality of the neural network output.
The hypothesis is that fewer bits in your numbers give you the same or nearly the same output quality, but at drastically fewer bit operations performed, and thus less Joule spent.
