I have encountered many real world cases where I needed less precision/range than what float/double had available. But usually I found fixed point was a better solution than reduced precision floats. I wonder what applications are there that can deal with reduced precision but somehow still need the range you get with an exponent?
16-bit floats with an 8-bit exponent and a 7+1 bit mantissa are popular for neural networks, because they have the same range as standard 32-bit floats while taking have the memory and memory bandwidth.
Interesting, although I believe most neural nets nowadays have moved to use linear (relu) activation, which again removes the need for exponent and would work really well with fixed point.
Ironically, the section labeled "What FloatX is NOT" tells you more about what it is than what it isn't.
Its a system for emulating narrower precision floating-point arithmetic using native machine-width floating point. The author's claim that its much faster than using the integer unit for this purpose.
Boost::multiprecision and MPFR are libraries to execute higher-precision arithmetic, commonly using the integer hardware to do so.
Emulating narrower using larger floating-point arithmetic can be dodgy, since you open yourself up to double rounding scenarios. For the IEEE 754 types (half, single, double, and quad), it is the case that the primitive operations (+, -, *, /, sqrt) are all correctly rounded if you emulated it by converting to the next size up, doing the math, and converting it down. For non-IEEE 754 types (such as bfloat16, or the x87 80-bit type), this is not the case, so double rounding is a possible concern.
Wow thanks! Yeah that wasn't clear to me at all. I hadn't considered the need for a floating point library that is less precise (and less performant!) than the 32/64 bit native types.
I'm trying to use 16 bit floats for matrix multiplication on x86-64. I found solutions for ARM and some NVIDIA GPUs but none for any X86-64 chips. Any pointers in this direction would be helpful.
Here is a little sample code I threw together, it shows the whole cycle, from conversion to half-floats to the conversion back to floats and performing a simple multiplication of the values:
https://dl.acm.org/doi/10.1145/3368086