The thing that made floating point easy to understand for me was the concept of a dyadic rational [1]. Those are just the rational numbers whose denominator is a power of two (e.g. 1/2, 3/16, 61/32). Fundamentally, binary floating point represents dyadic rationals (not counting special stuff like NaN, infinity, etc.) To represent such a number in floating point form, just write it as a/2^b. Then the mantissa is a, and the exponent is -b.
This intuitively makes it obvious why, for example, 1/3 can't be represented. I find the rational form like a/b easier to understand than the "scientific notation" form like a×2^b.
I think of a float as an integer with a shift. Take the mantissa, stick on the implicit leading 1, put it in the middle of an infinitely big string of zeroes with a decimal (well, binary) point at the right-hand end, then shift it by the exponent (which can be positive or negative).
If you're comfortable thinking about shifting integers, then perhaps this is a simple way to think about floats. For me, it's easier than thinking about them as fractions.
> a/2^b. Then the mantissa is a, and the exponent is -b
Why not just write a * 2^b. Your a here, if supposed to represent the mantissa, is not an integer. This form doesn’t really clarify the relationship to the concept of a dyadic rational.
To write the floating point number as a dyadic rational treating a as an integer you would have something like (2^(53) + a) / 2^(53 - b).
But arbitrary dydadic rationals cannot be represented by a single floating point number. If you want arbitrary dyadic rationals you need the sum of a list of floats.
* * *
I think what you might be getting at is starting by teaching people about fixed-point binary numbers? Or just binary fractions in general, analogous to the usual decimal fractions?
That would be clearer to say than anything about “dyadic rationals”. After going through a typical primary school, people have the concept of “terminating decimals”, rather than a concept of “ten-adic rationals”.
I wrote something similar ~25 years ago, when Win32 first appeared, and still use it today on the (relatively rare) occasion that I need to debug something that uses floating-point.
One interesting thing you'll notice about IEEE754 is that the values sort naturally if you interpret them as sign-magnitude integers:
In other words, if you count up in floats, it starts at 0 and goes up by increasingly larger amounts, then the number right "after" the "largest" is Infinity, followed by all the NaNs to the end of the range. The negative side is a "mirror image", the only difference being the highest (sign) bit is set.
> One interesting thing you'll notice about IEEE754 is that the values sort naturally if you interpret them as sign-magnitude integers
With a few bit-twiddling tricks you can even make them radix-sortable, as explained by Malte Skarupke:
> Michael Herf has also discovered a good way to generalize this to floating point numbers: Reinterpret cast the float to a uint32, then flip every bit if the float was negative, but flip only the sign bit if the float was positive. The same trick works for doubles and uint64s. Michael Herf explains why this works in the linked piece, but the short version of it is this: Positive floating point numbers already sort correctly if we just reinterpret cast them to a uint32. The exponent comes before the mantissa, so we would sort by the exponent first, then by the mantissa. Everything works out. Negative floating point numbers however would sort the wrong way. Flipping all the bits on them fixes that. The final remaining problem is that positive floating point numbers need to sort as bigger than negative numbers, and the easiest way to do that is to flip the sign bit since it’s the most significant bit.
Up until recently in most JS engines it was faster to sort typedarrays with a custom radix sort than with the built-in sort (seems to no longer be the case, thank goodness)
Yes, wonderful property. This also means that "next" and "previous" float can generally be computed by just interpreting the value as int and incrementing/decrementing by one, which allows easy computation of the epsilon or ULP (unit in the last place) [1].
Minor differences to the total ordering of the ints:
* IEEE NaNs always compare false
* IEEE +zero and -zero compare equal (that's the only case I'm aware of where you can have floats with x == y, but 1/x != 1/y, because +inf != -inf).
[1] Note that the ULP can get pretty large. If you have 1e23 as a float, you can add a million to it, and still get back the same number (because the next float is so far away).
Wow, I didn't expect a four year old project of mine to hit HN front page! If you're interested in this, you might enjoy some of the stuff we're working on at Figma. Here's a link to learn more: https://www.figma.com/careers/
I feel like I groked floating point numbers easier when looking at 16bit floating point numbers. The values are much smaller and easier to juggle in your head. The fact that the fraction part literally translated to x / 1024 suddenly became clear to me when you're staring at just 10 bits
It's certainly got more information but that combined with the "exact" values makes it a bit overwhelming/hard to understand, especially if you're trying to get a feel for it the first time and the half and the double have the same number of digits on screen. It's almost made more for investigation of something you ran across or need to hardcode than gaining intuition.
But most importantly I think best feature of the one in the submission is something it doesn't even mention in the description - it has a "click and drag to write bits" feature that I think is much easier to use/follow than the 2 dragging section on that one.
The 2-dragging section is much more efficient if you want to roughly pick arbitrary floats, and gives a better spatial map of the range of possible floats.
The individual bit flips also can be useful. It really depends what you want to use it for.
They do have the same number of decimal digits displayed. You can see it by clicking between "float" and "double": the decimal representation stays the same.
It makes it look like they have the same precision (well also, going from 64 to 32 bits zeros out a whole lot, which contributes to that impression). In fact, both decimal representations are usually truncated, and don't accurately represent the binary floating point number. It's a neat and useful tool regardless, and I'm being a bit pedantic.
I was playing with this yesterday for about an hour... getting so frustrated trying to figure out how to count from 0 to 20. But when I got there, I suddenly had a "omg, I know floating point!" moment.
Then this morning on the train I started wondering: "wait, but how does floating point addition even make sense?!"... I'm still stumped on that one - I need "Float Addition Toy" ;)
(Pro-tip: it took me way too long to figure out that you can click and drag to set/clear the bits quickly)
Basically, you rewrite one of the numbers so that the exponents match, then sum the mantissas. There are a few extra steps, though: https://imgur.com/fgOH4sS
A "floating point addition toy" would be way cool!
https://ciechanow.ski/exposing-floating-point/ is the clearest and most coherent explanation of floating-point numbers I've found; it was really what gave me the lightbulb moment I had been missing. This site is a great companion to it though.
I think I found a bug: If you set all bits to 0 except the sign bit you get 0 when you should get -0. In IEEE floating point representation 0 and -0 is not the same. This is why:
With the 32-bit one, if I set the highest bit of the exponent from 1 to 0, the exponent goes to -126 (ok), but if I then toggle the lowest bit of the exponent (rightmost green bit), it modifies the mantissa, not the exponent.
Is that a bug or do I not understand how IEEE754 works (very likely :p)?
If the exponent is 0, then:
the leading bit becomes 0
the exponent is fixed to -126 (not -127 as if we didn't have this exception)
Such numbers are called subnormal numbers (or denormal numbers which is synonym).
All normal floating-point numbers have an implicit leading significant bit of 1. If the exponent field goes from 1 to 0, then this corresponds to changing the implicit leading bit to a 0, not to decreasing the exponent.
And subnormal numbers are much more complicated to work with (for some reason). All I know is that DSPs would suddenly start missing their timings if games didn't clamp outputs, because suuuuper quiet reverb drifted into denormalized ranges and would more than double calculation times.
This anecdote is from the frontlines of the Xbox 360. :)
Many embedded CPUs with hw float (including DSPs) do not have hardware support for denormals; they trap to an ISR that implements them in software. This lets them write IEEE compliant on the spec sheet without having to spend transistors on denormals.
In addition, many hardware implementations are much slower on denormals; some x87 implementations were 100s of cycles slower on denormals, NaN and Infinity.
A colleague of mine was around during the IEEE-754 days and said the hardware people pushed against denormals and the software people won (possibly with a professor stacking the committee with former students of his? I'm not clear on the details).
Hmm. But if the exponent’s significant bit goes from 1 to 0, why would that affect the mantissa at all? Also, if the significant bit is always 1, how are negative exponents represented?
I understand there’s an implicit “hidden” bit, but what conditions cause it to become 0 instead of 1? And does that bit represent the sign of the exponent, or something else?
After reading https://stackoverflow.com/questions/8341395/what-is-a-subnor... more carefully, I am more confused than ever. Both visualization sites seem to be leaving out key details. And subnormals only seem to apply when exponent = 0; the visualization problem that OP mentioned happens whenever the leading exponent bit is set to 0, but the rest of the bits can be anything.
> But if the exponent’s significant bit goes from 1 to 0, why would that affect the mantissa at all?
It's not a matter of the "significant bit", but whether the WHOLE exponent is 0 or not. An exponent of 0 is a special case; as soon as you set ANY bit in that exponent, the mantissa goes back to the "implicit 1" prepended bit.
> Also, if the significant bit is always 1, how are negative exponents represented?
Exponents are represented in a "biased representation", counting up from 2-127.
> I understand there’s an implicit “hidden” bit, but what conditions cause it to become 0 instead of 1?
Exponent bits being all 0.
> And does that bit represent the sign of the exponent, or something else?
"Significant" is a typo. It should say "significand", i.e. mantisaa. All normal floating point numbers have an implicit most significant significand bit of 1, meaning they are 1.something * 2 ^ exponent. I cannot type the math properly at the moment.
Hmm, could you show a screenshot? When I toggle the rightmost green bit, it doubles the value as expected.
Note that float toy doesn't actually show the mantissa value, only the final value. So if the value goes from 6.709679e-39 to 1.8464622e-38, then it was doubled.
EDIT: Oh! I see what you mean. I was wrong; obviously it does show the mantissa.
Yes, that seems like a bug. Shouldn't the mantissa always be in the range [0.0 .. 1.0)?
On the other hand, maybe it behaves differently when the exponent is negative. If you don't understand how IEEE754 works, then apparently I don't either, so thanks for pointing that out.
On float.exposed, the exponent can’t ever go negative, which is clearly bogus. Exponent “128” should read “1”. It’s showing the raw binary unsigned value, which is misleading, since you want to know the int8 signed value.
But toggling the exponent bits should never affect the mantissa. That’s also bogus.
I think float toy is the superior visualization, and they should fix the bug. I wonder if there’s anywhere to report it....
Turns out, there's no bug in float toy. The only case where the mantissa becomes 0.xx instead of 1.xx is when exponent == 0, which is a subnormal number and is working as expected.
An exponent of 0 in IEEE format is a special case: It represents subnormal numbers, which do NOT prepend an implicit "1" to the mantissa: https://en.wikipedia.org/wiki/Denormal_number
This is awesome, I honestly learned more about floating point in five minutes of playing with this tool than I have in all my years as a coder. One of the main takeaways should be that 64 bits is usually way more precision than we need, 32 is usually more than enough. It would be cool if the tool also showed the new Bfloat16 type, so we could get some intuition for that as well. IIRC, Intel created Bfloat16 to better suit the needs of deep learning practitioners, who don't need much precision (especially for large values).
32 is plenty to represent the numbers we care about, but you'll find you probably want 64 to do math with those numbers. 64 has always been enough for me to never have to care about the precision, but whenever I've used 32 to save some ram, I've ended up having to think carefully about how some operation would reduce the precision in a material way.
Notably, 32 bit floats are insufficiently precise for representing latitudes and longitudes. I seem to recall doing the math on it, and in California the next line of longitude representable by 32 bit floats was like half a mile away. Don't quote me on the number though.
32 bit integers are precise enough. If you define 180 degrees West as INT_MIN and 180 degrees East as INT_MAX your precision is about a foot.
Although 32 bit floats are precise enough in some areas of Africa because it's closer to the 0,0 point. But it'll fall down if you're anywhere else in the world.
This rocks! I bet I grokked it way faster with this toy compared to a written explanation. An ounce of demonstration is worth a pound of documentation.
Definitely looking forward to the Unum Type III alternative to floats. This is a nice visualization and it gives you an idea of how many NaN's waste the bits being used.
yeah I was going to ask, what's the deal with all those wasted bits?
I'm not a floating-point fan (I do all computations in integer wherever possible/feasible), but I've been following the universal number story since it was announced because it seems less painful/gross than IEEE floating-point on many fronts, here's a github with a good about page about it: https://github.com/stillwater-sc/universal
I deep dove into ieee754 binary and decimal float (DPD and BID) while researching how to compress floating point values [1]. For 25 years it had been a magic black box that I couldn't trust for comparisons or printing, but now I get it. I get why it happened this way, what the benefits were, how to mitigate against its weirdness, and why ultimately decimal float needs to replace binary float in future. It's definitely worth learning.
This intuitively makes it obvious why, for example, 1/3 can't be represented. I find the rational form like a/b easier to understand than the "scientific notation" form like a×2^b.
[1]: https://en.wikipedia.org/wiki/Dyadic_rational