Hacker News new | past | comments | ask | show | jobs | submit login

How much was the difference? Was there any rounding involved at any step? When dealing with money, I see rounding and integer math all the time. As another comment has mentioned, within 53 bits of mantissa, the number range is so big, we are talking 16 digits. I''d be curious to see a real-world example where the float math is the source of error, as opposed to some other bug.



It doesn't take much imagination to construct an example. Knowing 0.1 isn't exactly represented make a formula with it that should be exactly a half cent. Depending on fp I precision it will be slightly above or below a half cent and rounding will not work as expected. We found this in prod at a rate of hundreds to thousands of times per day it only takes volume to surface unlikely errors.


I thought I was clear that the result was rounded. The fp imprecision affects the correctness of rounding e.g. half to nearest even.


It is the rounding that seems to be an issue here.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: