# Floating point error

Jay Carlson (nop@k2.ccs.neu.edu)
Wed, 13 Mar 1996 14:41:56 PST

- Messages sorted by:[ date ][ thread ][ subject ][ author ]

    > It appears that there's a problem with floating point in 1.8.0p1...
    > On a MOO running 1.8.0beta3 and on another running beta2:

    > ;1.6
    > => 1.6

    > On a MOO running 1.8.0p1 and on ANOTHER running p1:
    > ;1.6
    > => 1.6000000000000001
    ^^^^^1 2345678901234567    17 digits.

    I agree that this is probably a bug, but you should be aware that
    looking for 17 digits of precision sends you off into the weeds
    anyway.

    Each decimal digit has log2(10) bits of information in it.  So 17
    digits is around 56.5 bits of information.  If we go poke at
    /usr/include/float.h, we find out there's only 53 bits of mantissa in
    IEEE double-precision floats.  So we're trying to print more
    information than we actually have.

    More fun:

      ;1.6 == 1.6000000000000001
      => 1
      ;1.6000000000000002
      => 1.6000000000000003

    If you care about accuracy, and your application floating point, you
    should be careful.  Don't forget that floating point doesn't give you
    infinite precision, and that, for example, you almost NEVER want to
    use "a == b" on floating point numbers; use "abs(a - b) < .0001" or
    whatever is appropriate for your application.  Many of the standard
    numerical analysis issues could be magnified by persistence if you're
    not careful.

    Jay Carlson
    nop@nop.com    nop@ccs.neu.edu    nop@kagoona.mitre.org

    Flat text is just *never* what you want.   ---stephen p spackman