(INTERNAL) IEEE single-precision floating-point format
Anonym
Topic:
This is a response that Doug sent to a customer regarding precision.
Discussion:
For clarity, when I refer to "decimal" places, I refer to the number of places AFTER the decimal point. When I refer to "significant decimal" places, I refer to the HIGHEST ORDER decimal places. Hence, "significant decimal" places refers to places BEFORE AND AFTER the decimal point.
The IEEE single-precision floating-point format is a 32-bit format, where 24 bits are used to represent the mantissa. Twenty-four "significant binary" places translates to approximately eight "significant decimal" places.
It is meaningless to print more than eight "significant decimal" places for a single-precision floating-point number; any places beyond the highest order eight will be garbage. Actually, on most hardware, the eighth "significant decimal" place is often "fuzzy," so it is better to trust only seven "significant decimal" places.
If you wish to compare the single-precision and the double-precision values of pi (or any other floating-point number), only the first seven, or eight "significant decimal" places should be considered.
Here is an example of the comparison, using eight "significant decimal" places:
IDL> print, !pi, format='(d20.7)'
3.1415927
IDL> print, !dpi, format='(d20.7)'
3.1415927
These are the correct values of pi, ROUNDED to seven "decimal" places.
Notice that, if I display one more place (nine "significant decimal" places),
IDL> print, !pi, format='(d20.8)'
3.14159274
IDL> print, !dpi, format='(d20.8)'
3.14159265
the single-precision display is not correct. In fact, the lowest TWO "decimal" places (74) are now incorrect, because the display IMPLIES the value of pi ROUNDED to eight "decimal" places.
As expected, the double-precision display is the correct value of pi ROUNDED to eight "decimal" places.
The display of the double-precision value of pi is correct (and correctly ROUNDED), because there are enough "significant" places below the last display place (internally) to ensure proper display rounding. This is not the case for the single-precision value of pi.