Internal: Difference between FLOATing and DOUBLE-precision data type and their conversion
Help Article Update 2
Anonym
This article discusses the difference between FLOATing and DOUBLE-precision data and their conversion.
Many (most in fact) numbers cannot be EXACTLY represented as an IEEE floating point number of DOUBLE or FLOAT precision. Therefore, converting a number back and forth between FLOATs and DOUBLEs might not return consistent results. IDL, C, Fortran, use IEEE floating point and thus this inconsistency sometimes occurs. In the second example given, that of 95.2, the closest single precision representation is 95.19999694824218750000.
To see, try the following in IDL or C:Solution:
print, format='(f40.20)', 1234567890.1234560, 1234567890.1234560D0
;1234567936.00000000000000000000
;1234567890.12345600128173828125
;Neither number is exact, and the granularity of single
;precision IEEE numbers of range 1234567890 is 128.
print, format='(f40.20)', 95.2, 95.2D
;95.19999694824218750000 (closest IEEE single prec to 95.2)
;95.20000000000000284217
;Note that neither number is exact.