X
PrevPrev Go to previous topic
NextNext Go to next topic
Last Post 04 Apr 2005 06:26 PM by  anon
why fix((0.9-0.1)/0.1) equals to 7 ,not 8?
 1 Replies
Sort:
You are not authorized to post a reply.
Author Messages

anon



New Member


Posts:
New Member


--
04 Apr 2005 06:26 PM
    why fix((0.9-0.1)/0.1) equals to 7 ,not 8? and a=(0.9-0.1)/0.1 ,a is 8,correct~~ while fix((0.8-0.1)/0.1) also equals to 7? so strange~~

    Deleted User



    New Member


    Posts:
    New Member


    --
    04 Apr 2005 06:26 PM
    This probably has something to do with the nature of floating-point arithmetic and storage of decimal numbers. See the reference guide entry for the MACHAR routine for more information. According to the mathematical expression you have entered, the resulting value for the variable "a" is indeed 8.0 : IDL> a = (0.9-0.1) / 0.1 IDL> HELP, a A FLOAT = 8.00000 But upon closer inspection, the variable actually stores the closest possible representation of 8.0 given 32-bit IEEE precision : IDL> PRINT, a, FORMAT='(F9.7)' 7.9999995 The FIX data type conversion routine truncates at the decimal point, so the resulting value gets rounded down to 7. In general, it is best to make use of the ROUND routine first when performing conversion from floating-point to integer data types in IDL : IDL> b = FIX( ROUND( (0.9-0.1) / 0.1) ) IDL> HELP, b B INT = 8
    You are not authorized to post a reply.