Answer:
Hello,
This type of problem can be caused when trying to represent real decimal data values on a computer, which uses a limited precision binary representation for floating point values. As a rule of thumb, single precision floating point values can be trusted to approximately 6 or 7 significant digits (depending on the specific binary representation of the number).
Further, limited precision can propagate through calculations in prescribed ways such that results from parallel calculations that go through different computational paths can appear to give a different answers -- perhaps even different from the expected theoretical answer. However, in such cases, if you take into account only the proper number of "propagated* significant digits for those particular calculations, the results can be interpreted to be substantially the same as that expected from theoretical precise calculation.
This seems to be the case for your example statement:
print, where(x eq .9)
To illustrate:
IDL> x=indgen(21)*.05
IDL> print, x, format='(7f12.8)'
0.00000000 0.05000000 0.10000000 0.15000001 0.20000000 0.25000000 0.30000001
0.34999999 0.40000001 0.45000002 0.50000000 0.55000001 0.60000002 0.65000004
0.69999999 0.75000000 0.80000001 0.85000002 0.90000004 0.94999999 1.00000000
IDL> print, where(x eq .8)
16
IDL> print, .8, format='(7f12.8)'
0.80000001
IDL> print, where(x eq .95)
19
IDL> print, .95, format='(7f12.8)'
0.94999999
IDL> print, where(x eq 1)
20
IDL> print, 1, format='(7f12.8)'
1.00000000
IDL> print, where(x eq .9)
-1
IDL> print, .9, format='(7f12.8)'
0.89999998
IDL> print, 18*.05, format='(f10.8)'
0.90000004
IDL> print, .9, format='(f10.8)'
0.89999998
To compensate for the imprecise representation of floating point value when doing value comparisons, one approach that can be used is to include a"buffer" value to ensure that the floating point value approximations will fall within acceptable ranges. For example:
IDL> buf = .0000001
IDL> print, where(x le .9+buf and x ge .9-buf )
18
IDL> .9+.0000001
0.90000010
IDL> .9-.0000001
0.89999986
Another approach can be to work with the significant digit comparisons in the integer realm, where computer value representations can be trusted to be exact:
IDL> multiplier = 100000L
IDL> print, where(round(x*multiplier) eq round(.9*multiplier))
18
For additional information on this topic, please also see the following IDL help page on our Documentation Center:
http://www.exelisvis.com/docs/Accurac...
I hope this can behelpful to you.
Kind regards,