X
PrevPrev Go to previous topic
NextNext Go to next topic
Last Post 03 Mar 2009 10:16 AM by  anon
BYTSCL bug!?
 1 Replies
Sort:
You are not authorized to post a reply.
Author Messages

anon



New Member


Posts:
New Member


--
03 Mar 2009 10:16 AM
    I've recently been converting some IDL code to Java and found this problem: BYTSCL returns different results based on the data type you pass in. For example, the two lines below return different results for 9065. print, bytscl([4124, 9065, 18946], top = 2) print, bytscl(FLOAT([4124, 9065, 18946]), top = 2) That's fine. Later on I found the explanation in IDL's documentation "For floating-point input, each value is scaled using the formula (Top + 0.9999)*(x - Min)/(Max - Min). For integer input, each value is scaled using the formula ((Top + 1)*(x - Min) - 1)/(Max - Min). ". No problem, I can use their formulas. But the bad thing somehow they're not using the formula for integer to do the actual calculation. Here's the evidence print, bytscl([4124, 9065, 18946], top = 2) gives me 0 on 9065. But if you use their formula for compute it, ((2 + 1) * (9065 - 4124) - 1) / (18946 - 4124) gives me 1. This is frustrating me a lot because I need to duplicate the exactly same numeric results in Java. But with this buggy BYTSCL, it's almost impossible to duplicate its behavior. Anybody has any idea what formula they're using internally? Thanks!

    Deleted User



    New Member


    Posts:
    New Member


    --
    25 Mar 2009 03:09 PM
    This is due to differences in how the machine computes Integer calculations vs floating point calculations.  At times they will produce slightly different results, but this is to be expected.
    You are not authorized to post a reply.