X
PrevPrev Go to previous topic
NextNext Go to next topic
Last Post 21 Dec 2007 01:12 PM by  anon
0ULL computation behaviour
 1 Replies
Sort:
You are not authorized to post a reply.
Author Messages

anon



New Member


Posts:
New Member


--
21 Dec 2007 01:12 PM
    Hi, I try to do some basic computation of some 0ULL values and the results are all weird... This is even more weird because this computation was working like a charm on the former version of IDL and now I've got errors... So here is the problem: Waveform timestamp: 1151843761643267 Compute Waveform: 1151843761643600 (Waveform timestamp - Compute Waveform)/333 = 55395627848977631 or the correct answer would be: -1 Obviously the problem comes form the data format, but even when I specify every parameter format as follow: ulong64(Waveform timestamp) - ulong64(Compute Waveform))/333ULL I still get the wrong answer. Any idea. Cheers. Antoine C.

    Deleted User



    New Member


    Posts:
    New Member


    --
    21 Dec 2007 01:12 PM
    You have two issues here, Antoine. 'Waveform timestamp' and 'Compute Waveform' are unsigned integers, so the result of'Waveform timestamp - Compute Waveform' will be unsigned as well. Where you might expect that result to produce a small negative number, the unsigned datatype of the result will actually interpret a very large positive number. The second statement has an "order of operations" issue. The 'ulong64(Compute Waveform))/333ULL' division is being performed before the subtraction from 'ulong64(Waveform timestamp)'. You need more parentheses to fix the order of operations ... but then you will still have the problem with the fact that these integers are unsigned. Thus, the only way to get your '-1' answer, I think, is to run: (long64(Waveform timestamp) - long64(Compute Waveform)) / 333 James Jones ITT Technical Support
    You are not authorized to post a reply.