X
PrevPrev Go to previous topic
NextNext Go to next topic
Last Post 22 Jan 2017 09:26 AM by  anon
JSON_SERIALIZE precision
 1 Replies
Sort:
You are not authorized to post a reply.
Author Messages

anon



New Member


Posts:2
New Member


--
22 Jan 2017 09:26 AM
    JSON_SERIALIZE(x) will produce radically different output depending on the value of x. For example: IDL> a = 19.05 IDL> print, JSON_SERIALIZE(a) 19.04999923706055 IDL> a = 19.75 IDL> print, JSON_SERIALIZE(a) 19.75 Can anyone explain this enormous difference. System settings? Bug? Not acceptable -- producing huge JSON file output when I am supplying 2-decimal place precision input. C. Wallace

    Deleted User



    Basic Member


    Posts:143
    Basic Member


    --
    17 Feb 2017 04:21 PM
    This issue is most likely occurring due to limitations on how floating point numbers can be stored on systems. Detailed information about this topic can be found using following link: http://www.harrisgeospati...cleID/2723/2723.aspx If you use doubles instead of floats, this helps with the problem: IDL> a = 19.05d IDL> print, JSON_SERIALIZE(a) 19.05 -David Harris GS
    You are not authorized to post a reply.

    Forums Products IDL