READ_ASCII precision of floating-point values

Prev Go to previous topic

Go to next topic

Last Post 17 Nov 2011 04:14 AM by anon

2 Replies

Sort:


You are not authorized to post a reply.

Author

Messages

anon

New Member

Posts:

17 Nov 2011 04:14 AM
Hello, I read data from an ASCII file using aa=READ_ASCII(fn, template=sTemplate) the first line of the file is 632584.254 93979.473 3089.750 but IDL printf with float format code produces 632584.250000 93979.476563 3089.750000 So, what's wrong here? Is there a way to force IDL to use the full float precision when reading ASCII files? Or is it my mistake? best regards Leo

Deleted User

New Member

Posts:

17 Nov 2011 10:28 AM
"Mistake" is too harsh a word. Let's say it is a "misunderstanding" of the way computers store numbers. You may find some enlightenment in this article: http://www.idlcoyote.com/.../sky_is_falling.html

Deleted User

New Member

Posts:

18 Nov 2011 02:10 AM
Haha.. Good reading. Thanks a lot for this hint, that's exactly what I need! Sorry for the post, though...

You are not authorized to post a reply.