result is rounded, but why? - NV5 Geospatial - Help Forums

Prev Go to previous topic

Go to next topic

Last Post 19 Feb 2009 04:09 AM by anon

result is rounded, but why?

2 Replies

Sort:


You are not authorized to post a reply.

Author

Messages

anon

New Member

Posts:

19 Feb 2009 04:09 AM
hi all, i am a newbie in using idl and face a problem. it is a quite simple question and i hope anybody of you can help me. when i do a simple calculation like: m=float(8054/7). i got the result 1150.00, but the correct results is 1150.57. what is the reason for this and how can i manage this. thanks in advance, christine

Deleted User

New Member

Posts:

19 Feb 2009 09:57 AM
IDL dynamically sets the data type so it is treating these as integers. Try m=float(8054.00/7.00) bill

Deleted User

New Member

Posts:

20 Feb 2009 12:15 AM
hi bill, this was very simple...thanks a lot! bye, christine

You are not authorized to post a reply.