Wrong result in multiplication - NV5 Geospatial

Prev Go to previous topic

Go to next topic

Last Post 02 Apr 2014 08:33 AM by anon

Wrong result in multiplication

3 Replies

Sort:


You are not authorized to post a reply.

Author

Messages

anon

New Member

Posts:

02 Apr 2014 08:33 AM
Hello, IDL generates a wrong multiplication between two numbers. The sentence is: print, 1366 * 24 IDL generates: - 32752 If we try with a number greater than 1366 that multiplicates 24, IDL will always generate a wrong number. Thanks for your help.

Deleted User

New Member

Posts:

02 Apr 2014 12:08 PM
Hi Oscar, This has to do with choosing the right Data Type when you deal with numbers in a computer: http://www.exelisvis.com/.../IDL_Data_Types.html See that: Integer A 16-bit signed integer ranging from -32,768 to +32,767. In other words, you need to specify the data type Long (integer) since: Long A 32-bit signed integer ranging in value from -2 147 483 648 to +2 147 483 647. So, IDL> print, 1366L * 24 32784 Cheers, Fernando

Deleted User

New Member

Posts:

Deleted User

New Member

Posts:

You are not authorized to post a reply.