INTERNAL: Maximum array size limitation in IDL

Topic:
IDL has a 32-bit version and a 64-bit version, and depending on the platform, one or both of these versions may be available. The following article discusses the maximum array size limitations for the 32-bit version of IDL. Note that the 64-bit version does not have the same limitation.

When attempting to create an array that is slightly less than 2 GB using a 32-bit version of IDL the following error is displayed:

IDL> a = BYTARR(2UL^31-9)
% Array has too many elements.
% Execution halted at: $MAIN$   

Similarly, when attempting to create an array that is between 2 GB and 4 GB the following error is displayed:

IDL> a = BYTARR(2UL^31)
% Array dimensions must be greater than 0.
% Execution halted at: $MAIN$  

The following discussion explains these error messages in more detail.

Discussion:
The second error message can be misleading since 2UL^31 is clearly a positive number.

IDL> PRINT, 2UL^31
  2147483648

The reason for this error message is that IDL maintains array sizes internally as signed integers and not unsigned as in the example above. Once the number above is converted to a signed 32-bit integer, the result is negative and thus the error message is generated.

IDL> PRINT, LONG(2UL^31)
 -2147483648

The first error message happens for a similar reason. When IDL allocates an array, it allocates a few bytes more than the exact number of bytes the IDL programmer asked for. The reason for adding these few extra bytes is not relevant here. However, since IDL uses signed integers to store array sizes internally, the first error message will be issued when the requested number of bytes pluss the padding exceeds the range for a signed integer.

Note that this limitation applies to the maximum size for a single array, and does not apply to the total of all arrays. On a 32-bit Linux or Mac OS X system it is possible to allocate close to 3 GB total memory as long as each single array does not exceed 2 GB. For example this was tested on a Linux system using IDL 6.0:

IDL> a = BYTARR(1400 * 2UL^20, /NOZERO)  ;; 1400 MB
IDL> b = BYTARR( 800 * 2UL^20, /NOZERO)  ;; 800 MB
IDL> c = BYTARR( 600 * 2UL^20, /NOZERO)  ;; 600 MB
IDL> HELP, NAMES='*'
A               BYTE      = Array[1468006400]
B               BYTE      = Array[838860800]
C               BYTE      = Array[629145600]

How to find out if this limitation applies for a specific system?
Use the following command in IDL to determine if the 2 GB maximum array size limit applies:

IDL> PRINT, !VERSION.memory_bits

If this statement prints "64" then the this techtip does not apply. However, if it prints "32", then the discussion applies.

Other memory considerations, especially on the Windows platform, can be found in tech tip 3346.Solution:
[Edit this field in the IDL-based Tech Tip Editor, v60]