How does IDL release Virtual Memory?

IDL does free its dynamic memory when it is done with it, and the IDL code has consistency checks that guard against memory leaks.

You can use the IDL command:

    HELP, /MEMORY

to see how much memory IDL has allocated. However, just because IDL frees its memory does not mean that its virtual address space will shrink and reduce its demand on the pagefile. In fact, it will not.

This seeming contradiction is due to the fact that IDL uses the C language malloc(3) and free(3) functions to allocate and free memory. When a process calls free, the freed memory is not returned to the operating system. It is simply placed into a free list. The next time malloc is called, it will try to satisfy the request from this free list before it asks the operating system for more memory. Thus, the process size (and demand on the pagefile) represents the high water mark of memory usage, not its current usage.

It might be nice if free would return memory to the operating system and reduce the amount of pagefile currently claimed by the process, but this is not done. To show why, consider an example in which a process allocates two arrays, and then frees the first one. Here is what would happen:

1) The first request causes malloc to request memory from the operating system. The process address space is enlarged and the request is satisfied.

2) The second request causes the address space to be extended again.

3) The process frees the first array. free() wants to return its memory to the operating system and shrink the process size, but it can't because the second array follows it in the virtual address space and this would leave a hole.

4) At this point, free() could try to make things work by copying the second array into the space occupied by the first, but then it would have to search the process address and fix up any pointers to the second array to point at the new location. This is impossible because there is no way to tell such pointers from unrelated memory locations that happen to contain the same bit pattern.

So, the dynamic memory is being freed, and it is gathering in the free list. Since each request is larger than the one before it, malloc makes the process grow to satisfy it, until the pagefile quota is hit. Arguably, malloc/free should be trying to coalesce the free list to make large memory chunks out of small ones, but for reasons known only to their author, it is not.

Now of course, the question is what can you do to help reduce this fragmentation?

To reduce fragmentation we recommend that you start IDL, then make one massive array as large as possible, i.e. an array which uses up most of your virtual memory. Next set it to the integer zero. Then continue with normal IDL usage. This will help reduce the problem.

NOTE:  Until IDL 8,  IDL heap variables such as pointers and objects had to be manually freed by the user using the PTR_FREE and OBJ_DESTROY command. In IDL 8 automatic garbage collection was added to complete this task automatically. 

Reviewed by JU 9/16/2014