Problems Assigning LIST, HASH, etc. to Class or Structure Tags

[ Issues discussed in this article impact IDL versions between 8.0 and 8.8.1. The internal implementations of LIST, HASH, ORDEREDHASH, etc. have been updated in IDL 8.8.2 to address these and other problems. ]

 

Why is this class definition functional, but incorrect?

pro myClass__define
    !null = {myClass, myList : LIST()}
end

On initial inspection, the code compiles and runs so it would appear to be correct. After all, variables of type HASH, LIST and their kin are described in the IDL documentation as fundamental compound data types. The same term is used with respect to structure and object types, for example.

The former types are distinguished from object references but the reasons for this are murky. Because they are described as their own data types, a programmer would be expected to declare member variables in structure or class definitions with the empty LIST() type shown above, in the same way we would declare a tag that will hold an object reference with an OBJ_NEW() value.

The reality is that a LIST is a specific object class with its own methods and data.  This can be demonstrated quite simply via an ISA test.

IDL> ISA(LIST(), 'OBJREF')
   1

The obfuscation of this in the documentation has the potential to lead to unexpected and incorrect behavior.

Before discussing the negative impacts of declaring a LIST() as a member variable, let's first see the correct declaration. "Compound data types" like LIST and HASH in a structure or class definition should be set to OBJ_NEW() rather than null instances of the specific classes, for example

pro myClass__define
    !null = {myClass, myList : OBJ_NEW(), myHash : OBJ_NEW()}
end

The invalid behaviors generated by declaring a LIST() in this case aren't necessarily obvious.

Consider the case where you have developed an application or utility such as an ENVI Toolbox Extension and you intend to distribute it to your colleagues or customers as a compiled SAVE file.

IDL> .full_reset_session
IDL> .compile -v '\myclass__define.pro'
% Compiled module: MYCLASS__DEFINE.
IDL> RESOLVE_ALL

So far so good. But let's see what happens when we SAVE the compiled code.

IDL> save,/routines,/verbose,file='myclass__define.sav'
% SAVE: Portable (XDR) SAVE/RESTORE file.
% SAVE: Saved procedure: COLLECTION::CLEANUP.
% SAVE: Saved procedure: COLLECTION::_OVERLOADBRACKETSLEFTSIDEARRAY.
% SAVE: Saved procedure: COLLECTION__DEFINE.
% SAVE: Saved procedure: HASH::CLEANUP.
% SAVE: Saved procedure: HASH::DUMPTABLE.
...

By declaring the LIST() as a member variable, methods for HASH and LIST along with the undocumented "COLLECTION" superclass will be compiled and included in my SAVE file, because they are not C-level routines.

Setting the SKIP keyword to RESOLVE_ALL in an attempt to ignore these methods will fail. The act of initializing the null LIST object in the structure definition loads in the IDL library file of compiled routines in which LIST, HASH, and COLLECTION are stored, and that act cannot be undone.

IDL> help,/str,routine_info('list', /source, /function)
** Structure <16591d70>, 2 tags, length=32, data length=32, refs=1:
   NAME            STRING    'LIST'
   PATH            STRING    'C:\Program Files\Exelis\IDL84\lib\datatypes\hash.sav'

The side effect of this is that when the SAVE file is restored, the versions of these methods in the file will overwrite any existing methods already in IDL's memory. This could well occur in the case where the operator is loading an ENVI Toolbox Extension into an already-running ENVI session. There is high probability that LIST and HASH have already been loaded by that point in ENVI.

As long as the version of IDL matches between the "default" method definitions and those restored from your custom SAVE file, you should not run into any issues.

But if you are restoring between different versions of IDL (for example an IDL 8.0 SAVE file restored in ENVI 5.2/IDL 8.4) you are likely to encounter problems. This is generally true for ALL compiled routines exchanged between versions of IDL and is one motivation behind the SKIP keyword to RESOLVE_ALL. But there is no SKIP work-around for "built-in" IDL types.

There is no guarantee that method signatures will match between versions, particularly for the great number of undocumented methods that are required by the internals of these classes.

Code which executes later on in that IDL session, calls that expect updated syntax or features, may fail - with or without obvious errors. Unexpected and hard-to-reproduce conditions may result from code that previously ran without issues.

From an IDL user's perspective, for Exelis to move the internals of these classes down to the internals or DLM level would offer the best solution.