New Features Coming Soon to IDL - With a Few Brief Examples
Anonym
IDL 8.4.1 is scheduled to release within the next few weeks. Although this is only a point release rather than a full new version, there are a few features that the IDL team has implemented that are very exciting to me.
HDF and NetCDF Updates
When working with scientific data, I often find myself using IDL to open and read HDF4 and HDF5 files. Although the design of HDF5 is quite a bit simpler than HDF4, working with the API for both formats can sometimes be rather painful, and I almost always need to reference the documentation examples just to get started.
One of my favorite routines in IDL is H5_PARSE (which is a function in the IDL library and not part of the standard HDF5 interface). The reason I like this function so much is that I can use it to query datasets and attributes in the file without needing to write multiple lines of code to search for and access the desired information (and I can't forget to ensure the file gets closed even if an error occurs). Everything I want is simply dumped into a hierarchical structure.
My one complaint, however, about H5_PARSE is that it can be rather slow, especially with large files. The reason for this is that the structure is created dynamically as IDL descends through the file, using this logic:
sTree = CREATE_STRUCT(sTree, tagname, sMember)
Just like how appending an array is very inefficient, creating a structure in this fashion is slow and memory intensive, particularly when using the /READ_DATA keyword.
Luckily, IDL’s new OrderedHash data type provides a handy alternative; adding fields to a hash is much more efficient than appending a structure. Additionally, an ordered hash not only preserves the order, it also preserves names and cases (structures are not case-sensitive and special characters and spaces in object names must be converted to underscores to be valid structure tags).
Furthermore, what is cool about an ordered hash is that you can use Implied Print, and IDL will print out the file’s information in a hierarchical JSON format, mimicking the structure of the file. To use an ordered hash in place of a structure, use the /ORDEREDHASH keyword when calling H5_PARSE. Here is an example using an HDF5-EOS file:
file = 'C:\hdf5\hdfeos\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5'
result = H5_PARSE(file, /ORDEREDHASH)
result
IDL will print:
{
"_NAME": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
"_ICONTYPE": "hdf",
"_TYPE": "GROUP",
"_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
"_PATH": "/",
"_COMMENT": "",
"HDFEOS": {
"_NAME": "HDFEOS",
"_ICONTYPE": "",
"_TYPE": "GROUP",
"_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
"_PATH": "/",
"_COMMENT": "",
"ADDITIONAL": {
"_NAME": "ADDITIONAL",
"_ICONTYPE": "",
"_TYPE": "GROUP",
"_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
"_PATH": "/HDFEOS",
"_COMMENT": "",
"FILE_ATTRIBUTES": {
"_NAME": "FILE_ATTRIBUTES",
"_ICONTYPE": "",
"_TYPE": "GROUP",
"_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
"_PATH": "/HDFEOS/ADDITIONAL",
"_COMMENT": "",
"InstrumentName": {
"_NAME": "InstrumentName",
"_ICONTYPE": "text",
"_TYPE": "ATTRIBUTE",
"_NDIMENSIONS": 0,
"_DIMENSIONS": 0,
"_NELEMENTS": 1,
"_DATATYPE": "H5T_STRING",
"_STORAGESIZE": 3,
"_PRECISION": 24,
"_SIGN": "",
"_DATA": "OMI"
etc.
Another new function that IDL 8.4.1 will offer is HDF_PARSE. This is the equivalent to H5_PARSE but for HDF4 files. It will walk through a file and return information about groups, scientific datasets, images, palettes, annotations, and VData. This information will be returned in an ordered hash, following the same structure that the information is stored in the file. HDF_PARSE will always return an ordered hash; if you would like a structure instead, you can convert the hash to a structure using OrderedHash::ToStruct. Here is an example using a MODIS file:
file = 'C:\scratch\modisl2\MOD10_L2.A2002013.0930.003.2002017035237.hdf'
result = HDF_PARSE(file)
result
IDL prints:
{
"_NAME": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
"_TYPE": "GROUP",
"_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
"_PATH": "/",
"MOD_Swath_Snow": {
"_NAME": "MOD_Swath_Snow",
"_TYPE": "GROUP",
"_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
"_PATH": "/",
"Geolocation Fields": {
"_NAME": "Geolocation Fields",
"_TYPE": "GROUP",
"_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
"_PATH": "/MOD_Swath_Snow",
"Latitude": {
"_NAME": "Latitude",
"_TYPE": "SD",
"_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
"_PATH": "/MOD_Swath_Snow/Geolocation Fields",
"_NDIMENSIONS": 2,
"_DIMENSIONS": [271, 406],
"_IDL_DATATYPE": "FLOAT",
"_DATA": "<unread>",
"long_name": {
"_NAME": "long_name",
"_TYPE": "ATTRIBUTE",
"_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
"_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
"_IDL_DATATYPE": "STRING",
"_DATA": "Coarse 5 km resolution latitude"
},
"units": {
"_NAME": "units",
"_TYPE": "ATTRIBUTE",
"_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
"_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
"_IDL_DATATYPE": "STRING",
"_DATA": "degrees"
},
"valid_range": {
"_NAME": "valid_range",
"_TYPE": "ATTRIBUTE",
"_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
"_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
"_IDL_DATATYPE": "FLOAT",
"_DATA": [-90.000000, 90.000000]
},
"_FillValue": {
"_NAME": "_FillValue",
"_TYPE": "ATTRIBUTE",
"_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
"_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
"_IDL_DATATYPE": "FLOAT",
"_DATA": [-999.00000]
},
"source": {
"_NAME": "source",
"_TYPE": "ATTRIBUTE",
"_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
"_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
"_IDL_DATATYPE": "STRING",
"_DATA": "MOD03 geolocation product; data read from center pixel in 5 km box"
}
},
"Longitude": {
etc.
IDL 8.4.1 will also include new NetCDF routines for querying, retrieving and creating NetCDF files. These routines include NCDF_GET, NCDF_LIST, and NCDF_PUT
Overloading the ++ and -- Operators
I have always liked IDL’s ability to overload operators of classes that inherit IDL_Object. This gives me the ability to create objects that behave like their own data types.
One set of operators that couldn't be overloaded until now is the increment/decrement operator, ++/--. I like this operator because not only is it easier to type var++ than var = var + 1, it is more efficient. There are also cases where “increment” has a meaning other than “add one,” such as if the value is not numeric. Here is an example of an alphabet class that increments through letters (all lower-case). Once the value reaches the letter “z,” the next value in the sequence will be “aa.” When it reaches “az,” incrementing it again will change the value to “ba,” etc.
pro SequentialLetters::_OverloadPlusPlus
compile_opt idl2, hidden
if (StrCmp(self.value, '')) then begin
self.value = 'a'
return
endif
byteValue = Byte(self.value)
if (byteValue[-1] eq 122) then begin
if (StrLen(self.value) eq 1) then begin
self.value = 'aa'
endif else begin
; Create a new SequentialLetters object to recursively handle all but the
; last letter.
newObj = Obj_New('SequentialLetters', VALUE=StrMid(self.value, 0, StrLen(self.value)-1))
newObj++
self.value = newObj.VALUE + 'a'
Obj_Destroy, newObj
endelse
endif else begin
byteValue[-1]++
self.value = String(byteValue)
endelse
end
;-------------------------------------------------------------------------------
pro SequentialLetters::_OverloadMinusMinus
compile_opt idl2, hidden
; If there is only one character and it is 'a', then the value becomes
; an empty string.
if (StrCmp(self.value, 'a')) then begin
self.value = ''
return
endif
byteValue = Byte(self.value)
if (byteValue[-1] eq 97) then begin
; Create a new SequentialLetters object to recursively handle all but the
; last letter.
newObj = Obj_New('SequentialLetters', VALUE=StrMid(self.value, 0, StrLen(self.value)-1))
newObj--
self.value = newObj.VALUE + 'z'
Obj_Destroy, newObj
endif else begin
byteValue[-1]--
self.value = String(byteValue)
endelse
end
;-------------------------------------------------------------------------------
pro SequentialLetters::SetProperty, VALUE=value, _REF_EXTRA=extra
compile_opt idl2
if (N_Elements(value) gt 0) then begin
self.value = value
endif
if (ISA(extra)) then begin
self.IDL_Object::SetProperty, _EXTRA=extra
endif
end
;-------------------------------------------------------------------------------
pro SequentialLetters::GetProperty, VALUE=value, _REF_EXTRA=extra
compile_opt idl2
if (Arg_Present(value)) then begin
value = self.value
endif
if (ISA(extra)) then begin
self.IDL_Object::GetProperty, _EXTRA=extra
endif
end
;-------------------------------------------------------------------------------
function SequentialLetters::Init, _REF_EXTRA=extra
compile_opt idl2
if (~self.IDL_Object::Init()) then return, 0
if (ISA(extra)) then begin
self.SetProperty, _EXTRA=extra
endif
return, 1
end
;-------------------------------------------------------------------------------
pro SequentialLetters__Define
compile_opt idl2
!null = {SequentialLetters, $
inherits IDL_Object, $
value: ''}
end
Here is an example of using this object:
myLetters = Obj_New('SequentialLetters', VALUE='z')
myLetters++
print, myLetters.VALUE
IDL prints:
aa
myLetters--
print, myLetters.VALUE
IDL prints:
z
Other Updates
A full list of new features and product improvements in IDL 8.4.1 will be published soon. Stay tuned!