Serializing IDL Objects and Structures

In my most recent IDL Data Point blog, I showed some example code for extending the IDL_Variable's static methods to include user-defined methods, allowing simplified syntax for serializing data of most variable types.

 

The specific example showed a technique for serializing IDL variables, turning them into compressed ASCII byte streams that can be more easily transmitted under certain protocols, for example via HTTP to ENVI Services Engine instances running on a cloud.

 

 

Object reference and structure data types are excluded from being extended by the IDL_Variable "superclass" definition that's common to other variable representations.

 

 

In this article I'll briefly discuss a mechanism to serialize objects and structures.

 

 

We can't retroactively extend IDL_Variable's static methods to provide the simplified syntax we want for objects and structures.

 

 

An alternate approach to solve the problem is to create a pair of static methods in a new class.  One method would take as input the reference to the variable to be serialized and return the serialization.  The second would invert the operation.

 

 

The static methods for compression and decompression uses the same technique described in the earlier article, calling IDL_BASE64 and ZLIB_COMPRESS or ZLIB_UNCOMPRESS for all variable types other than object reference and structure.

 

 

We need a different solution for them.

 

FUNCTION jp_Serializer::Serialize, v, DIM = Dim, TYPECODE = TypeCode
COMPILE_OPT STATIC
dim = v.Dim
typeCode = v.TypeCode
IF (typeCode EQ 8 || typeCode EQ 11) THEN BEGIN
    ; TODO What about structure (8) and object reference (11)?
ENDIF ELSE BEGIN
    RETURN, IDL_BASE64(ZLIB_COMPRESS(v))
ENDELSE
END

 

 

Notice that namespace partitioning argues for the use of an identifier unlikely to collide.  Here I'm using "jp_" as a prefix. 

 

 

One trick for serialization is to store the variable into an IDL SAVE file, read the file's contents via READ_BINARY into a BYTARR, then serialize the BYTARR using the same technique we apply to the other data.

 

 

    tempFile = FILEPATH('temp.sav', /TMP)
    SAVE, v, /COMPRESS, FILE = tempFile
    b = READ_BINARY(tempfile, DATATYPE = 1)
    FILE_DELETE, tempfile
    RETURN, IDL_BASE64(ZLIB_COMPRESS(b))

 

 

Note: A more robust implementation would generate a unique, unused file name, perhaps  a topic for another blog post.

 

 

In a similar fashion, the de-serialization converts the uncompressed byte stream, writes it to a file, then reads the data as a SAVE file.

 

 

    tempFile = FILEPATH('temp.sav', /TMP)
    OPENW, lun, tempFile, /GET_LUN
    WRITEU, lun, ZLIB_UNCOMPRESS(IDL_BASE64(var), TYPE = 1)
    FREE_LUN, lun
    RESTORE, tempFile
    FILE_DELETE, tempFile
    RETURN, v

 

 

We know that the variable to be returned is named "v" since that's the way it was saved in the serialization routine, above.  It has "magically" appeared in the context of the routine via RESTORE operation.

 

 

Here is the class with both static methods fully implemented

 

 

 

 

 

 

 

Here's an example of serialization of a "regular" variable

 

IDL> b = bindgen(2,2,2)
IDL> b
   0   1
   2   3

   4   5
   6   7
IDL> s = jp_Serializer.Serialize(b, DIM=dim, TYPECODE=type)
IDL> s,dim,type
eJxiYGRiZmFlYwcAAAD//wMAAFwAHQ==
           2           2           2
           1

 

And the de-serialization follows the same pattern

 

IDL> print, jp_Serializer.Deserialize(s, DIM=dim,TYPECODE=type)
   0   1
   2   3

   4   5
   6   7

 

Now let's see what happens with an object reference

 

IDL> o = orb()
IDL> so = jp_Serializer.Serialize(o)
IDL> so.strlen()
        9384
IDL> dso = jp_Serializer.Deserialize(so, TYPECODE=11)
IDL> dso
<ObjHeapVar28(ORB)>
IDL> xobjview, dso

 

 

Remember that when we SAVE an object reference, it makes a copy of the specified object's data. Additionally it makes a copy of any objects that are referenced within the specified object, so use this capability carefully.

 

 

Obviously, when passing serialized objects between IDL sessions, it's imperative that the required class and structure definitions are synchronized between those sessions and that class definitions are executed before the de-serialized file is restored.  Can you think of a technique for serializing and deserializing executable code to ensure compatibility, assuming the same release version of IDL?  (Hint: One technique would involve SAVE files for routines, very much like we've used above with data files.)

 

 

Why use static methods at all?  Why not simply create functions, in the traditional sense?

 

 

By grouping related functionality in a single class, I can easily extend the functionality without worrying about overwriting other functions.  The code is co-located in a single source file for easy maintenance.  Name space is defined by the class, providing some level of protection against conflicts.

 

 

It would be nice if there were a way to make a method "final" in the sense of other languages where the executive would disallow the re-compilation or sub-classing of methods at run-time.  This could be accomplished, say, via a new COMPILE_OPT.  Although I routinely take advantage of the flexible run-time behaviors that IDL offers relative to other languages, sometimes I'd like to have more control  How about you?