CMSV_RDATA Name
CMSV_RDATA Author
Craig B. Markwardt, NASA/GSFC Code 662, Greenbelt, MD 20770
craigm@lheamail.gsfc.nasa.gov Purpose
Read SAVE-formatted data variable record from input block or file unit
Calling Sequence
CMSV_RDATA, BLOCK, POINTER, SIZE, DATA, UNIT=UNIT, $
TEMPLATE=TEMPLATE, /TEMPORARY, PTR_INDEX=PTR_INDEX, $
PTR_CALLBACK=PTR_CALLBACK, PTR_OFFSETS=PTR_OFFSETS, $
OFFSET=OFFSET, STATUS=STATUS, ERRMSG=ERRMSG
Description
CMSV_RDATA reads the data portion of an IDL SAVE variable record.
An IDL variable is stored in two components: the type descriptor
which describes the name, type, and dimensions of the variable;
and the data record, which contains the raw data of the variable.
This procedure reads the raw data and returns it to the user. The
initial type portion of the record must have already been read
using the CMSV_RVTYPE procedure.
CMSV_RDATA supports the following variable types:
BYTE(1),INT(2),LONG(3) - integer types
UINT(12),ULONG(13),LONG64(14),ULONG64(15) - integer types (IDL >5.2 only)
FLOAT(4),DOUBLE(5),COMPLEX(6),DCOMPLEX(9) - float types
STRING(7) - string type
STRUCT(8) - structure type
POINTER(10) - pointer type - SEE BELOW
NOT SUPPORTED - OBJ(11) - object reference type - NOT SUPPORTED
Arrays and structures containing any of the supported types are
supported (including structures within structures).
For scalars and arrays of numeric or string types, the caller must
only supply the SIZE parameter, which specifies the type and
dimensions of the variable to be read. This information can be
obtained from the CMSV_RVTYPE routine. The data is returned in the
output parameter DATA.
For structure data, in addition to specifying the SIZE array, the
user must also supply a "template" describing the structure into
which the data will be read. This template is simply a "blank"
form of the data structure, and is returned by CMSV_RVTYPE.
Thus, a simple way to read structure, numeric or string data is
the following code (with error checking removed)
CMSV_RVTYPE, block, pointer, name, size, template=template, unit=unit
CMSV_RDATA, block, pointer, size, data, template=template, unit=unit
[ This code assumes the record header has been read with
CMSV_RREC. ]
==================================================================
Research Systems, Inc. has issued a separate license intended
to resolve any potential conflict between this software and the
IDL End User License Agreement. The text of that license
can be found in the file LICENSE.RSI, included with this
software library.
==================================================================
POINTER DATA
Pointer data stored in IDL SAVE files are particularly difficult
to manage, because the actual heap variables are stored in
separate records which *precede* the record of interest. Thus, if
your application requires the reading of pointer data, you must
perform special processing in your own code in order to support
it. In essence, you must maintain an inventory of heap variables
as they are encountered in the file.
If these procedures are not followed then pointer data will not be
read, and a LONG integer value appears in the pointers' places.
Under IDL 4, pointer data can never be read.
This is accomplished by placing some additional logic in your file
processing loop. There are four separate components to this: (1)
loop initialization; (2) reading a HEAP_INDEX record; (3) parsing
a HEAP_DATA record; and (4) passing extra arguments to CMSV_RDATA.
The additional state information is maintained in two variables
named PTR_INDEX, which keeps track of the heap variable numbers,
and PTR_OFFSETS, which stores the file location of each variable.
(1) Loop initialization: is quite simple, use the following code:
ptr_index = [0L]
ptr_offsets = [0L]
ptr_data = [ptr_new()]
(2) Reading HEAP_INDEX, which is an array of values indicating
the heap variable numbers of each heap variables. These
values are stored in PTR_INDEX:
CMSV_RHEAP, block, pointer, index, unit=unit
ptr_index = [ptr_index, index]
ptr_offsets = [ptr_offsets, lonarr(n_elements(index))]
ptr_data = [ptr_data, ptrarr(n_elements(index))]
(3) Parse the HEAP_DATA record. Here were are interested in the
heap variable number, and the file offset.
opointer = pointer
CMSV_RVTYPE, block, pointer, vindex, /heap, unit=unit
vindex = floor(vindex(0))
wh = where(ptr_index EQ vindex)
ptr_offsets(wh(0)) = offset + opointer
Keep in mind that the file offset is OFFSET+POINTER.
(4) Pass extra parameters to CMSV_RDATA. The user simply passes
these extra variables to the CMSV_RDATA procedure, which
automatically recognizes heap data and reads it from the
appropriate location.
CMSV_RVTYPE, block, pointer, name, size, unit=unit, template=tp
CMSV_RDATA, block, pointer, size, data, template=tp, $
unit=unit, ptr_offsets=ptr_offsets, $
ptr_index=ptr_index, ptr_data=ptr_data
If this technique is used properly, only those heap variables
which are needed are read. Thus, there are never any lost or
dangling pointers. Since each bit of heap data is stored in a
variable returned to the user, it is not necessary to
PTR_FREE(ptr_data); in fact, doing so would corrupt the input
data.
BLOCK, POINTER, OFFSET
This procedure can read data from a byte array, a file unit, or
both. In fact, this procedure is designed to implement "lazy"
reading from a file, which is to say, it normally reads from a
byte array of data. However, if the requested data goes beyond
the end of the byte array, more data is read from the file on
demand. This way the user gets the benefit of fast memory access
for small reads, but guaranteed file access for large reads.
The terminology is as follows: BLOCK is a byte array which
represents a portion of, or an entire, IDL SAVE file. The block
may be a cached portion of an on-disk file, or an entire in-memory
SAVE file. POINTER is the current file pointer within BLOCK
(i.e., the next byte to be read is BLOCK[POINTER]). Hence, a
POINTER value of 0 refers to the start of the block. OFFSET is
the file offset of the 0th byte of BLOCK; thus "POINT_LUN,
OFFSET+POINTER" should point to the same byte as BLOCK[POINTER].
The following diagram shows the meanings for BLOCK, POINTER and
OFFSET schematically:
0 <- OFFSET -> |
FILE |----------------|------*--------|--------->
BLOCK |------*--------|
0 ^ POINTER
This procedure is part of the CMSVLIB SAVE library for IDL by
Craig Markwardt. You must have the full CMSVLIB core package
installed in order for this procedure to function properly.
Inputs
BLOCK - a byte array, a cache of the SAVE file. Users will
usually not access this array directly. Users are advised
to clear BLOCK after calling POINT_LUN.
POINTER - a long integer, a pointer to the next byte to be read
from BLOCK. CMSVLIB routines will automatically
advance the pointer.
SIZE - an array of integers describing the type and dimensions of
the variable to be read, in the format returned by the
SIZE() routine. This parameter is required.
DATA - upon output, the data variable. If any heap data is read,
the user is ultimately responsible for freeing it.
Keywords
UNIT - a file unit. If a library routine reads to the end of
BLOCK, or if BLOCK is undefined, then this file UNIT will
be accessed for more data. If undefined, then BLOCK must
contain the entire file in memory.
TEMPLATE - for structure data (data type 8), a "blank" structure
containing the fields and data values to be read in.
This structure is returned by CMSV_RVTYPE.
This keyword is mandatory for structure data.
TEMPORARY - if set, BLOCK becomes undefined upon return.
PTR_OFFSETS - array of file offsets, as described above. Default:
pointer data is converted to an integer.
PTR_INDEX - array of heap variable indices, as described above.
Default: pointer data is converted to an integer.
PTR_DATA - array of pointers, as described above.
Default: pointer data is converted to an integer.
OFFSET - the file offset of byte zero of BLOCK. Default: 0
(OFFSET is used by this routine)
STATUS - upon return, this keyword will contain 1 for success and
0 for failure.
ERRMSG - upon return with a failure, this keyword will contain the
error condition as a string. Example
See Also
CMRESTORE, SAVE, RESTORE, CMSVLIB
Modification History
Written, 2000
Documented, 24 Jan 2001
Added UNDEFINED data type for IDL >5.3, CM, 21 Apr 2001
Fixed bug for pointers within structures, CM, 21 Apr 2001
Add support for IDL 4 byte-compiled strings, CM, 22 Apr 2001
Make version checks with correct precision, 19 Jul 2001, CM
Added notification about RSI License, 13 May 2002, CM
Clarify and speed some of the code, 22 Nov 2009, CM
NOTE: remember to modify CMSVLIB.PRO when changing library!