MRDFITS
Name
MRDFITS
Purpose
Read all standard FITS data types into arrays or structures.
Explanation
Further information on MRDFITS is available at
http://idlastro.gsfc.nasa.gov/mrdfits.html
**This version requires a post March 2009 version of fxposit.pro**
Calling Sequence
Result = MRDFITS( Filename/FileUnit,[Exten_no/Exten_name, Header],
/FPACK, /NO_FPACK, /FSCALE , /DSCALE , /UNSIGNED,
ALIAS=strarr[2,n], /USE_COLNUM,
/NO_TDIM, ROWS = [a,b,...], $
/POINTER_VAR, /FIXED_VAR, EXTNUM=
RANGE=[a,b], COLUMNS=[a,b,...]), ERROR_ACTION=x,
COMPRESS=comp_prog, STATUS=status, /VERSION,
/EMPTYSTRING )
Inputs
Filename = String containing the name of the file to be read or
file number of an open unit. If an empty string is supplied,
then user will be prompted for the file name. The user
will also be prompted if a wild card is given in the file
name, and there is more than one file name match.
If the file name ends in .gz or .fz (or .Z on Unix systems)
the file will be dynamically decompressed.
or
FiluUnit = An integer file unit which has already been
opened for input. Data will be read from this
unit and the unit will be left pointing immediately
after the HDU that is read. Thus to read a compressed
file with many HDU's a user might do something like:
lun=fxposit(filename, 3) ; Skip the first three HDU's
repeat begin
thisHDU = mrdfits(lun, 0, hdr, status=status)
... process the HDU ...
endrep until status lt 0
Exten_no= Extension number to be read, 0 for primary array.
Assumed 0 if not specified.
If a unit rather than a filename
is specified in the first argument, this is
the number of HDU's to skip from the current position.
Exten_name - Name of the extension to read (as stored in the EXTNAME
keyword). This is a slightly slower method then specifying
the extension number.
Outputs
Result = FITS data array or structure constructed from
the designated extension. The format of result depends
upon the type of FITS data read.
Non-group primary array or IMAGE extension:
A simple multidimensional array is returned with the
dimensions given in the NAXISn keywords.
Grouped image data with PCOUNT=0.
As above but with GCOUNT treated as NAXIS(n+1).
Grouped image data with PCOUNT>0.
The data is returned as an array of structures. Each
structure has two elements. The first is a one-dimensional
array of the group parameters, the second is a multidimensional
array as given by the NAXIS2-n keywords.
ASCII and BINARY tables.
The data is returned as a structure with one column for
each field in the table. The names of the columns are
normally taken from the TTYPE keywords (but see USE_COLNUM).
Bit field columns
are stored in byte arrays of the minimum necessary
length. Spaces and invalid characters are replaced by
underscores, and other invalid tag names are converted using
the IDL_VALIDNAME(/CONVERT_ALL) function.
Columns specified as variable length columns are stored
with a dimension equal to the largest actual dimension
used. Extra values in rows are filled with 0's or blanks.
If the size of the variable length column is not
a constant, then an additional column is created giving the
size used in the current row. This additional column will
have a tag name of the form L#_"colname" where # is the column
number and colname is the column name of the variable length
column. If the length of each element of a variable length
column is 0 then the column is deleted.
Optional Output
Header = String array containing the header from the FITS extension.
Optional Input Keywords
ALIAS The keyword allows the user to specify the column names
to be created when reading FITS data. The value of
this keyword should be a 2xn string array. The first
value of each pair of strings should be the desired
tag name for the IDL column. The second should be
the FITS TTYPE value. Note that there are restrictions
on valid tag names. The order of the ALIAS keyword
is compatible with MWRFITS.
COLUMNS - This keyword allows the user to specify that only a
subset of columns is to be returned. The columns
may be specified either as number 1,... n or by
name or some combination of these two.
If /USE_COLNUM is specified names should be C1,...Cn.
The use of this keyword will not save time or internal
memory since the extraction of specified columns
is done after all columns have been retrieved from the
FITS file. Structure columns are returned in the order
supplied in this keyword.
COMPRESS - This keyword allows the user to specify a
decompression program to use to decompress a file that
will not be automatically recognized based upon
the file name.
/DSCALE - As with FSCALE except that the resulting data is
stored in doubles.
/EMPTYSTRING - There was a bug in memory management for IDL versions
prior to V8.0, causing a memory leak when reading
empty strings in a FITS table. Setting /EMPTYSTRING will
avoid this problem by first reading strings into bytes and
then converting. However, there is a performance penalty.
ERROR_ACTION - Set the on_error action to this value (defaults
to 2).
/FIXED_VAR- Translate variable length columns into fixed length columns
and provide a length column for truly varying columns.
This was only behavior prior to V2.5 for MRDFITS and remains
the default (see /POINTER_VAR)
/FPACK - If set, then assume the FITS file uses FPACK compression
(http://heasarc.gsfc.nasa.gov/fitsio/fpack/). MRDFITS
will automatically detect FPACK compressed files, but it is
more efficient to supply the /FPACK keyword. A file with an
extension of .fz is assumed to be Fpack compressed.
/NO_FPACK - If present, then MRDFITS will not uncompress an extension
compressed with FPACK, but will just read the compressed
binary stream.
/FSCALE - If present and non-zero then scale data to float
numbers for arrays and columns which have either
non-zero offset or non-unity scale.
If scaling parameters are applied, then the corresponding
FITS scaling keywords will be modified.
NO_TDIM - Disable processing of TDIM keywords. If NO_TDIM
is specified MRDFITS will ignore TDIM keywords in
binary tables.
/POINTER_VAR- Use pointer arrays for variable length columns.
In addition to changing the format in which
variable length arrays are stored, if the pointer_var
keyword is set to any value other than 1 this also disables
the deletion of variable length columns. (See /FIXED_VAR)
Note that because pointers may be present in the output
structure, the user is responsible for memory management
when deleting or reassigning the structure (e.g. use HEAP_FREE
first).
RANGE - A scalar or two element vector giving the start
and end rows to be retrieved. For ASCII and BINARY
tables this specifies the row number. For GROUPed data
this will specify the groups. For array images, this
refers to the last non-unity index in the array. E.g.,
for a 3 D image with NAXIS* values = [100,100,1], the
range may be specified as 0:99, since the last axis
is suppressed. Note that the range uses IDL indexing
So that the first row is row 0.
If only a single value, x, is given in the range,
the range is assumed to be [0,x-1].
ROWS - A scalar or vector specifying a specific row or rows to read
(first row is 0). For example to read rows 0,
12 and 23 only, set ROWS=[0,12,23]. Valid for images, ASCII
and binary tables, but not GROUPed data. For images
the row numbers refer to the last non-unity index in the array.
Note that the use of the ROWS will not improve the speed of
MRDFITS since the entire table will be read in, and then subset
to the specified rows. Cannot be used at the same time as
the RANGE keyword
/SILENT - Suppress informative messages.
STRUCTYP - The structyp keyword specifies the name to be used
for the structure defined when reading ASCII or binary
tables. Generally users will not be able to conveniently
combine data from multiple files unless the STRUCTYP
parameter is specified. An error will occur if the
user specifies the same value for the STRUCTYP keyword
in calls to MRDFITS in the same IDL session for extensions
which have different structures.
/UNSIGNED - For integer data with appropriate zero points and scales
read the data into unsigned integer arrays.
/USE_COLNUM - When creating column names for binary and ASCII tables
MRDFITS attempts to use the appropriate TTYPE keyword
values. If USE_COLNUM is specified and non-zero then
column names will be generated as 'C1, C2, ... 'Cn'
for the number of columns in the table.
/VERSION Print the current version number
Optional Output Keywords
EXTNUM - the number of the extension actually read. Useful if the
user specified the extension by name.
OUTALIAS - This is a 2xn string array where the first column gives the
actual structure tagname, and the second gives the
corresponding FITS keyword name (e.g. in the TTYPE keyword).
This array can be passed directly to
the alias keyword of MWRFITS to recreate the file originally
read by MRDFITS.
STATUS - A integer status indicating success or failure of
the request. A status of >=0 indicates a successful read.
Currently
0 -> successful completion
-1 -> error
-2 -> end of file
Examples
(1) Read a FITS primary array:
a = mrdfits('TEST.FITS') or
a = mrdfits('TEST.FITS', 0, header)
The second example also retrieves header information.
(2) Read rows 10-100 of the second extension of a FITS file.
a = mrdfits('TEST.FITS', 2, header, range=[10,100])
(3) Read a table and ask that any scalings be applied and the
scaled data be converted to doubles. Use simple column names,
suppress outputs.
a = mrdfits('TEST.FITS', 1, /dscale, /use_colnum, /silent)
(4) Read rows 3, 34 and 52 of a binary table and request that
variable length columns be stored as a pointer variable in the
output structure
a = mrdfits('TEST.FITS',1,rows=[3,34,52],/POINTER)
Restrictions
(1) Cannot handle data in non-standard FITS formats.
(2) Doesn't do anything with BLANK or NULL values or
NaN's. They are just read in. They may be scaled
if scaling is applied.
Notes
This multiple format FITS reader is designed to provide a
single, simple interface to reading all common types of FITS data.
MRDFITS DOES NOT scale data by default. The FSCALE or DSCALE
parameters must be used.
Null values in an FITS ASCII table are converted to NaN (floating data),
or -2147483647L (longwords) or '...' (strings).
Procedures Used
The following procedures are contained in the main MRDFITS program.
MRD_IMAGE -- Generate array/structure for images.
MRD_READ_IMAGE -- Read image data.
MRD_ASCII -- Generate structure for ASCII tables.
MRD_READ_ASCII -- Read an ASCII table.
MRD_TABLE -- Generate structure for Binary tables.
MRD_READ_TABLE -- Read binary table info.
MRD_READ_HEAP -- Read variable length record info.
MRD_SCALE -- Apply scaling to data.
MRD_COLUMNS -- Extract columns.
Other ASTRON Library routines used
FXPAR(), FXADDPAR, FXPOSIT, FXMOVE(), MATCH, MRD_STRUCT(), MRD_SKIP
Modification History
V1.0 November 9, 1994 ---- Initial release.
Creator: Thomas A. McGlynn
V1.1 January 20, 1995 T.A. McGlynn
Fixed bug in variable length records.
Added TDIM support -- new routine mrd_tdim in MRD_TABLE.
V1.2
Added support for dynamic decompression of files.
Fixed further bugs in variable length record handling.
V1.2a
Added NO_TDIM keyword to turn off TDIM processing for
those who don't want it.
Bug fixes: Handle one row tables correctly, use BZERO rather than
BOFFSET. Fix error in scaling of images.
V1.2b
Changed MRD_HREAD to handle null characters in headers.
V2.0 April 1, 1996
-Handles FITS tables with an arbitrary number of columns.
-Substantial changes to MRD_STRUCT to allow the use of
substructures when more than 127 columns are desired.
-All references to table columns are now made through the
functions MRD_GETC and MRD_PUTC. See description above.
-Use of SILENT will now eliminate compilation messages for
temporary functions.
-Bugs in handling of variable length columns with either
a single row in the table or a maximum of a single element
in the column fixed.
-Added support for DCOMPLEX numbers in binary tables (M formats) for
IDL versions above 4.0.
-Created regression test procedure to check in new versions.
-Added error_action parameter to allow user to specify
on_error action. This should allow better interaction with
new CHECK facility. ON_ERROR statements deleted from
most called routines.
- Modified MRDFITS to read in headers containing null characters
with a warning message printed.
V2.0a April 16, 1996
- Added IS_IEEE_BIG() checks (and routine) so that we don't
worry about IEEE to host conversions if the machine's native
format is IEEE Big-endian.
V2.1 August 24, 1996
- Use resolve_routine for dynamically defined functions
for versions > 4.0
- Fix some processing in random groups format.
- Handle cases where the data segment is--legally--null.
In this case MRDFITS returns a scalar 0.
- Fix bugs with the values for BSCALE and BZERO (and PSCAL and
PZERO) parameters set by MRDFITS.
V2.1a April 24, 1997 Handle binary tables with zero length columns
V2.1b May 13,1997 Remove whitespace from replicate structure definition
V2.1c May 28,1997 Less strict parsing of XTENSION keyword
V2.1d June 16, 1997 Fixed problem for >32767 entries introduced 24-Apr
V2.1e Aug 12, 1997 Fixed problem handling double complex arrays
V2.1f Oct 22, 1997 IDL reserved words can't be structure tag names
V2.1g Nov 24, 1997 Handle XTENSION keywords with extra blanks.
V2.1h Jul 26, 1998 More flexible parsing of TFORM characters
V2.2 Dec 14, 1998 Allow fields with longer names for
later versions of IDL.
Fix handling of arrays in scaling routines.
Allow >128 fields in structures for IDL >4.0
Use more efficient structure copying for
IDL>5.0
V2.2b June 17, 1999 Fix bug in handling case where
all variable length columns are deleted
because they are empty.
V2.3 March 7, 2000 Allow user to supply file handle rather
than file name.
Added status field.
Now needs FXMOVE routine
V2.3b April 4, 2000
Added compress option (from D. Palmer)
V2.4 July 4, 2000 Added STATUS=-1 for "File access error" (Zarro/GSFC)
V2.4a May 2, 2001 Trim binary format string (W. Landsman)
V2.5 December 5, 2001 Add unsigned, alias, 64 bit integers. version, $
/pointer_val, /fixed_var.
V2.5a Fix problem when both the first and the last character
in a TTYPEnn value are invalid structure tag characters
V2.6 February 15, 2002 Fix error in handling unsigned numbers, $
and 64 bit unsigneds. (Thanks to Stephane Beland)
V2.6a September 2, 2002 Fix possible conflicting data structure for
variable length arrays (W. Landsman)
V2.7 July, 2003 Added Rows keyword (W. Landsman)
V2.7a September 2003 Convert dimensions to long64 to handle huge files
V2.8 October 2003 Use IDL_VALIDNAME() function to ensure valid tag names
Removed OLD_STRUCT and TEMPDIR keywords W. Landsman
V2.9 February 2004 Added internal MRD_FXPAR procedure for faster
processing of binary table headers E. Sheldon
V2.9a March 2004 Restore ability to read empty binary table W. Landsman
Swallow binary tables with more columns than given in TFIELDS
V2.9b Fix to ensure order of TFORMn doesn't matter
V2.9c Check if extra degenerate NAXISn keyword are present W.L. Oct 2004
V2.9d Propagate /SILENT to MRD_HREAD, more LONG64 casting W. L. Dec 2004
V2.9e Add typarr[good] to fix a problem reading zero-length columns
A.Csillaghy, csillag@ssl.berkeley.edu (RHESSI)
V2.9f Fix problem with string variable binary tables, possible math
overflow on non-IEEE machines WL Feb. 2005
V2.9g Fix problem when setting /USE_COLNUM WL Feb. 2005
V2.10 Use faster keywords to BYTEORDER WL May 2006
V2.11 Add ON_IOERROR, CATCH, and STATUS keyword to MRD_READ_IMAGE to
trap EOF in compressed files DZ Also fix handling of unsigned
images when BSCALE not present K Chu/WL June 2006
V2.12 Allow extension to be specified by name, added EXTNUM keyword
WL December 2006
V2.12a Convert ASCII table column to DOUBLE if single precision is
insufficient
V2.12b Fixed problem when both /fscale and /unsigned are set
C. Markwardt Aug 2007
V2.13 Use SWAP_ENDIAN_INPLACE instead of IEEE_TO_HOST and IS_IEEE_BIG
W. Landsman Nov 2007
V2.13a One element vector allowed for file name W.L. Dec 2007
V2.13b More informative error message when EOF found W.L. Jun 2008
V2.14 Use vector form of VALID_NUM(), added OUTALIAS keyword
W.L. Aug 2008
V2.15 Use new FXPOSIT which uses on-the-fly byteswapping W.L. Mar 2009
V2.15a Small efficiency updates to MRD_SCALE W.L. Apr 2009
V2.15b Fixed typo introduced Apr 2009
V2.15c Fix bug introduced Mar 2009 when file unit used W.L. July 2009
V2.16 Handle FPACK compressed files W. L. July 2009
V2.17 Use compile_opt hidden on all routines except mrdfits.pro W.L. July 2009
V2.18 Added /EMPTYSTRING keyword W. Landsman August 2009
V2.18a Fix Columns keyword output, A. Kimball/ W. Landsman Feb 2010
V2.18b Fix bug with /EMPTYSTRING and multidimensional strings
S. Baldridge/W.L. August 2010
V2.18c Fix unsigned bug caused by compile_opt idl2 WL Nov 2010
V2.19 Use V6.0 operators WL Nov 2010
V2.19a Fix complex data conversion in variable length tables WL Dec 2010
V2.19b Fix bug with /FSCALE introduced Nov 2010 WL Jan 2011
V2.19c Fix bug with ROWS keyword introduced Nov 2010 WL Mar 2011
V2.20 Convert Nulls in ASCII tables, better check of duplicate keywords
WL May 2011