WEBGET
Name
WEBGET()
Purpose
Use the IDL SOCKET procedure to get data from http servers
Explanation
WEBGET() can access http servers - even from behind a firewall -
and perform simple downloads. Currently, text and FITS files can be
accessed.
Calling Sequence
a=webget(URL)
Inputs
URL - scalar string giving a fully qualified url of the form
'http://server.eso.org/path/file.html'. WEBGET() can
also use other valid URLs that contain 'GET' or 'POST' codes.
Optional Input Keyword Parameters
COPYFILE - if set to a valid filename (file must have write permission),
the data contents of the web server's answer is copied to that
file.
HTTP10 - If set, then use the HTTP 1.0
POST - if set to a structure, the structure tags and values
will be used as post variables and POST'ed to the URL.
If POST is not set, the normal HTTP GET is used to
retrieve the URL.
/SILENT - If set, the information error messages are suppressed
TIMEOUT - Integer scalar giving number of seconds to wait to connect
or for data to arrive before giving up and issuing an error.
Default=15 seconds
OUTPUTS: A structure with the following fields:
.Header - the HTTP header sent by the server
.Text - The text part of the downloaded file. If the
content type of the file was not of class
'text', this will be an empty string.
.ImageHeader - Header file of a FITS-image. FITS images
are read when the content type is
'image/fits' or 'application/octet-stream'
(for dss-access). If the file is not a FITS
image, this will be an empty string.
.Image - The FITS image read from the server. If the file
did not contain a FITS image, this will be zero.
Restrictions
The mime-type recognition is extremely limited. Only the content-type is
determined. Any text-file will be stored in out.Text. The only other
category which can be fetched is FITS files, which will be stored in
out.Image and out.ImageHeader.
PROXY: If you are behind a firewall and have to access the net through a
Web proxy, set the environment variable 'http_proxy' to point to
your proxy server and port, e.g.
'setenv http_proxy=http://web-proxy.mpia-hd.mpg.de:3128'
The URL *MUST* begin with "http://".
Procedure
Open a socket to the webserver and download the header. After deciding
whether it is text or binary, either store the text or try to read a
FITS file.
Example
IDL> a=webget('http://www.mpia.de/index.html')
IDL> print,a.Text
or
> PointingRA=0.0
> PointingDE=30.0
> QueryURL = strcompress("http://archive.eso.org/dss/dss/image?ra="+$
> string(PointingRA)+$
> "&dec="+$
> string(PointingDE)+$
> "&x=10&y=10&Sky-Survey=DSS1&mime-type=download-fits", $
> /remove)
> a=webget(QueryURL)
> tvscl,a.Image
> print,a.ImageHead
Modification History
Written by M. Feldt, Heidelberg, Oct 2001 <mfeldt@mpia.de>
Use /swap_if_little_endian keyword to SOCKET W. Landsman August 2002
Less restrictive search on Content-Type W. Landsman April 2003
Modified to work with FIRST image server- A. Barth, Nov 2006
Better recovery from errors W. Landsman April 2007
Add support for POST access J.D. Smith June 2007
Recognize "fits" image type used by SKYVIEW W. Landsman June 2007
Upgraded, partially, to HTTP 1.1 M. Perrin, July 2007
The HTTP 1.1 support is presently INCOMPLETE: virtual servers are
supported, but chunked transfer encoding is not yet supported, so
technically this is not fully HTTP 1.1 compliant.
Added http10 keyword W. Landsman August 2007
Assume since V5.6, sockets always available W. Landsman Nov 2007
Fix problem when using proxy server W. Landsman July 2008
Fix problem with /SILENT keyword W. Landsman Jan 2009
Added check for missing Mime TYPE in CLASSANDTYPE, Zarro, December 2011
Timeout applies to connecting as well as reading, default is now 15
seconds W Landsman January 2012