Using the 'IDLnetURL' Class to Execute an FTP GET

Topic
The 'IDLnetURL' class was just introduced in IDL 6.4. It has a dual role of implementing an easy interface to both HTTP messaging as well as FTP messaging. The initial documentation for this new class shows an example of the use of this class's 'Get' method, but only in the context of 'HTTP' protocol, not 'FTP' protocol. Since the behavior of 'Get' is somewhat different for FTP than for HTTP, this Help Article is designed to show a comprehensive example of FTP GET'ing which illustrates many of the intricate facets of the IDL approach.

Discussion
If you are creating an 'IDLnetURL' object for the sole purpose of FTP GET'ing a file whose FTP path location is already well-known to you, then the syntax of the new 'IDLnetURL' class is pretty uncomplicated:

; Import the National Weather Service's ASCII text file
; 'denver.txt' to the local hard drive
oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME='ftp', $
URL_HOST='tgftp.nws.noaa.gov', $
URL_USERNAME='anonymous', URL_PASSWORD='', $
URL_PATH='data/forecasts/city/co/denver.txt')
myDownloadRelativePath = 'My Documents\denver_forecast.txt'
downloadLoc = oURL->Get(FILENAME=myDownloadRelativePath)
print, downloadLoc
;C:\Documents and Settings\jjones\My Documents\denver_forecast.txt

Two critical points to remember are a) that 'Get( )' only knows the location of the target file via the URL_PATH property of the IDLnetURL object, and b) the FILENAME keyword to 'Get( )' defines the path to the destination file on the hard drive; it is not the path to the source file on the FTP site.

If you are creating an 'IDLnetURL' object for the purpose of first getting a directory listing of a particular target directory before GET'ing any of the files in that directory, watch out for the following syntax pitfalls:

; Note how the URL keyword, unlike URL_PATH above, needs a
; complete URL specification
; *** WHAT NOT TO DO ***
oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME='ftp', $
URL_HOST='tgftp.nws.noaa.gov', $
URL_USERNAME='anonymous', URL_PASSWORD='')
print, oURL->GetFTPDirList(URL='data/forecasts/city/co'), $
FORMAT='(A)'
;IDLNETURL::GETFTPDIRLIST: CCurlException: Error: url does not
;contain a : cannot locate scheme in url (typically http:)..
; *** THE CORRECT WAY ***
print, oURL->GetFTPDirList( $
URL='ftp://tgftp.nws.noaa.gov/data/forecasts/city/co'), $
FORMAT='(A)'
;-rw-r--r-- 1 3108 1 333 Aug 15 20:39 alamosa.txt
;-rw-r--r-- 1 3108 1 331 Aug 15 20:19 aspen.txt
;-rw-r--r-- 1 3108 1 etc. ...

This is another option for implementing an FTP "DIR" command:

; Note how the URL_PATH ***must*** be a directory path
; *** WHAT NOT TO DO ***
oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME='ftp', $
URL_HOST='tgftp.nws.noaa.gov', $
URL_USERNAME='anonymous', URL_PASSWORD='', $
URL_PATH='data/forecasts/city/co/denver.txt')
print, oURL->GetFTPDirList(), FORMAT='(A)'
;IDLNETURL::GETFTPDIRLIST: CCurlException: 
;Error: Ftp Dir Request Failed. Error = Server denied you
;to change to the given directory, Curl Error Code = 9.
;Last Ftp Response = 550 Failed to change directory.
; *** THE CORRECT WAY ***
oURL->SetProperty, URL_PATH='data/forecasts/city/co'
print, oURL->GetFTPDirList(), FORMAT='(A)'
;-rw-r--r-- 1 3108 1 333 Aug 15 20:39 alamosa.txt
;-rw-r--r-- 1 3108 1 331 Aug 15 20:19 aspen.txt
;-rw-r--r-- 1 3108 1 etc. ... ; In a subsequent FTP "GET" you need to reset the URL_PATH to a
; file path now to avoid the below error
oURL->Get(FILENAME='denver.txt') ; Misunderstanding of FILENAME
IDL> print, oURL->Get()
;IDLNETURL::GET: CCurlException: Error: Ftp Get Request Failed.
;Error = RETR response: 550, Curl Error Code = 19.
;Last Ftp Response = 550 Failed to open file.
; *** THE CORRECT WAY***
oURL->SetProperty, URL_PATH='data/forecasts/city/co/denver.txt'
downloadLoc = oURL->Get(FILENAME='/tmp/denver_forecast.txt')

The example code below shows one way to do a basic "mining" of all the files found in one given directory on an FTP site. Note that to have a maximally robust implementation of the below, a programmer should consider setting both the /VERBOSE and the CALLBACK_FUNCTION properties of the IDLnetURL object. That is demonstrated in IDL Online Help, but is outside the focus of this tech tip.

Solution:

; Filename: 'import_colorado_weather_forecast_files'
; Calling Syntax: IMPORT_COLORADO_WEATHER_FORECAST_FILES
PRO import_colorado_weather_forecast_files
; First get the directory contents
targetDir = 'data/forecasts/city/co'
oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME='ftp', $
    URL_HOST='tgftp.nws.noaa.gov', $
    URL_USERNAME='anonymous', URL_PASSWORD='', $
    URL_PATH=targetDir)
dirArray = oURL->GetFTPDirList()
; A quick way to get just the file names
nFiles = n_elements(dirArray)
targetFiles = strarr(nFiles)
for i = 0, nFiles-1 do begin
    lineElements = strsplit(dirArray[i], /EXTRACT)
    ; Get the last element of the split string
    targetFiles[i] = lineElements[n_elements(lineElements)-1]
endfor
; GET all the target FTP files into our download dir using
; the same file names as are used by the FTP site.
; NOTE: This FOR loop could be combined with the above.
downloadDir = 'C:\temp'
for i = 0, nFiles-1 do begin
    ; For each GET we must update the URL_PATH property
    oURL->SetProperty, URL_PATH=targetDir + '/' + targetFiles[i]
    downloadPath = filepath(targetFiles[i], ROOT_DIR=downloadDir)
    print, oURL->Get(FILENAME=downloadPath)
endfor
obj_destroy, oURL
END