The IDLAsyncSpawnJob class represents a unit of work to be done at some point in the future by spawning an external process to perform the work. On creation, specify the scalar string to be passed into that process via STDIN. After the job is complete, you can retrieve the contents of the process's STDOUT output. Submitting these to an IDLAsyncQueue will allow them to be performed as resources become available.

Superclasses

Examples

; Use S3 folder for a dataset covering Boulder, CO, taken April 4, 2017
folderUrl = 'https://s3-us-west-2.amazonaws.com/landsat-pds/L8/033/032/LC80330322017104LGN00'
; Dataset name is same as folder name, and is root of all files
datasetName = File_BaseName(folderUrl)
 
; List of suffixes to apply to dataset name for all files needed to
; comprise a full Landsat 8 dataset.  Band 8 is the panchromatic band,
; which is the largest, so start downloading it first.
suffixes = [ '_B8.TIF', $
             '_B1.TIF', $
             '_B2.TIF', $
             '_B3.TIF', $
             '_B4.TIF', $
             '_B5.TIF', $
             '_B6.TIF', $
             '_B7.TIF', $
             '_B9.TIF', $
             '_B10.TIF', $
             '_B11.TIF', $
             '_BQA.TIF', $
             '_MTL.txt' ]
 
; create a folder in the temp folder for the local files
localFolder = FilePath(datasetName, /TMP)
if (~File_Test(localFolder)) then File_MkDir, localFolder
 
; construct IDLAsyncJoin object to use for waiting on all jobs to complete
oJoin = IDLAsyncJoin()
; construct IDLAsyncQueue to manage parallel execution of jobs
oQueue = IDLAsyncQueue(CONCURRENCY=6)
; create object array to hold all jobs
jobs = ObjArr(N_Elements(suffixes))
 
; Define compound command to execute in the bridge.
; This command creates an IDLnetURL object, then calls its Get() method,
; passing in the remote URL and local filename to use.  The command must
; also exit IDL so that the job can terminate.
; An alternate command that works using a procedure is:
;   idlCmdFormat = 'download_S3_URL, S3_URL=''%s'', LOCAL_FILE=''%s'' & exit'
idlCmdFormat = 'u = IDLnetURL() & !null = u.Get(URL=''%s'', FILE=''%s'') & exit'
 
case (!version.OS) of
    'Win32': idlExecutable = FilePath('idl.exe', SUBDIR=['bin', 'bin.'+!version.ARCH])
    else: idlExecutable = FilePath('idl', SUBDIR=['bin'])
endcase
 
; construct a unique job for each remote file to download
foreach suf, suffixes, i do begin
    file = datasetName + suf
    remote = folderUrl + '/' + file
    local = FilePath(file, ROOT=localFolder)
    ; populate the IDL command string with current file suffix
    idlCmd = String(Format=idlCmdFormat, remote, local)
    ; construct IDL Command Line spawn command
    execCmd = idlExecutable + ' -e "' + idlCmd + '"'
    ; construct IDLAsyncSpawnJob to execute compound command in IDL Command Line,
    ; passing in join object
    jobs[i] = IDLAsyncSpawnJob(execCmd, JOIN=oJoin)
endforeach
 
tic
; submit all the jobs to the queue for execution
oQueue.SubmitJob, jobs
; wait for all jobs to be done
oJoin.WaitForJoin
toc

Syntax

Result = IDLAsyncSpawnJob(Exec [, JOIN=IDLAsyncJoin] [, STDIN=String] [, WAIT=Float])

Arguments

Exec

Specify a scalar string for the external application to spawn. Standard use of the PATH environment variable and current working directory will apply if this is a relative path. This string can contain command line arguments if needed.

The STDERR keyword is not set when calling SPAWN, so output on STDERR will not be captured unless you make that part of this command string. You can redirect STDERR to a file or multiplex it into stdout as needed.

Note: On Windows the /NOSHELL keyword is set in the call to SPAWN, to ensure that standard redirection syntax can be used.

Note: On Unix the /SH keyword is set in the call to SPAWN, to ensure the Bourne shell is used and standard redirection syntax can be used.

Keywords

JOIN (optional)

Specify an IDLAsyncJoin object that is passed to the IDLAsyncJob base class Init() method for handling.

STDIN (optional)

Specify a scalar string that will be sent to the external application via STDIN.

WAIT (optional)

Specify the amount of time in seconds to wait between polling the external process for output or completion. If this is not set, then the default value is 0.1 seconds.

Methods

Properties

IDLAsyncSpawnJob inherits all properties from IDLAsyncJob. The IDLAsyncSpawnJob properties below are Get-only.

COMMAND

The external process to spawn when this job is run. This was the Exec argument used in constructing this job.

EXIT_STATUS

The exit code of the spawned external process when it terminates.

PID

The process ID of the spawned external process.

STDOUT

A List containing all the lines of text the spawned external process wrote to STDOUT.

IDLAsyncSpawnJob::GetReturnValue

The IDLAsyncSpawnJob::GetReturnValue method is an override of the IDLAsyncJob interface method. The only variable that can be retrieved is STDOUT, which is a List containing all the string output read from the STDOUT pipe connected to the external process.

Note: This will only contain information from STDERR multiplexed into STDOUT if that redirection is specified in the COMMAND property.

Syntax

IDLAsyncSpawnJob.GetReturnValue, STDOUT=list

Arguments

None

Keywords

STDOUT

The contents read from STDOUT from the external application. This is stored as a list of strings, to allow users to insert carriage returns between lines as needed.

IDLAsyncSpawnJob::OnDone

The IDLAsyncSpawnJob::OnDone method is an implementation override of the IDLAsyncJob callback invoked on this job by the Done event handler. This will ensure the LUN returned in the UNIT keyword of Spawn will be freed.

Syntax

IDLAsyncSpawnJob.OnDone

Arguments

None

Keywords

None

IDLAsyncSpawnJob::OnStart

The IDLAsyncSpawnJob::OnStart method is an implementation override of the IDLAsyncJob callback invoked on this job by the Start event handler. The basic sequence of operations is:

  • Spawn the process, opening a pipe for communication.
  • Print the input data to the pipe to be sent to STDIN.
  • Create a timer to wait for the process to complete.

Syntax

IDLAsyncSpawnJob.OnStart

Arguments

None

Keywords

None

Version History

8.7

Introduced

See Also

IDLAsyncJob