Batch processing with IDL

Following the theme of Mike Galloy's post, I'd like to show an example of noninteractive batch processing with IDL. The example below uses Linux and bash, though I used to do similar things on Solaris (and SunOS before that!) with tcsh, so it should work with slight modifications on other UNIX-based operating systems and command shells. I'll start with my canonical bash statement:

$ nohup nice $IDL_DIR/bin/idl < in.pro > out.txt 2> err.txt &

The first two commands are optional: nohup allows a process to continue even if you've started it on a remote server then disconnected from it, while nice decrements the priority of the process (to be "nice" to other users), thus preventing the process from grabbing all the CPU time on the machine. Because nohup and nice don't recognize aliases, I directly call the IDL executive script in the bin/ directory of the IDL distribution. If you don't use nohup and nice, you can invoke IDL with the standard "idl" alias instead. The character "<" redirects stdin, the contents of the file in.pro, to IDL; the character ">" redirects stdout to the file out.txt. Console and error messages are redirected with "2>" to err.txt. The ampersand "&" backgrounds the process, allowing interactive shell access. Some detail is wrapped up in the file in.pro. It's an IDL batch file, which consists of single IDL statements, no loops (though there is a trick to get around this). A batch file is not a program—it cannot be compiled. Rather, each statement in the file is interpreted and executed sequentially by IDL. Here's in.pro:

print, 'Job started'
n = 100
task1, n
;task2
;task3
print, 'Job complete'

Note there is no END statement. The job of in.pro is to line up the tasks I want to execute; here, I've called them TASK1, TASK2, etc., each of which is an IDL program (I've chosen to use procedures to represent the tasks, but any program type will work). In this example, I'll execute a single task:

pro task1, n
   compile_opt idl2

   if n eq !null then n = 100

   x = findgen(n)
   p = plot(x, /buffer, title='Plot #000')
   p.save, 'plot000.png', resolution=96

   for i=1, n-1 do begin
      si = string(i, format='(i3.3)')
      p.title = 'Plot #' + si
      p.setdata, x + i
      p.save, 'plot' + si + '.png', resolution=96
   endfor

   p.close
end

TASK1 generates a series of N plots in an offscreen buffer, saving each plot, in turn, to a PNG file. It also serves as a simple example of batching with (New) Graphics—note the use of the SetData method to replace the plot data in each iteration. After executing the bash statement above, here's the contents of out.txt:

$ cat out.txt
Job started
Job complete

and err.txt:

$ cat err.txt
IDL Version 8.2.1 (linux x86_64 m64). (c) 2012, Exelis Visual Information Solutions, Inc.
Trial version expires on 30-sep-2013.
Licensed for personal use by ITTVISEvaluation Purposes Only only.
All other use is strictly prohibited.

% Compiled module: TASK1.
% Loaded DLM: PNG.

and a (truncated) directory listing:

$ ls | head
err.txt
in.pro
out.txt
plot000.png
plot001.png
plot002.png
plot003.png
plot004.png
plot005.png
plot006.png

Success! I've shown here a simple example of batch processing with IDL. Each step of the technique I've outlined above can have added complexity, which I hope to elaborate on in future posts. I hope you find this example useful, at least as a starting point.