12924 Rate this article:

Using IDL_IDLBridge to run multiple processes in parallel


I noticed a question on the newsgroup about running multiple time consuming IDL commands in parallel processes. This is a topic that we in the Custom Solutions group come accross fairly frequently. Below is a simple example of how to manage a number of processes running at the same time.

Notice in the code, that it is often important to control the working directory as well as the search path for all of the bridge processes so that the bridge processes can find the necessary IDL code as well as any data needed.

pro multi_process
  compile_opt idl2, logical_predicate
  cd, current=dir
  n_proc = 10
  tlb = widget_base(title='Initializing',/column, xsize=500, ysize=500)
  t = widget_text(tlb, value='Initializing...')
  widget_control, tlb, /realize
  br = objarr(n_proc)
  for i=0, n_proc-1 do begin
    br[i] = IDL_IDLbridge()
  widget_control, tlb, /destroy
  tlb = widget_base(title='Processing progress',/column)
  t = widget_table(tlb, xsize=2, ysize=n_proc, value=strarr(2,n_proc), $
    column_labels=['Process #','Status'], column_width=[250,250])
  widget_control, tlb, /realize
  ; Monitor progress
  status = replicate(1, n_proc)
  table = strarr(2,n_proc)
  table[0,*] = strtrim(indgen(n_proc),2)
  codes = ['Idle','Executing','Completed','Error','Aborted']
  while max(status) gt 0 do begin
    wait, 1
    for i=0, n_proc-1 do begin
      status[i] = br[i]->Status()
    table[1,*] = codes[status]
    widget_control, t, set_value=table
  widget_control, tlb, /destroy

This code example will display a simple GUI showing the status of the IDL processes.

This is a great approach if you want to speed up some time consuming IDL processes that can run individually on the same system. In more complex situations, you want to instead look into ENVI Services Engine as it provides much more functionality for configuring how "jobs" are being run, and also supports distributing jobs over multiple systems (nodes). Despite the word "ENVI" here, it also supports running pure IDL code.