How to retrieve output keywords from Call_Procedure

When working in IDL, often it will seem like doing something is just not possible. That is, until you talk to the right person who knows language tricks you never would have dreamed of. In my case that happened while I was working on our new ENVITask subclass for wrapping any IDL procedure. I wanted to use Call_Procedure to invoke it, but could not figure out how to get the values of output keywords from the procedure, so I had to resort to using the much more flexible but slower Execute() approach. Then I had a conversation with Adam Lefkoff, creator of the original version of ENVI and all around IDL guru, and he showed me a trick to take advantage of that works with Call_Procedure.

Let me step back a bit and describe the problem better before I demonstrate the solution. Let's say there is an IDL procedure that you want to call, but you have the names of the keywords as string literals instead of having a priori knowledge of them at compile time. Perhaps they come from a database or external source. You have two options for invoking this procedure: Execute and Call_Procedure:

pro testProcedure, FOO=foo, BAR=bar, _REF_EXTRA=refExtra

  compile_opt idl2

 

  help, foo, OUTPUT=fooOut

  help, bar, OUTPUT=barOut

  print, fooOut, barOut

 

  if (ISA(refExtra)) then begin

    foreach keyword, refExtra do begin

      value = Scope_VarFetch(keyword, /REF_EXTRA)

      help, value, OUTPUT=valueOut

      print, keyword, valueOut

    endforeach

  endif

end

 

pro Execute_Wrapper, procName, _REF_EXTRA=refExtra

  compile_opt idl2

 

  params = Hash()

 

  foreach keyword, refExtra do begin

    params[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

  endforeach

 

  print, 'in Execute_Wrapper'

  command = procName + ', _EXTRA=params.ToStruct()'

  ret = Execute(command)

end

 

pro Call_Procedure_Wrapper, procName, _REF_EXTRA=refExtra

  compile_opt idl2

 

  params = Hash()

 

  foreach keyword, refExtra do begin

    params[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

  endforeach

 

  print, 'in Call_Procedure_Wrapper'

  Call_Procedure, procName, _EXTRA=params.ToStruct()

end

 

pro test_procedure_wrapping

  compile_opt idl2

 

  Execute_Wrapper, 'testProcedure', FOO='foo', BAR=!pi, BAZ=List(1,2), QUX=Hash(1,2)

  Call_Procedure_Wrapper, 'testProcedure', FOO='foo', BAR=!pi, BAZ=List(1,2), QUX=Hash(1,2)

end

The example might be a bit contrived, but it demonstrates how you Scope_VarFetch() each of the values in _REF_EXTRA and put them into a Hash object. The Hash is then converted to a struct when invoking the procedure, and by assigning that struct to the _EXTRA keyword it will connect all the keywords appropriately. We can see from the output that both Execute and Call_Procedure work:

in Execute_Wrapper

FOO             STRING    = 'foo'

BAR             FLOAT     =       3.14159

BAZ VALUE           LIST  <ID=1  NELEMENTS=2>

QUX VALUE           HASH  <ID=6  NELEMENTS=1>

in Call_Procedure_Wrapper

FOO             STRING    = 'foo'

BAR             FLOAT     =       3.14159

BAZ VALUE           LIST  <ID=31  NELEMENTS=2>

QUX VALUE           HASH  <ID=36  NELEMENTS=1>

The problem is that the procedure only has input keywords, no output keywords. If we want to be able to dynamically invoke a procedure that has output keywords, a little more effort is required. Here is the updated version of the Execute wrapper in action:

 

pro testProcedure, FOO=foo, BAR=bar, OUT_HASH=outHash, _REF_EXTRA=refExtra

  compile_opt idl2

 

  outHash = Hash('FOO', foo, 'BAR', bar)

 

  if (ISA(refExtra)) then begin

    foreach keyword, refExtra do begin

      outHash[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

    endforeach

  endif

end

 

pro Execute_Wrapper, procName, OUT_NAMES=outNames, _REF_EXTRA=refExtra

  compile_opt idl2

 

  params = Hash()

 

  foreach keyword, refExtra do begin

    if (outNames.HasValue(keyword)) then continue

    params[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

  endforeach

 

  print, 'in Execute_Wrapper'

  command = procName + ', _EXTRA=params.ToStruct()'

  foreach name, outNames do begin

    command += ', ' + name.ToUpper() + '=' + name.ToLower()

  endforeach

  ret = Execute(command)

 

  foreach name, outNames do begin

    (Scope_VarFetch(name, /REF_EXTRA)) = Scope_VarFetch(name)

  endforeach

end

 

pro test_procedure_wrapping

  compile_opt idl2

 

  execOutHash = 0

  Execute_Wrapper, 'testProcedure', FOO='foo', BAR=!pi, BAZ=List(1,2), QUX=Hash(1,2), OUT_NAMES='OUT_HASH', OUT_HASH=execOutHash

  print, execOutHash, /IMPLIED

end

The Execute_Wrapper routine now requires knowledge of which keywords in _REF_EXTRA are input and which are output. So when we construct the Hash we omit the output keywords, but then append them to the command string  as new variables in the local scope. After invoking the procedure, these local variables will have the correct output values, but we need the very odd looking line with two calls to Scope_VarFetch() to copy them into the _REF_EXTRA bag to get them back out to the routine that called Execute_Wrapper. The Scope_VarFetch() call on the left uses the /REF_EXTRA keyword to connect with the callstack, but we need to encapsulate it in parentheses to make it an assignment operation. The Scope_VarFetch() call on the right is used to get the value of the named local variable for this assignment. We also need to make sure that the OUT_HASH keyword is present in the call to Execute_Wrapper so that this _REF_EXTRA trick will work.

While this is functional, it looks a little complicated and redundant with the OUT_HASH keyword repeated as a string literal. The other problem is that it is still using Execute(), which isn't as efficient as Call_Procedure. But we can't modify the Call_Procedure_Wrapper routine in the same manner and have success. The reason for this is that when the struct is built to pass into Call_Procedure it contains copies of the values of each keywords, not references to the variables used to build it. So we need to introduce a new function to get the output keyword values:

pro testProcedure, FOO=foo, BAR=bar, OUT_HASH=outHash, _REF_EXTRA=refExtra

  compile_opt idl2

 

  outHash = Hash('FOO', foo, 'BAR', bar)

 

  if (ISA(refExtra)) then begin

    foreach keyword, refExtra do begin

      outHash[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

    endforeach

  endif

end

 

function Procedure_Wrapper, procName, _REF_EXTRA=refExtra

  compile_opt idl2

 

  Call_Procedure, procName, _EXTRA=refExtra

 

  results = Hash()

  foreach keyword, refExtra do begin

    results[keyword] = Scope_Varfetch(keyword, /REF_EXTRA)

  endforeach

 

  return, results

end

 

pro Call_Procedure_Wrapper, procName, OUT_NAMES=outNames, _REF_EXTRA=refExtra

  compile_opt idl2

 

  params = Hash()

 

  foreach keyword, refExtra do begin

    params[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

  endforeach

 

  print, 'in Call_Procedure_Wrapper'

  results = Procedure_Wrapper(procName, _EXTRA=params.ToStruct())

 

  foreach name, outNames do begin

    (Scope_VarFetch(name, /REF_EXTRA)) = results[name]

  endforeach

end

 

pro test_procedure_wrapping

  compile_opt idl2

 

  callOutHash = 0

  Call_Procedure_Wrapper, 'testProcedure', FOO='foo', BAR=!pi, BAZ=List(1,2), QUX=Hash(1,2), OUT_NAMES='OUT_HASH', OUT_HASH=callOutHash

  print, callOutHash, /IMPLIED

end

This new function behaves similarly to the modified Execute_Wrapper routine, but since it passes all of its keywords through to Call_Procedure it uses references not values. It can then use Scope_VarFetch(/REF_EXTRA) to copy the values from those keywords into a Hash that it returns to its caller. In there we again use Scope_VarFetch(/REF_EXTRA) to copy the output values from that Hash into the variables from its calling routine. The use of the OUT_NAMES keyword is not completely necessary, but it avoids copying input variables back to the caller, which may or may not be expensive.

In the ENVITask context, we know which keywords are inputs and which are outputs, so instead of passing in the OUT_NAMES keyword and returning a Hash of value we can make Procedure_Wrapper a member function and set the output parameter values directly. But as you'll see in ENVI 5.2 SP1 we can wrap any IDL procedure that uses keywords in an ENVITaskFromProcedure object that knows how to map input and output keywords to ENVITaskParameters for you automatically.