Server Side TCP/IP Sockets Officially Documented in IDL 8.5

Although the functionality to create a server-side TCP/IP SOCKET has been present since IDL 6.3, until IDL 8.5 it's not been officially documented.

Imagine, for instance, interactively sharing data, graphics, and code with remote users of IDL without leaving the friendly confines of the IDL Workbench.

Despite its stealthy nature, this feature has been used by our Custom Solutions Group for over a decade on a wide variety of projects. Oh yes. It's very nice.

It's even been discussed in passing in a previous IDL Data Point blog and IDL newsgroup public forums over the years, all the way back to 2006.

Establishing a listener is as simple as adding the "new" /LISTEN keyword. The online help is a little cryptic about what this means:

LISTEN

Set this keyword to tell this socket to listen on the specified port.

The general syntax is

SOCKET, listenerLUN, listenerPort, /GET_LUN, /LISTEN

Unlike the low-level BSD sockets, the IDL form of the listener does not allow for the option of multiple simultaneous connection attempts.For the sake of simplicity,

I'm omitting the various *_TIMEOUT keywords. But in the real world, you will always want to set CONNECT_TIMEOUT, READ_TIMEOUT, and WRITE_TIMEOUT to non-zero values any time you create a SOCKET. The default is to set an "infinite" timeout, which makes debugging in particular more difficult than you would generally want.

The listener socket simply waits for attempted connections on the specified port from a client. The listener socket is not the socket that's responsible for the data transfer.

How does your code know when a connection has been attempted from a client? This isn't obvious from the docs, but the trick is to call the FILE_POLL_INPUT function and check for a non-zero status return value. 

status = FILE_POLL_INPUT(listenerLUN, TIMEOUT = .01)

Select your own TIMEOUT value based on your system requirements. This check is generally performed within the context of a TIMER callback so the main IDL thread doesn't need to block until an asynchronous connection attempt is made from a client.

When the status changes to 1, the server side code then makes a second call to SOCKET, passing the logical unit number used in the first SOCKET call as the argument to the new ACCEPT keyword. This keyword is also cryptically documented.

ACCEPT

Set this keyword to only accept communications on a specified LUN.

The listener socket essentially "hands off" the actual data communication between the client and server to this second socket.

IF (status) THEN BEGIN

  SOCKET, clientLUN, ACCEPT = listenerLUN, /GET_LUN, /RAW_IO

ENDIF

All your read and write operations will be over this new clientLUN rather than the listenerLUN.

At this point your server is free to respond to additional connection requests on the listener socket. In this way you can potentially connect to multiple clients simultaneously. But recall that IDL's interpreter loop is single-threaded.  Plan your protocol wisely.

If you're exchanging binary data such as floating point numbers or integers across different platforms, you may wish to set the SOCKET keywords /SWAP_IF_BIG_ENDIAN or /SWAP_IF_LITTLE_ENDIAN on either the client or server, depending on your needs.

The two ends of the connection are complete. Now the processes simply need to exchange data according to a protocol of your choosing via WRITEU and READU, or PRINTF and READF.

The transfer protocol generally proceeds with a different TIMER-driven asynchronous monitoring of the data on the socket, with the same FILE_POLL_INPUT idiom used to check for availability.

At the present time, there is no built-in IDL interpreter for HTML (above and beyond the DOM or SAX XML parsers.) Helpers such as the new IDL-Python bridge, also new in IDL 8.5, may provide tools for rapid development.

Particularly when exchanging "large" chunks of data I recommend that you should use the keyword TRANSFER_COUNT to ensure your data buffer has transferred completely. For example if you have a "large" data buffer, the READU operation over the socket is not guaranteed to transfer the entire buffer at one time.

Recall that on the receiver side, you can't READU into an expression like data[tc:*]. Your code needs to account for that as well, building up the final vector in "chunks" as they arrive.

In a scheme like this, your protocol may predefine data sizes to be passed back and forth so sender and receiver agree. Alternatively, the sender must first send over the socket as part of the protocol the length and type of data prior to sending the data in order for the receiver to build the appropriate input buffer before attempting to read into it.

For small-scale tasks that could benefit from IDL services over sockets, this may represent a substantial portion of the solution you need. For large-scale projects, those that require native multi-threading, enhanced security, or the support of specific protocols such as HTTP, the ENVI Services Engine or Jagwire may be more appropriate for exposing ENVI and IDL functionality to TCP/IP clients.

Below are a pair of routines, one that acts a server and another that acts as a client. They can be executed on the same node, but you should run them in two different IDL processes, perhaps one in the Workbench and the other from the command line. You will likely want to set the !DEBUG_PROCESS_EVENTS system variable to 0 if you want to set breakpoints and observe the code behavior. The routines make heavy use of TIMER functionality and there will be unexpected interactions if !DEBUG_PROCESS_EVENTS is 1 and a breakpoint is reached.

If you want to run these examples on separate nodes, modify the SOCKET call in the IDLClient routine to point to the server box instead of 'localhost'.

The server-side code follows:

Pro ClientCallback, ID, H
Compile_Opt IDL2
Catch, ErrorNumber
If (ErrorNumber ne 0) then Begin
    Catch, /Cancel
    ; Unable to send for some reason.  Try HELP, /LAST_MESSAGE
    ; if you want to know why.  Try again.
    !null = Timer.Set(.01, 'ClientCallback', H)
    Return
EndIf
; Send 10,000 random numbers as integers to the client.
Buffer = UInt(RandomU(seed, 1.e5)*5)
WriteU, H['lun'], Buffer, Transfer_Count = TC
If (TC ne 0) then begin
  Flush, H['lun']
  H['bcount'] = H['bcount'] + 1L
  Print, 'wrote buffer to client ', H['bcount']
EndIf Else Begin
  If (TC ne Buffer.Length) then Begin
    Print, 'Only sent ' + TC.ToString()
  EndIf
EndElse
If (H['bcount'] eq 1000) then Begin
  ; Only reply to the first 1000 requests, then close down the socket.
  Free_LUN, H['lun'], /Force
  !null = Timer.Set(.1, 'ListenerCallback', H['listenerlun'])
  Print, 'Closed client socket, listening for new connection requests'
EndIf Else Begin
  !null = Timer.Set(.01, 'ClientCallback', H)
EndElse
End 

Pro ListenerCallback, ID, ListenerLUN
Compile_Opt IDL2
Status = File_Poll_Input(ListenerLUN, Timeout = .1)
If (Status) then Begin
  Print, 'Made a connection, starting client connection.'
  Socket, ClientLUN, Accept = ListenerLUN, /Get_LUN, /RawIO, $
    Connect_Timeout = 30., Read_Timeout = 30., Write_Timeout = 30.
  !null = Timer.Set(.01, 'ClientCallback', Hash('lun', ClientLUN, $
    'bcount', 0L, 'listenerlun', listenerlun))
EndIf Else Begin
  !null = Timer.Set(.1, 'ListenerCallback', ListenerLUN)
Endelse
End


Pro IDLServer
Compile_Opt IDL2
Port = (UInt(Byte('IDL85Rocks'), 0, 2))[1] 
Socket, ListenerLUN, Port, /Listen, /Get_LUN, $
    Read_Timeout = 60., Write_Timeout = 60., /RawIO
!null = Timer.Set (.1, 'ListenerCallback', ListenerLUN)
End

This is the client-side code:

Pro ServerCallback, ID, H
Compile_Opt IDL2
Catch, ErrorNumber
If (ErrorNumber ne 0) then Begin
    Catch, /Cancel
    Help, /Last_Message
    Return
EndIf
If (File_Poll_Input(H['lun'], Timeout = .01)) then Begin
  ; The protocol is simply to get 10,000 integers from the server
  ; with each "read".  The client doesn't send any data to the server.
  BigBuffer = UintArr(1.e5)
  Length = 0L
  CBuffer = BigBuffer
  Repeat Begin 
    ReadU, H['lun'], CBuffer, Transfer_Count = TC
    If (TC ne 0) then Begin
        BigBuffer[Length] = CBuffer[0:TC - 1]
        Length += TC
        If (Length lt BigBuffer.Length) then Begin
            CBuffer = UintArr(BigBuffer.Length - Length)
        EndIf
    EndIf
  EndRep Until Length eq BigBuffer.Length
  Print, 'Got buffer, total = ' + (Total(BigBuffer)).ToString()
  H['bcount']++
  If (H['bcount'] eq 1000) then begin
    ; Got all 1000 expected buffers of 10,000 integers so stop listening for data on the socket.
    Print, 'Received last buffer'
    Free_LUN, H['lun'], /Force
    Return
  EndIf
EndIf Else Begin
  Print, 'no data on socket'
EndElse
; Get the next buffer
!null = Timer.Set(.001, 'ServerCallback', H)
End


Pro IDLClient
Compile_Opt IDL2
Port = (UInt(Byte('IDL85Rocks'), 0, 2))[1]
Socket, ServerLUN, 'localhost', Port, /Get_LUN, Connect_Timeout = 10., $
  Read_Timeout = 10., Write_Timeout = 10., /RawIO
!null = Timer.Set (.001, 'ServerCallback', Hash('lun', ServerLUN, 'bcount',  0L))
End

Start the server as

IDL> idlserver

In the second IDL process, start the client as

IDL> idlclient

For further reading on the topic of exchanging object data between IDL sessions, including a reference example, see an earlier blog post.