The SHMMAP procedure maps anonymous shared memory, or local disk files, into the memory address space of the currently executing IDL process. Mapped memory segments are associated with an IDL array specified by the user as part of the call to SHMMAP. The type and dimensions of the specified array determine the length of the memory segment.
The array can be of any type except pointer, object reference, or string. (Structure types are allowed as long as they do not contain any pointers, object references, or strings.) By default, the array type is single-precision floating-point; other types can be chosen by specifying the appropriate keyword.
Once such a memory segment exists, it can be tied to an actual IDL variable using the SHMVAR function, or unmapped using SHMUNMAP.
Why Use Mapped Memory?
- Shared memory is often used for interprocess communication. Any process that has a shared memory segment mapped into its address space is able to "see" any changes made by any other process that has access to the same segment. Shared memory is the default for SHMMAP, unless the FILENAME keyword is specified.
- Memory-mapped files allow you to treat the contents of a local disk file as if it were simple memory. Reads and writes to such memory are automatically written to the file by the operating system using its standard virtual memory mechanisms, which has the potential to be faster than standard Input/Output using Read/Write system calls.
SHMMAP uses the following rules, in the specified order, to determine which method to use:
- If the FILENAME keyword is present, SHMMAP creates a memory-mapped file segment.
- If the SYSV keyword is used under UNIX, a System V shared memory segment is created or attached. Use of the SYSV keyword under Windows will cause an error to be issued.
- If none of the above options are specified, SHMMAP creates an anonymous shared memory segment. Under UNIX, this is done with Posix shared memory. Under Windows, the CreateFileMapping() system call is used.
Note: Unlike most IDL functionality, incorrect use of SHMMAP can corrupt or crash your IDL process. You should be familiar with the memory and file mapping features of your operating system and the terminology used to describe such features.
Example
See Additional Examples below for more information on using SHMMAP.
Create a shared memory segment of 1000000 double-precision data elements, and then fill it with a DINDGEN ramp:
SHMMAP, 'MYSEG', /DOUBLE, 1000000
z = SHMVAR('MYSEG')
z[0] = DINDGEN(1000000)
Note: When using shared memory, using the explicit subscript of the variable (z, in this case) maintains the variable’s connection with the shared memory segment. When not using shared memory, assignment without subscripting is more efficient and is recommended.
Syntax
SHMMAP [, SegmentName] [, D1, ..., D8] [, /BYTE] [, /COMPLEX] [, /DCOMPLEX] [, /DESTROY_SEGMENT] [, DIMENSION=value] [, /DOUBLE] [, FILENAME=value] [, /FLOAT] [, GET_NAME=value] [, GET_OS_HANDLE=value] [, /INTEGER] [, /L64] [, /LONG] [, OFFSET=value] [, OS_HANDLE=value] [, /PRIVATE] [, SIZE=value] [, /SYSV] [, TEMPLATE=value] [, TYPE=value] [, /UINT] [, /UL64] [, /ULONG]
Arguments
SegmentName
A scalar string supplying the name by which IDL will refer to the shared memory segment. The SegmentName string must be a valid IDL identifier (i.e. follow the same rules as an IDL variable name). This name is only used by IDL, and does not necessarily correspond to the name used for the shared memory segment by the underlying operating system (see the OS_HANDLE keyword for more details).
If SegmentName is not specified, IDL will generate a unique name. The generated SegmentName can be obtained using the GET_NAME keyword.
Di
The dimensions of the result. The Di arguments can be either a single array containing the dimensions or a sequence of scalar dimensions. Up to eight dimensions can be specified.
Keywords
BYTE
Set this keyword to specify that the memory segment should be treated as a byte array.
COMPLEX
Set this keyword to specify that the memory segment should be treated as a complex, single-precision floating-point array.
DCOMPLEX
Set this keyword to specify that the memory segment should be treated as a complex, double-precision floating-point array.
DESTROY_SEGMENT
The UNIX anonymous shared memory mechanisms (Posix shm_open() and System V shmget()) create shared memory segments that are not removed from the operating system kernel until explicitly destroyed (or the system is rebooted). At any time, a client program can attach to such an existing segment, read or write to it, and then detach.
For UNIX anonymous shared memory (Posix or System V), the default behavior is for IDL to destroy any shared memory segments it created when the segments are unmapped, and not to destroy segments it did not create. The DESTROY_SEGMENT keyword is used to override this default: set DESTROY_SEGMENT to 1 (one) to indicate that IDL should destroy the segment when it is unmapped, or 0 (zero) to indicate that it should not destroy it. This destruction occurs when the segment is unmapped via SHMUNMAP.
The DESTROY_SEGMENT keyword is ignored under the Windows operating system. Under UNIX, it is ignored for mapped files.
DIMENSION
Set this keyword equal to a vector of 1 to 8 elements specifying the dimensions of the result. Setting this keyword is equivalent to specifying an array via the D argument.
DOUBLE
Set this keyword to specify that the memory segment should be treated as a double-precision floating-point array.
FILENAME
By default, SHMMAP maps anonymous shared memory. Set the FILENAME keyword equal to a string containing the path name of an existing file to create a memory-mapped file segment. A shared mapped file can serve as shared memory between unrelated processes. The primary difference between anonymous shared memory and mapped files is that mapped files require a file of the specified size to exist in the filesystem, whereas anonymous shared memory has no user-visible representation in the filesystem.
By default, files are mapped as shared, meaning that all processes that map the file will see any changes made. All changes are written back to the file by the operating system and become permanent. You must have write access to the file in order to map it as shared.
To change the default behavior, set the PRIVATE keyword. When a file is mapped privately, changes made to the file are not written back to the file by the operating system, and are not visible to any other processes. You do not need write access to a file in order to map it privately — read access is sufficient.
Note: The non-private form of file mapping corresponds to the MAP_SHARED flag to the UNIX mmap() function, or the PAGE_READWRITE to the Windows CreateFileMapping() system call.
FLOAT
Set this keyword to specify that the memory segment should be treated as a single-precision floating-point array.
GET_NAME
If SegmentName is not specified, IDL automatically generates a name. Set this keyword equal to a named variable that will receive the name assigned by IDL to the memory segment.
GET_OS_HANDLE
Set this keyword equal to a named variable that will receive the operating system name (or handle) for the memory segment. The meaning of the operating system handle depends on both the operating system and the type of memory segment used. See the description of the OS_HANDLE keyword for details.
INTEGER
Set this keyword to specify that the memory segment should be treated as an integer array.
L64
Set this keyword to specify that the memory segment should be treated as a 64-bit integer array.
LONG
Set this keyword to specify that the memory segment should be treated as a longword integer array.
OFFSET
If present and non-zero, this keyword specifies an offset (in bytes) from the start of the shared memory segment or memory-mapped file that will be used as the base address for the IDL array associated with the memory segment.
Note: Most computer hardware is not able to access arbitrary data types at arbitrary memory addresses. The specific rules differ between machines, but in many cases the address of a data object must be evenly divisible by the size of that object. IDL will issue an error if you specify an offset that is not valid for the array specified.
Note: The actual memory-mapping primitives require offsets to be integer multiples of the virtual memory pagesize (typically a power of two such as 8K or 64K). IDL will automatically adjust the offset to point to the correct memory page, and then include an additional offset to point to the specified byte within that page. In summary, IDL may map slightly more memory than your request would seem to require, but never more than a single page.
OS_HANDLE
Set this keyword equal to the name (or handle) used by the underlying operating system for the memory segment. If you do not specify the OS_HANDLE keyword, SHMMAP will under some circumstances provide a default value. The specific meaning and syntax of the OS_HANDLE depends on both the operating system and the form of memory used. See the following sections for operating-system specific behavior, and Types Of Memory Segments for behavior differences based on the form of memory used.
Posix (UNIX) Shared Memory
Use the OS_HANDLE keyword to supply a string value containing the system global name of the shared memory segment. Such names are expected to start with a slash (/) character, and not to contain any other slash characters. You can think of this as mimicking the syntax for a file in the root directory of the system, although no such file is created. See your system documentation for the shm_open() system call for specific details. If you do not supply the OS_HANDLE keyword, SHMMAP will create one for you by prepending a slash character to the value given by the SegmentNameargument.
UNIX System V Shared Memory
Use the OS_HANDLE keyword to supply an integer value containing the system global identifier of an existing shared memory segment to attach to the process. If you do not supply the OS_HANDLE keyword, then SHMMAP creates a new memory segment. The identifier for this segment is available via the GET_OS_HANDLE keyword.
Windows Anonymous Shared Memory
Use the OS_HANDLE keyword to supply a global system name for the mapping object underlying the anonymous shared memory. If the OS_HANDLE keyword is not specified, SHMMAP uses the value of the SegmentName argument.
UNIX Memory-Mapped Files
The OS_HANDLE keyword has no meaning for UNIX memory-mapped files and is quietly ignored.
Windows Memory-Mapped Files
Use the OS_HANDLE keyword to supply a global system name for the mapping object underlying the mapped file. Use of the OS_HANDLE will ensure that every process accessing the shared file will see a coherent view of its contents, and is thus recommended for Windows memory-mapped files. If you do not supply the OS_HANDLE handle keyword for a memory-mapped file, no global name is passed to the Windows operating system, and a unique mapping object for the file will be created.
PRIVATE
Set this keyword to specify that a private file mapping is required. In a private file mapping, any changes written to the mapped memory are visible only to the process that makes them, and such changes are not written back to the file. This keyword is ignored unless the FILENAME keyword is also present.
Mapping a file as shared requires that you have write access to the file, but a private mapping requires only read access. Use PRIVATE to map files for which you do not have write access, or when you want to ensure that the original file will not be altered by your process.
Note: Under UNIX, the private form of file mapping corresponds to the MAP_PRIVATE flag to the mmap() system call. Under Windows, the non-private form corresponds to the PAGE_WRITECOPY option to the Windows CreateFileMapping() system call. When your process alters data within a page of privately mapped memory, the operating system performs a copy on writeoperation in which the contents of that page are copied to a new memory page visible only to your process. This private memory usually comes from anonymous swap space or the system pagefile. Hence, private mapped files require more system resources than shared mappings.
It is possible for some processes to use private mappings to a given file while others use a public mapping to the same file. In such cases, the private mappings will see changes made by the public processes up until the moment the private process itself makes a change to the page. The pagesize granularity and timing issues between such processes can make such scenarios very difficult to control. Avoid combining simultaneous shared and private mappings to the same file.
SIZE
Set this keyword equal to a size vector specifying the type and dimensions to be associated with the memory segment. The format of a size vector is given in the description of the SIZE function.
SYSV
Under UNIX, the default form of anonymous memory is Posix shared memory, (shm_open() and shm_unlink()). Specify the SYSV keyword to use System V shared memory (shmget(), shmctl(), and shmdt()) instead.
TEMPLATE
Set this keyword equal to an IDL variable that has the desired type and dimensions to be associated with the memory segment. The actual data contents of the variable are ignored.
TYPE
Set this keyword to specify the type code for the memory segment. See the description of the SIZE function for a list of IDL type codes.
UINT
Set this keyword to specify that the memory segment should be treated as an unsigned integer array.
ULONG
Set this keyword to specify that the memory segment should be treated as an unsigned longword integer array.
UL64
Set this keyword to specify that the memory segment should be treated as an unsigned 64-bit integer array.
Types Of Memory Segments
SHMMAP is a relatively direct interface to the shared memory and file mapping primitives provided by the underlying operating system.
UNIX
In modern UNIX systems, the mmap() system call forms the primary basis for both file mapping and anonymous shared memory.
UNIX Memory-Mapped Files
To memory map a file under UNIX, you open the file using the open() system call, and then map it using mmap(). Once the file is mapped, you can close the file, and the mapping remains in place until explicitly unmapped, or until the process exits or calls exec() to run a different program.
If more than one process maps a file at the same time using the MAP_SHARED flag to mmap(), then those processes will be able to see each others’ changes. Hence, memory-mapped files are one form of shared memory.
Posix Shared Memory
Posix shared memory uses the shm_open() and ftruncate() system calls to create a memory segment that can be accessed via a file descriptor. This descriptor is then used with the mmap() system call to map the memory segment. In this case no scratch file is required (the disk space comes from the system swap space). Posix shared memory segments exist in the operating system until explicitly destroyed (using the shm_unlink() system call). Posix shared memory allows the caller to supply the name of the segment, which makes it easy for multiple processes to map the same segment.
Posix shared memory is the default for SHMMAP on all UNIX platforms.
UNIX System V Shared Memory
For System V IPC shared memory, the shmget() function is used to create a shared memory segment. The operating system assigns each such segment a unique integer ID when it is created. Once a shared memory segment exists, the shmdt() function can be used to map it into the address space of any process that knows the identifier. This segment persists in the OS kernel until it is explicitly destroyed via the shmctl() function, or until the system is rebooted. This is true even if there are no processes currently mapped to the segment. Note that this can create a problem since shared memory segments that are not explicitly destroyed can cause memory leaks in the operating system. Hence, it is important to properly destroy such segments when they are no longer required.
Although System V shared memory is available on all UNIX platforms, there are some limitations on its utility:
- Many systems place extremely small limits on the size allowed for such memory segments. These limits are often kernel parameters that can only be adjusted by the system administrator.
- The caller does not have the option of naming the shared memory segment. Instead, the operating system assigns an arbitrary number, which means that processes that want to map such a segment have to use some form of interprocess communication to find the correct identifier.
Unless you have a specific requirement to use System V shared memory, we recommend that you use the default Posix shared memory.
Microsoft Windows
Under Microsoft Windows, the CreateFileMapping() system call is used for shared memory as well as memory-mapped files. To map a file, IDL opens the file and then passes the handle for that file to CreateFileMapping(). To create a region of anonymous mapped memory, IDL passes a special file handle (0xffffffff) to CreateFileMapping(). In this case, the disk space used to back the shared memory is taken from the system pagefile. CreateFileMapping() accepts an optional parameter (lpname), which if present, is used to give the resulting memory mapping object a system global name. If you specify such a name (via the OS_HANDLE keyword), and a mapping object with that name already exists, you will receive a handle to the existing mapping object. Otherwise, CreateFileMapping() creates a new mapping object for the file. Hence, to create anonymous (no file) shared memory between unrelated processes, IDL calls CreateFileMapping() with the special 0xffffffff file handle, and specifies a global name for it.
Note: To get coherent access to a memory-mapped file, every process should specify the same OS_HANDLE value to ensure that they use the same mapping object. With coherent access, all processes see exactly the same memory at exactly the same time because they are all mapping the same physical page of memory. If you are using read-only access to a mapped file, you need not be concerned with coherent access.
Note: Under Windows, when attaching to an existing memory object by providing the global segment name, IDL is not able to verify that the memory segment returned by the operating system is large enough to satisfy the IDL array specified to SHMMAP for its type and size. If the segment is not large enough, the IDL program will crash when it attempts to access memory addresses beyond the end of the segment. Hence, the IDL user must ensure that such pre-existing memory segments are long enough for the specified IDL array.
The Windows operating system automatically destroys a mapping object when the last process with an open handle to it closes that handle. To continue to use the shared memory, be sure to keep at least one process open that is using the shared memory segment.
Reference Counts And Memory Segment Lifecycle
You can see a list of all current IDL memory segments using:
HELP,/SHARED_MEMORY
To access a current segment, it must be tied to an IDL variable using the SHMVAR function. IDL maintains a reference count of the number of variables currently accessing each memory segment, and does not allow a memory segment to be removed from the IDL process as long as variables that reference it still exist.
SHMMAP will not allow you to create a new memory segment with the same SegmentNameas an existing segment. One way to ensure that segment names are unique is to not provide the SegmentName argument when calling SHMMAP. In this case, SHMMAP will automatically choose a unique name, which can be obtained using the GET_NAME keyword.
The SHMUNMAP procedure is used to remove a memory segment from the IDL session, and, depending upon the segment, it may remove the memory segment from the system. If no variables from the current IDL session are accessing the segment, the segment is removed immediately. If variables in the current IDL session are still referencing the segment, the segment is marked for removal when the last such variable drops its reference. Once SHMUNMAP is called on a memory segment, no additional calls to SHMVAR are allowed for it within the current IDL session; this means that a segment marked by SHMUNMAP as UnmapPending cannot be used for new variables within the current IDL session.
Note: IDL has no way to determine whether a process other than itself is accessing a shared memory segment. As a result, it is possible for IDL to destroy a memory segment that is in use by another process. The specific details depend on the type of memory segment, and the options used with SHMMAP when the segment was loaded. When creating applications that use shared memory, you should ensure that all applications that use the segment communicate regarding their use of the shared memory and act in a manner that avoids this pitfall.
Additional Examples
Anonymous Shared Memory
Create a shared memory segment and let IDL choose the segment name:
SHMMAP, /DOUBLE, DIMENSION=[1000000], GET_NAME=segname
z = SHMVAR(segname)
z[0] = DINDGEN(1000000)
HELP,/SHARED_MEMORY
IDL prints:
IDL_SHM_10428_0 DOUBLE = <WindowsAnonymous(IDL_SHM_10428_0), Offset(0), Refcnt(1)> Array[1000000]
Later, when we are finished with the segment, we unmap it and delete the variable. Notice that IDL does not free the segment until the last variable has been deleted:
SHMUNMAP, segname
HELP,/SHARED_MEMORY
IDL prints:
IDL_SHM_10428_0 DOUBLE = <WindowsAnonymous(IDL_SHM_10428_0), Offset(0), UnmapPending, Refcnt(1)> Array[1000000]
Now we delete the last reference:
z = 0
HELP,/SHARED_MEMORY
IDL prints nothing.
Memory-Mapped File
Create the same shared memory segment as the previous example, but use a temporary file instead of anonymous shared memory. Note that for efficiency we use POINT_LUN and just write out a single byte to ensure that the file is the correct length and is initialized to all zeroes:
filename = FILEPATH('idl_scratch', /TMP)
OPENW, unit, filename, /GET_LUN
POINT_LUN, unit, 1000000 - 1
WRITEU, unit, 0b
FREE_LUN, unit
SHMMAP, /DOUBLE, DIMENSION=[1000000], GET_NAME=segname, $
FILENAME=filename, OS_HANDLE='idl_scratch'
z = SHMVAR(segname)
Later, when we are finished with the segment, we release it and delete the file:
z = 0
SHMUNMAP, segname
FILE_DELETE,filename
Version History
|
5.6 |
Introduced |
|
6.0 |
Added FILENAME and PRIVATE keywords
|
See Also
SHMDEBUG, SHMUNMAP, SHMVAR