Integrating ENVI with ArcToolbox and ModelBuilder

Integrating ENVI with ArcToolbox and ModelBuilder

This document introduces a novel approach to the integration of ENVI-based processing capabilities within ArcGIS's analytical framework. Integrating GIS and Remote Sensing data analysis tools using this approach provides for numerous end-user benefits including:

• Perform analysis on various geospatial data sources (both GIS and Remote Sensing) from within a common software product, minimizing software training requirements
• Incorporate ENVI data exploitation methods into visual models of analytical tradecraft interactively using ArcGIS's ModelBuilder tools, streamlining an analyst's workflow
• Merge Remote Sensing data and derived products with GIS layers to create GEOINT fusion products
• Leverage ArcGIS Server to deploy geospatial processing tools and models to a web environment, maximizing access to analytical methods and robust server-based processing resource

Introduction

Deriving informational products from geospatial data often requires performing both image processing tasks and GIS analysis utilizing separate software applications. Many times these software applications are run concurrently and intermediate products are manually transferred between applications until a finished product is created. This rather inefficient process is necessary since the user typically cannot create the desired output product(s) using a single software package -- the image processing and GIS software are used in tandem.

Over the past few years, NV5 Geospatial and Esri have been working towards improving the interoperability between their respective products: ENVI and ArcGIS. This whitepaper discusses a methodology developed jointly by NV5 Geospatial and Esri that enables ENVI based image and data processing functionality written in IDL to be called directly from ArcToolbox and ModelBuilder.

Users have expressed a keen interest in improving the interoperability between image processing packages like ENVI and GIS packages like ArcGIS. Improved interoperability accomplishes many tasks including:

• Increased productivity
• Reduced training and expertise requirements for multiple software packages
• Helps to justify the investments and costs in purchasing these products
• Reduction in costs since custom software tools are developed once and then leveraged from both applications

This is an exciting and powerful development because once ENVI based functionality is made available in ArcToolbox and ModelBuilder, the ArcGIS user can utilize these tools as if they were available natively. An increase in productivity can be achieved since the ArcGIS user can stay in a familiar environment and the output products generated by the ENVI routines are pushed directly into ArcMap and/or a geodatabase.

The ENVI based tools can be easily shared with other users in the same manner as Arc Toolbox tools and Model Builder models are currently shared. Although the ENVI based tools may appear to be native ArcToolbox tools an ENVI license is still required to run them.

This whitepaper provides an overview of the methodology for integrating ENVI capabilities with ArcGIS via ArcToolbox and Model Builder.

Methodolgy

The goal of this methodology is to provide access to specific ENVI routines from within the Arc Toolbox and ModelBuilder framework thus enabling the ArcGIS user to run these routines either individually or as part of a custom model created in ModelBuilder.

The first step in the methodology is to identify the specific ENVI routine that will be integrated. This routine can be an existing ENVI routine or a custom routine. Ideally, the ENVI routine should be one that requires no user interaction during execution. In the ENVI environment, this type of routine is one that can be run in batch mode. The next step is to create a tool in the ArcToolbox that gathers any inputs from the user, calls the ENVI routine and passes the required inputs to the routine. The following section will briefly describe components of this methodology.

ENVI routines suitable for integration with ArcToolBox

An ENVI routine suitable for integration with ArcToolbox can either be an existing ENVI routine or a custom routine developed by the user. This is typically done utilizing the Interactive Data Language (IDL), which is one of the core products available from NV5 Geospatial. In fact, most of the ENVI package is written using IDL. IDL is an interpreted language designed to facilitate processing, analysis, and visualization of all sorts of 2D imagery and 3D datasets.

Routines suitable for integration with ArcToolBox are those that can be run in ENVI batch mode, which means that during execution no user interaction is required. The required inputs to the routine are passed via parameter or keyword variables.

ENVI is essentially an IDL application and thus its routines are IDL functions or procedures. Therefore, the external batch scripts are really starting IDL and running an IDL routine. Our methodology for integrating ENVI routines with ArcToolbox leverages this ability to call IDL and ENVI from the command line.

Details of how to call IDL from the operating system command line can be found in the IDL help system.

ArcToolbox and ModelBuilder

ArcToolbox and ModelBuilder are components of the geoprocessing framework in ArcGIS. The purpose of the geoprocessing framework is to enable the user to automate processing tasks and to perform modeling and analysis. The geoprocessing framework provides the basis for spatial analysis and modeling through numerous tools and operators that work on geographic datasets.

ArcToolbox is a collection of GIS tools organized into a number of toolboxes and toolsets within each toolbox, thus providing organized access to all of the tools in ArcGIS and a mechanism for finding and executing the desired analytical tools. The typical tool takes ArcGIS datasets as inputs, such as feature classes, rasters, and tables, applies an operation on the data, and creates a new dataset. Figure 1 shows the ArcToolbox dialog containing the standard set of tools along with a custom ENVI Toolbox.

ArcGIS's ModelBuilder allows the user to graphically develop and execute geoprocessing models which are created by dragging and dropping tools from ArcToolbox into the ModelBuilder interface. ModelBuilder allows an analyst to visually model their tradecraft and share those models with other analysts. Tools made available within the ModelBuilder environment can be chained together using logical operators, such as arrows, that describe the pattern upon which analysis is performed. Figure 2 shows an example of a model that consists a several different tools available in ArcToolbox.

In the Modelbuilder example shown in Figure 2, data inputs are blue circles, processing tasks are yellow squares, and derived products (either final or intermediate) are green circles. Data flows from the upper left to the bottom right, and in this rather simple diagram, a total of 7 data processing tasks are applied and 7 different geospatial products are generated.

Functionality that is not available in ArcGIS can be added by developing scripts using the Python programming language. The new functionality then becomes a new tool available within ArcToolbox, which can be used just like any of the pre-defined tools including as a component within a model in ModelBuilder.

More detailed information on the ArcGIS geoprocessing framework can be found in the ArcGIS Desktop Help or on the ArcGIS website at https://www.esri.com.

Creating an ArcToolbox script that calls an ENVI routine

The Python programming language is utilized to create ArcToolbox scripts in the ArcGIS geoprocessing framework. Python is also an interpreted language and available in the open source (https://www.python.org). Programmatic access to the geoprocessing framework using Python is available from the ArcObjects component called the geoprocessor object. The geoprocessor object exposes the available geoprocessing functions to Python. The various geoprocessing functions enable the Python script to interact with the user, data, and the ArcGIS user interface.

Providing a detailed description of how to develop a Python script is beyond the scope of this whitepaper. For additional details refer to the ArcGIS Desktop Help system. The Python script will have several basic components:

• Include the required ArcGIS library functions
• Create an instance of the geoprocessor object
• Create a command string consisting of the path to the IDL executable, the name of the ENVI routine, and the required input arguments.
• Pass the command string to the operating system for execution.

The Python language provides a function for submitting and executing operating system commands. This is a key feature of our methodology because it provides the mechanism by which the Python script calls the ENVI routine.

Adding the Script to ArcToolbox

Once the Python script has been created, it can be added to ArcToolbox and thus making it available to the user just like any other tool. The basic steps for adding a script to ArcToolbox include:

• Naming the script
• Identifying the Python script that will be run
• Setting the script parameters that correspond to the parameters in the Python script

Figure 3 shows an example of parameters for a Python script that calls an ENVI routine for computing NDVI. The three parameters for this script are the name of the ENVI routine that will be called, the name of the input file, and the name of the output file.

After finishing this dialog, the script is added to ArcToolbox and is now available for use.

Running the Script

Using the NDVI example, when the user clicks on the NDVI script tool in ArcToolbox, the dialog shown in Figure 4 appears. The three parameters shown in the dialog are the same ones that were set above as shown in Figure 3. One of the parameters is the name of the ENVI routine to run. Our methodology includes this parameter to allow for flexibility when sharing the tool with other users since each user may install the tool in different locations.

After setting the parameters to valid values, the script is started when the user presses the OK button. A status dialog, similar to one shown in Figure 5, appears providing the user feedback regarding the progress of the script.

Note that while the script is running the user has no direct interaction with ENVI. The ENVI routine executes using the parameter information and then exits. The output of this particular example is shown in Figure 6.

One of the features of our methodology is to ensure that the output products are automatically loaded into ArcMap or geodatabase. Although this strategy is flexible and can be easily changed as desired, it allows the user to begin using the results immediately rather than having to manually load the file into ArcMap.

A ModelBuilder Example

Once a script is added to ArcToolbox it can be used just like any other tool, including as part of the model created using ModelBuilder. Figure 7 shows an example of a model that takes 6 input lidar files computes a digital elevation model (DEM) for each dataset, mosaics the 6 DEMs and then computes topographic features from the DEM. The final products are then loaded directly into ArcMap as shown in Figure 8.

The yellow boxes represent the scripts that will be run as part of this model. The blue boxes are the input parameters required by each script. The green boxes indicate the outputs.

This particular example illustrates the power and flexibility of this methodology. The model combines 3 different scripts, that each call a separate ENVI routine, into a single workflow that is run on multiple files and without user interaction. The user runs this entire workflow from the ArcGIS environment and obtains that the output products from the workflow directly into the open ArcMap session. In the background though, ArcGIS is leveraging functionality from ENVI that either doesn’t already exist in ArcGIS or is more efficiently performed using ENVI’s implementation.

Geospatial Analytical Methods on ArcGIS Server

ENVI tools embedded within the ArcGIS environment can be deployed in several different scenarios. They can be used individually or within a model from ArcGIS desktop or used to develop data processing services that can be published to ArcGIS® Server for use across a workgroup or enterprise. For ArcGIS® Server environments a new version of ENVI, ENVI for ArcGIS® Server, is also now available. This tool allows multiple, concurrent users of ENVI image processing tools from a web front-end. ENVI for ArcGIS® Server exposes ENVI capabilities natively within ArcGIS® Server allowing users to leverage server class resources to speed processing and analysis workflows on large data stores. Additionally, custom ENVI tools, integrated imagery and GIS processing models, developed using the paradigm described within this document, can be deployed to the ArcGIS Server environment for use within an enterprise system.

ENVI tools embedded within the ArcGIS Server environment provide for numerous benefits. First, Geospatial Analysts can gain access to powerful ENVI-based image processing tools through a web interface, cutting down on the training burden of learning new software applications. Since the ENVI's tools can be fully integrated in the ArcGIS environment (both desktop and server) ENVI's tools appear as additional ArcGIS Toolboxes. If an analyst knows how to use ArcGIS analytical tools, they already know how to take advantage of ENVI's embedded remote sensing analysis tools since the interfaces follow a standard paradigm. A second benefit of the ArcGIS Server-ENVI integration is the ability to rapidly support the data processing requests of multiple, concurrent, remote users by leveraging centralized server-based processing resources. System engineering focus can be allocated to developing high-performance server side processing hardware infrastructure, while remote users simply require access to the proper web portal. Raw data can sit on the server side and only those derived products, commonly a small fraction in size relative to the original data, are delivered to the end user's desktop. Finally, users do not require access to a desktop license of ENVI or ArcGIS to apply geospatial analytical tasks. License distribution and maintenance costs are minimized per user, while access to the analytical methods provided by these software tools can be maximized.

A few examples of ENVI-based processing tools deployed through an ArcGIS Server web interface can be accessed by navigating to the following url: envidemo.ittvis.com. See the screen capture below for an example of a webpage that can be used to process imagery data remotely using ENVI's tools deployed to ArcGIS® server.

Application of ENVI+ArcGIS integration to LiDAR related tasks

The integration of ENVI and ArcGIS processing concepts through the use of ArcToolbox and ArcGIS's model builder environment described in this paper has been generic to this point. The following chapter outlines how this approach can be applied specifically to accomplish LiDAR related processing goals in a more effective, efficient manner.

Many of NGA IB's processing tools to access, process and exploit point cloud lidar datasets are structured in a manner by which analysts must navigate GUI-based tools to perform their individual processing tasks. While numerous processing tasks often occur in succession, there is no capability currently available for an analyst to visually model the processes they follow to ingest data, perform data processing tasks and generate an output product. Processing steps are applied independently for each instance of a workflow.

Many of the tools used by NGA IB are developed within the ENVI/IDL framework, therefore making them prime candidates to integrate into the ArcGIS geoprocessing environment. By integrating into the geoprocessing environment, IB analysts may take advantage of the ModelBuilder tools and the ArcGIS Server options described in the previous sections. It should be noted that although the methods described in this paper outline ENVI/IDL tool integration with ArcGIS, it is possible to incorporate other processing tools outside of the ENVI/IDL framework with ArcGIS (i.e. compiled C code).

To describe how a common workflow may be enabled in ArcGIS, the following series of steps are used as an example.

1. 1. Access point cloud LiDAR data file

1. 2. Perform coincidence processing to remove anomalous data points

1. 3. Compute some basic raster-based products such as an elevation model or an intensity map

1. 4. Create some primitive-level products such as slope, aspect, hillshade maps, curvature

1. 5. Detect the presence of target features within a point cloud file (i.e. Helicopter Landing Zone)

1. 6. Save results off to central repository (database)

1. 7. View resultant data product(s)

Each of these steps require interaction with their own GUI to collect user input concerning target data file(s), user defined parameters and output file locations. Each one of these steps can be expressed as a GeoProcessing routine within ArcGIS, therefore each can be deployed to the ArcGIS ModelBuilder environment. Using ENVI tools contained within ArcGIS's geoprocessing environment, a visual model can be created to represent this workflow. Once this model is created and saved, a user simply has to declare the input dataset(s) upon which to pass into the model and the process chain previously declared is applied to the input datasets. Figure 9 show how these processing tasks can be structured within ArcGIS to accomplish an efficient data processing workflow.

The process flow progresses starting from the top left where the data inputs are declared. Upon creation of a model similar to what is seen in figure 9-1, the model can be deployed to end users in 2 different ways.

First, the model can be saved and shared with other desktop users that have ENVI and ArcGIS fully installed and licensed. The model will appear as an analytical method available within that user's ArcToolbox. Quite simply, the end user can double click on the model, pass in the location of one or more LAS files, and the processes outlined in the model will be applied, in succession, to those input files. The developer of the model has the option to expose as much (or as little) flexibility to the end user as they would like.

For example, the developer may only want the end user to have the ability to define input LAS files, and the output location (i.e. folder) where products are written. In this case, when a user launches the model, the only decisions they make are the input file names and output folder locations. Conversely, the developer may expose the ability to let the end-user define various parameters and input variables such as the type of digital elevation model algorithm to apply, the types of derivative products to create, and the helicopter type(s) that will be used in the HLZ calculation. The developer of the model has this flexibility and can expose model parameters to end users by simply selecting/deselcting a checkbox associated with each input model parameter.

In the same fashion this model can be expressed through a web portal using ENVI's integration with ArcGIS Server. Users can access data remotely, apply analytical tools using models defined by other developers and derive a vast number of geospatial data products. In this manner, the user no longer needs to have all of their resources local.

The analytical methods, data sources and software licenses can reside in a central location.

General Requirements

The methodology described in this whitepaper was developed using ENVI v4.7 and ArcGIS Desktop v9.3 on Windows XP. Python v2.5, which is shipped with ArcGIS, was used to develop the scripts.

Implementation

NV5 has experience in the development and deployment of custom remotely sensed data processing solutions based directly on customer requirements. This experience includes, but is not limited to, creation of unique data exploitation algorithms and techniques along with batch processing scripts. These development capabilities can be leveraged to implement the methodologies outlined in this document with specific, user declared exploitation routines enabled within the ArcGIS processing environment. Our developers can assist in identifying and implementing the proper processing tasks to effectively advance the analytical tools contained within ArcGIS.

Summary

This whitepaper describes an approach developed jointly by NV5 Geospatial and Esri that integrates ENVI capability into the ArcGIS environment via ArcToolbox and ModelBuilder. There are several advantages for utilizing this approach.

• This approach demonstrates true integration between ENVI and ArcGIS and provides access to ENVI’s powerful image and data processing capabilities to a class of ArcGIS users that may not have ever considered using ENVI in the past.
• Facilitates adding functionality available in ENVI that either is not available in ArcGIS or is not as efficient as the ENVI implementation.
• Once added to ArcToolbox, the scripts that call ENVI routines can be used just like any other tool in ArcToolbox, as well as, inclusion into custom models in ModelBuilder.
• The ENVI capability is accessed from within the ArcGIS environment and the output products are pushed directly into ArcMap and/or a geodatabase. This feature increases productivity since the user can access this capability from one software package environment.
• Users experience a greater return on their investment in both ENVI and ArcGIS since more sophisticated tools can be developed that leverage the strengths of each software package. In addition, the same functionality developed in ENVI can also be accessed from ArcGIS with minimal effort and can be easily shared with other users.
• The methodology is flexible and robust enough to rapidly integrate any existing or custom ENVI routines, especially those that can be run in ENVI batch mode.
• Capabilities outlined in this document can be deployed in both ArcGIS desktop and ArcGIS Server environments

Acknowledgments

NV5 would like to acknowledge the various employees that made significant contributions to the development of this methodology. In addition, we also acknowledge the contributions of Anthony Calamito and Dan Zimble of Esri for their contributions.

Points of Contact

To gather further information regarding this concept, please use the contact information below to discuss this capability further with NV5 Geospatial:

Kevin Lausten: 303-413-3925

Greg Terrie: 571-203-7388