ENVI Catalog Background and Troubleshooting

This topic describes the scanning and cataloging process used in ENVI Catalog. Included are recommendations for best results and troubleshooting tips to diagnose problems. For full details on how to use ENVI Catalog, see the ENVI Catalog topic.

The Internal Catalog and the ability to browse it is included with the ENVI installation for collections of up to 250 images. The ability to create custom catalogs, scan directories and populate catalogs, and merge and prune catalogs requires a separate ENVI Catalog license; contact your sales representative for more information.

How ENVI Catalog Works

File cataloging is performed in the following two phases:

Scan directories to find files.
Catalog the found files by opening and analyzing them.

Understanding the scanning and cataloging process will help you to plan jobs, monitor progress, and recover from interruptions.

Scanning Phase

Scanning searches directories to find remote sensing images based on file extension. Files that meet the scan criteria are added to a candidate database.

ENVI Catalog uses the following tools to scan folders and build a database of candidate files:

On Windows: PowerShell
On Linux and macOS: the find utility

Both tools are optimized for large directory trees and can handle hundreds of thousands of files quickly.

The scan results in an SQLite candidate database file (typically named candidate_xxxxxx.sqlite) that contains one row per discovered file. Each row contains the file URI, its processing status, how many times it attempted to add the file to the database, and which ENVI Server worker is handling it. The candidate database is a work queue.

Cataloging Phase

Cataloging is managed by ENVI Server, allowing processing to run unattended, out-of-process, and in parallel. ENVI Catalog starts an ENVI Server instance on your local machine on port 50500 and submits one job per worker. For each job, ENVI Catalog does the following:

Checks out one file from the candidate database (no two workers process the same file).
Immediately submits the next job to ENVI Server, to keep the pipeline full.
Attempts to open the checked-out file as an ENVI Raster.
If the file opens successfully, reads its metadata (spatial reference, bands, dimensions, acquisition time, etc.), creates a thumbnail, and writes a row to the catalog database.
Returns, at which point ENVI Server cleans up the session. The next job that was submitted in step 2 is already running in parallel, so workers will stay busy as long as there are files in the queue. The number of jobs running at any one time is bounded by the number of workers you specify when you set up the cataloging job.

Files that open successfully are recorded with a "success" status. Files that cannot be opened by ENVI are recorded with a status of "error" or "skip" and are not retried. If a worker process crashes unexpectedly during step 3, the file is automatically re-queued by the Scanner Monitor (see the next section) and is given one more attempt for success, or it is retired with a "give-up" status.

Monitoring a Scan

When scanning and cataloging is in progress, a Scanner Monitor appears in the ENVI Catalog browser. The monitor polls the candidate database every few seconds and shows live counts (such as the number of files pending, in progress, successful) and basic database performance metrics under multi-worker loads.

The Scanner Monitor also performs background maintenance every 60 seconds. It checks whether any worker processes unexpectedly disappeared unexpectedly, such as if ENVI Server was terminated or the machine ran out of memory. If a dead worker is detected, the file that was checked out by the worker is released back to the queue so it can be picked up by another worker. If a file causes a worker to crash twice, it is retired with a "give-up" status rather than continuing to loop.

When all workers are finished processing jobs, the Scanner Monitor shows a completion summary.

Checking the File Status

You can inspect the candidate database at any time to look further into the status of the files that were processed or to do troubleshooting. See Candidate Database Browser.

Processing Files by Priority Tiers

ENVI Catalog groups candidate files into priority tiers based on their extension. Instead of processing the files in the order in which they were found, they are processed in their priority order. Lower-priority tiers are processed after the workers have exhausted the higher-priority queue.

The priority order reflects how remote sensing datasets are typically packaged. Most sensor products include companion metadata or a header file alongside the image data. For example, an .xml sidecar for WorldView and GeoEye products, an .hdr ENVI header for ENVI-native rasters, an .ads, or .dim file for SPOT and DIMAP datasets, and so on. ENVI uses those companion files as the entry point for opening the full raster correctly, with all bands, projection, and acquisition metadata in place.

Companion and metadata formats are processed early and raw image files are processed last, after their companion files. When a companion file is processed first and the raster is successfully cataloged through it, the raw image files (.tif, .img, and others) that belong to that dataset will be skipped because the raster is already cataloged.

Stopping a Cataloging Process

The recommended method to terminate the cataloging process is to click the Cancel button at the top-right of the Scanner Monitor.

If you closed ENVI without canceling the cataloging process and want to stop ENVI Server manually, use the steps below. Because the workers run inside ENVI Server (a separate process from the main ENVI window), closing ENVI does not stop the process. To stop cataloging immediately, you must terminate the ENVI Server process directly.

On Windows:

Open the Task Manager with Ctrl+Shift+Esc and select the Details icon.
Look for a process named python > envi_server.
Right-click the process and select End task.

On Linux or macOS:

Run: fuser -k 50500/tcp

-or-

Find the PID with: ps aux | grep envi_server
Then run: kill <PID>

After terminating the server, any files in the candidate database that were in "opening-raster" status remain in that state. When you restart the cataloging process, a recovery pass will automatically reset those rows to "pending" and they will be processed again. See the next section for restart steps.

Restarting a Cataloging Process

Any files awaiting processing when cataloging stopped will be processed after the restart.

Open the ENVI Catalog browser and select Edit > Candidates to Catalog (or the equivalent menu item for your workflow).
Set the Candidate Database parameter to the .sqlite candidate file that was in use before the cataloging process was terminated.
Click OK.

The catalog database will be updated with any newly processed files along with the files processed in the previous run.

Tips:

If the previous run was terminated abruptly (power loss, forced quit, ENVI crash), some files may still have a status of "opening-raster" from the previous session. These rows have stale worker process IDs (PIDs) that no longer correspond to any running process. When you restart the job, the first worker to find an empty queue will run the dead-worker recovery pass and automatically reset those rows to "pending." You do not need to manually edit the database.
Files with a status of "give-up" crashed or failed twice after two tries and will not be retried when you restart the process.
You can optionally start fresh by re-scanning the folder and rebuilding the candidate database. Start the scan and catalog process without specifying an existing candidate database; this will create a new candidate file.

Best Practices

This section provides recommendations for setting parameters for the scanning and cataloging process.

Determining the Number of Workers to Use

If your computer has 16 GB RAM, set the Workers parameter to 8 and monitor the computer's memory usage during the first few minutes. During the cataloging process, it is standard for CPU use to reach 100% during the cataloging process. This is expected behavior and indicates that the system is fully leveraging processing power.

Memory (RAM) use, however, is a more critical metric to monitor:

RAM consumption fluctuates based on raster size and format. Different imagery types encountered during scanning will cause varying memory loads.
Avoid allowing memory consumption to approach 100%, as this can severely degrade system stability and performance.
If monitoring indicates that memory consumption consistently remains well below capacity, you can optimize throughput by increasing the number of workers for future scans. If you change the number of workers, restart ENVI Server for the change to take effect.

If you need to catalog a large dataset faster than a single machine allows, you can do the following:

Use more workers on a single machine with sufficient RAM. ENVI Server scales well up to the point where memory or I/O becomes the bottleneck.
Split the source folders between multiple machines, each running its own independent Scan and Catalog job, then use Merge Catalogs to create a single final catalog.

Choosing Extensions to Scan

For faster processing and to reduce the size of candidate database, shorten the list of file extensions to scan for in the Extensions parameter. If you know what format(s) your directory structure contains, you can limit your search to include extensions for only those formats, rather than scanning all the extensions known to ENVI (the default).

Excluding Non-Imagery Folders

When setting up a scan, use the Exclude Pattern parameter to specify directories that do not contain imagery.

Excluding these folders upfront keeps the candidate database smaller and avoids spending worker time on files that will only produce skips or errors.

Cataloging Files on Shared Network Drives

For optimal performance and reliability, keep both the candidate and database files for the catalog on your local drive.

Using a UNC or network drive is not recommended because network filesystems are less reliable at maintaining the database locks required between multiple workers. Using network shares (like SMB or NFS) risks running into latency, lock contention errors, or even data corruption due to incompatible file-locking semantics. Storing these files locally avoids these concurrency issues.

Keeping the ENVI Catalog Browser Open While Cataloging

The ENVI Catalog browser hosts the Scanner Monitor, which performs the 60-second dead-worker recovery check.

Closing ENVI While the Cataloging Process is Running

ENVI Server is started as a persistent, independent process. It continues to run even after the main ENVI window is closed and your cataloging job will keep running. You can reopen ENVI, open the Catalog Browser, and the Scanner Monitor will reconnect to the in-progress job.

if you close the Catalog Browser window within ENVI, the scanner monitor will also close. This stops the 60-second dead-worker recovery check, but cataloging itself continues in ENVI Server. Reopen the ENVI Catalog browser to see progress again.

Adding Additional Files to a Catalog

If you add new files to a folder you have already scanned, you do not need to rescan the whole folder. You can re-run a targeted scan on the new subfolder or set of extensions and specify the same candidate output file as the existing candidate database as long as the catalog .sqlite file is accessible.

New files will be appended and existing files will not be processed again.

Updating Catalogs on a Schedule

If new imagery is added to your folders on a frequent basis, you can use the ENVI Modeler to create a job that will look for and add new rasters at a scheduled interval. In the ENVI Modeler, chain these three tasks together:

ScanFilesForCatalog scans the target folder and writes to a candidate database. For the directory input, specify the folder(s) or subfolder(s) where the new imagery will arrive. For the candidate database output, you can reuse the same candidate database for each run. New files will be appended to the candidate database and any files already in the database marked with "success" or "skip" will be skipped.
PopulateCatalog opens each pending candidate file and adds it to the catalog. Point the candidate database input to the candidate file from the ScanFilesForCatalog task and for the catalog database output, specify the catalog you want to add the files to. New files will be added to the catalog database and any files already in the catalog will be skipped.
PruneCatalog (optional) checks the catalog database for entries whose files no longer exist at their recorded URI and removes them. Include this step if files in your archive are ever deleted, renamed, or moved, so the catalog stays clean. Be sure the input point to the same catalog database that was specified in PopulateCatalog.

Connect the outputs of each task to the inputs of the next, save the model, and run it once manually to confirm it works as expected.

To run the model on a schedule, open it in the ENVI Modeler and select Run > Run Model on a Schedule. You can configure it to run daily, weekly, monthly, or at any custom interval. ENVI will launch the model automatically at the scheduled time, even if the ENVI application is not running. The results will be waiting for you the next time you open ENVI Catalog.

See Run Models on a Schedule for additional details.

Troubleshooting

The Scanner Monitor does not show progress after several minutes have elapsed:

One of the following could be the cause:

ENVI Server did not start. Ensure that port 50500 is not already in use by another application, then restart ENVI and try again.
All files are being skipped immediately (they are already in the catalog, the wrong extension, or not georeferenced). Open the Candidate Database Browser and filter the records by status to see what is happening.
The first raster files encountered take a very long time to open (for example, large HDF5 or radar images). The monitor will show 0 progress until the first raster successfully completes.

The Scanner Monitor shows a scan found significantly more files than it cataloged :

If the Scanner Monitor shows that a scan found significantly more files than it cataloged - such as 50,000 files were found but only 12,000 were added - open the Candidate Database Browser to view the status breakdown. The most common reasons files were not added to the catalog are:

skip: ENVI opened the file but it was one of the following: already in the catalog, not georeferenced (if you required georeferencing), or was recognized as a component of a multi-file raster that was already cataloged under its parent.
error: ENVI could not open the file. The error column describes why. Companion files (.hdr, .aux.xml, etc.) frequently appear here if they were included in the scan.
give-up: Two attempts were made to catalog the file and failed. This is usually due to a memory or crash issue.

Product	ENVI

Version	6.3