Which upgrade? What version of ENVI worked and which one is different?

I am now using IDL 8.7.1/ ENVI5.5.1

I wrote the code in IDL 8.6/ENVI 5.4

I ran the code in IDL 8.7.1/ENVI5.5.1 and it worked for a few runs, but then it stopped working if I had an image with more than 4 bands. My previous version of the code did a linear stretch over the whole raster prior to displaying and saving individual band geotiffs. However, when my raster had more than 4 bands, with a wider range of values, the linear stretch failed. I want the linear stretch to be applied on a band by band basis, so I modified the code that worked on IDL 8.6/ENVI 5.4 and now neither the old code I wrote, nor the newer code, works in IDL 8.7.1/ENVI 5.5.1

What appears to be happening is that the ENVILinearPercentStretchRaster (API) AND the ENVITask LinearPercentStretchRaster are not excluding the pixels with DIV values from the histogram, and so the 2% stretch is being performed on all pixels, instead of the just ones with actual value. And then the ENVITask ColorSliceClassification, which I am using to turn a grayscale image to a rainbow image, assigns the lowest color (purple) to both my no-data pixels and the lowest slice value pixels. I have tried updating the DIV in the metadata, to no avail. I would like to be able to pass the raster through these commands, and not have to get the data from the raster each time, to manually change the values in the pixels that should be flagged no-data.

Am I missing a flag in the commands? Or another step that should occur?

Thank you!!

Do you see the data ignore value set in the header after running the StretchRaster task? I also don't see the percent is set in the task, for example:
Task.PERCENT = [5.0]

If the DIV is set in the header after running StretchRaster, then I believe it is being used and passed on during the stretch.

Hi,

Thanks for finding the stretch percent error. I added the 2% line. My entire code looks like this:

print, 'Choose the ORIGINAL IMAGE file: '
file = dialog_pickfile(filter = '*.hdr')
raster = e.openraster(file)

print, 'Choose the VPCAscores file: '
file = dialog_pickfile(filter = '*VPCA*.hdr')
vpca = e.openraster(file)
;spatialref1 = vpca.spatialref
;v = vpca.getdata()
;v[where(v eq -9999, /NULL)] = !values.f_nan
;vraster = enviraster(v, spatialref= spatialref1)
;vraster.save

print, 'Choose the VPCAscores_FLIP file: '
file = dialog_pickfile(filter = '*VPCA_FLIP*.hdr')
vpcaflip = e.openraster(file)
spatialref1 = vpcaflip.spatialref
vf = vpcaflip.getdata()
vf[where(vf eq -9999, /NULL)] = !values.f_nan
vfraster = enviraster(vf, spatialref= spatialref1)
vfraster.save

bands = vpca.nbands

b = bands-1

for n = 0,b do begin
Subset = ENVISubsetRaster(vpca, bands=n)

Task = ENVITask('LinearPercentStretchRaster')
task.input_raster=subset
Task.OUTPUT_RASTER_URI = e.GetTemporaryFilename()
task.percent = [2.0]
task.execute
vpcastretch = task.output_raster

;vpcas = ENVILinearPercentStretchRaster(subset, percent=2.0)
metadata = vpcastretch.metadata
metadata.updateitem, 'data ignore value', [241]

Task = ENVITask('ColorSliceClassification')
task.input_raster=vpcastretch
Task.OUTPUT_RASTER_URI = e.GetTemporaryFilename()
task.execute

vpcascore = task.output_raster

Y = ''
read, Y, prompt='ENTER FILENAME FOR VPCAscore TIFF - I.E. DATE_INSTRUMENT_#VPCA#: ' ;NAME THE TIFF VPCA SCORE FILE
vpcascore.export, Y, 'tiff'


Subset = ENVISubsetRaster(vfraster, bands=n)

vpcas = ENVILinearPercentStretchRaster(subset, percent=2.0)
metadata = vpcas.metadata
metadata.additem, 'data ignore value', [0]

Task = ENVITask('ColorSliceClassification')
task.input_raster=vpcas
Task.OUTPUT_RASTER_URI = e.GetTemporaryFilename()
task.execute
vpcascoreflip = task.output_raster

Y = ''
read, Y, prompt='ENTER FILENAME FOR VPCAscore_FLIP TIFF - I.E. DATE_INSTRUMENT_#VPCA#: ' ;NAME THE TIFF VPCA SCORE FILE
vpcascoreflip.export, Y, 'tiff'

endfor

view = e.getview()
rgb = view.createlayer(raster, bands = [7,5,3], /clear_display)

view.zoom, /full_extent

Y = ''
read, Y, prompt='ENTER FILENAME FOR RASTER RGB TIFF - I.E. DATE_INSTRUMENT_RGB: ' ;NAME THE TIFF RGB FILE

view.chiptofile, Y, 'tiff'

end

The output tiff files that I used to get had black for no-data pixels, and then all the pixels with a valid value were colored. Now I get either a black and purple image, an image that is entirely purple, or entirely red. I think that the color slice classification task is not paying attention to the pixel state flag. I also think that the stretch raster task (or API, the result is the same) is not actually stretching my values over a histogram.

Is there a way I can send you my output images?

Thanks!

OK. I figured out what the issue is with that dataset - the range of pixel values is really big, so the histogram is wonky. I can deal with that. However, when I use the code with a different dataset, the second part works (where I replace the no-data value of -9999 with an NaN value after I pull in the raster, and then use the ENVI API LinearPercentStretch) and gives me an output tif that is almost right. However, I think the colorsliceclassificationtask is grouping pixels with no-data and the highest value pixels into one color bin, so pixels that should have a value have the same color as the no-data ones. Is there a way to tell the colorsliceclassificationtask to set pixels with a pixel state of DIV to a specific color? When I display the raster in the ENVI GUI, I do not have this pixel drop-out problem

Any values set to DIV (data ignore value) will be transparent, so they have no color assignment at all. Below is a simple test I did with our sample dataset, qb_boulder_msi. I rotated the data so there is background which is set to DIV=0 in the header. When I use Color Slice task on this on this, the result sets the DIV of 0 to transparent in the result so those areas are not in the slice or colored at all.

ENVI> file='C:\TEMP\qb_boulder_rotate.dat'
ENVI> Raster = e.OpenRaster(File)
ENVI> Task = ENVITask('LinearPercentStretchRaster')
ENVI> Task.INPUT_RASTER = Raster
ENVI> Task.PERCENT = [5.0]
ENVI> Task.Execute
ENVI> DataColl.Add, Task.OUTPUT_RASTER
ENVI> subset=ENVISubsetRaster(task.output_raster, bands=0)
ENVI> ClassTask = ENVITask('ColorSliceClassification')
ENVI> ClassTask.INPUT_RASTER = subset
ENVI> classtask.execute
ENVI> DataColl.Add, classTask.OUTPUT_RASTER

Can you check the stretch raster header or metadata to see if the DIV is set? Is it also set in the output from the Color Slices?
The only place I set you set the data ignore is after the stretch in the updateitem:
metadata.updateitem, 'data ignore value', [241]

Is the background a value of 241? Is it unique from the rest of the data? Setting data values directly to NaN is not the same as setting the DIV.

You mention setting NaN as the DIV. I don't believe this is supported and I also don't believe the TIFF export in this way supports a data ignore value or a data ignore value set to Nan. So lets see if we can get this working as expected just in an ENVI format file.

The DIV is set in the metadata at all steps, and the Pixel State is correct for the data / no-data pixels that I randomly selected to check. The reason I set the DIV to 241 is because that is the actual pixel value in the pixels that should be set to no-data ( and where the pixel state is also no-data). It is NOT a unique value, that is the crux of the issue. The pixels that are flagged no-data are being assigned a pixel value that is non-unique.

It is set in the output from the color slice, but the DIV is different than what I set as the DIV after the raster stretch.

The other question is: does the linear stretch stretch each band on that band's histogram? Or does it compute the histogram for all bands together, and stretches each band based on the full-raster histogram? I want each band stretched individually, which is why that step is inside my for loop.

The stretch is applied to each band histogram.
It is hard to know what is going on with your data and DIV. I am not sure how pixels that are flagged as no-data are 'being assigned' to a pixel value that is non-unique, as they would be assigned to a value you set as the DIV - assuming you are doing the flagging. Can you run this workflow through the GUI and get the expected outcome?.
You might try going through tech support so we can see your data.

OK. cool - the stretch being applied to each band can change my code so do the stretch once on the whole raster, and then just change the color on a band by band basis. I will go back to that process & see what is going on.

When I do this in the GUI it works perfectly. I display each band as greyscale with the default 2% stretch. I then change the color table to rainbow. The pixels with values are colored, the no-value pixels are white on the screen. I then export to a TIFF.

I want a code, though, because doing that for each image by hand is time consuming. The difference in my tiff export is that the no-value pixels are black, not white, and there are pixels (that have a value) that are colored the same color as the no-value pixels. Sometimes that it only a few pixels, total, but sometimes it is a more significant number that are discolored & lost.

Thank you for your help! I will start a ticket in the tech support after I update the code so the stretch is performed first.

Cheers,
Dulci

When exporting to TIFF, be sure to include the DATA_IGNORE_VALUE keyword.
subraster.Export, newFile, 'TIFF', data_ignore_value=0