| 17 Aug 2010 12:24 PM |
|
|
Hi All,
I've been using ENVI for a few weeks now and I have had nothing but problems with an annoying 'bug' in the program. I'm not sure if it's appropriate to report it as a bug though since it's possible that I'm just missing something. I have ASTER L1B data in .hdf format from NASA that I've opened in ENVI and processed in a number of ways. It seems that nearly everytime I open ENVI to view or process my data the corrdinates in the header files are different. In some cases, after a processing chain the images line up properly using the mosaic widget (which I've been using just to make sure that the images are where they ought to be at least with respect to one another) and then, after the same processing chain, some of the files end up the completely wrong position. None of my processing has involved changing the coordinates of the images beyond resmapling when I attempt to stack them. Here's the processing chain in outline (completed with 3 data granules, two of which abut and one that overlaps the other two):
1. convert L1B bands into TOA reflectance using bandmath (since there doesn't appear to be a built-in ENVI function for doing this)
2. use the perprocessing toolset to convert L1B TIR data into at sensor temperature (ASTemp) estimates
3. stack the TOA VNIR, TOA SWIR, and ASTemp TIR bands for each granule
4. load the stacks into the mosaic widget to find that nothing is in the right place (or alternatively, depending on the alignment of the stars I suppose, find that two of the three are in the correct posision while the other is not)
I'm working in Ubuntu 10,04 LTS on a Dell Precision T7500 with three monitors and ENVI 4.7. Has anyone else experienced buggy behaviour with ENVI regarding coordinates? It also seems that after the processing chain is complete, occasionally the header file(s) have been corrupted somehow and in the 'Avilable Bands List' the band labels and map info are incorrect. For example, the TIR band stack should look something like this:
Stack Name
-ASTER TIR Band10 (wavelength)
-ASTER TIR Band11 (wl)
-ASTER TIR Band12 (wl)
-ASTER TIR Band13 (wl)
-ASTER TIR Band14 (wl)
-mapinfo
However, sometimes it looks like this:
Stack Name
-ASTER TIR Band1 (wavelength)
-ASTER TIR Band2 (wl)
-ASTER TIR Band3N (wl)
-ASTER TIR Band3N (wl)
-ASTER TIR Band3N (wl)
-mapinfo
This happens occasionallly with and without processing. I've tried restarting the processing chaing several times from the raw data to compensate for any possible corruption along the way, but I have yet to succeed completely. Currently, I've managed to get all three granules processed and stacked so that the header information about coordinates appears correct and the lists are displayed properly, but in the mosaic widget all of the granules are presented as if they completely overlap oneanother (which makes no sense at all given the UL coordinates for each granule in the headers, which is why I'm thinking that there might be some kind of bug). I do ensure that I use the mosaic widget with the georeference option and not the pixel option. Trouble like this with the coordinates of images is making it impossible to get my work done. Any ideas, suggestions, comments, similar experiences? Thanks,
Chris
Oh I should also note that I've succesfully opened the .hdf files in ArcGIS (to confirm the coordinates) and the images are properly georeferenced - they load into ArcMap without a problem and appear in the correct places.
|
|
|
|
MariM
Veteran Member
Posts:2396  
| 17 Aug 2010 01:04 PM |
|
|
I can't say I have run into similar problems using ASTER 1B data in ENVI. Your processing sounds fairly complicated. If you believe it to be a bug, it would be best to contact technical support so they can work with you directly with your datasets.
That said, after your preprocessing and prior to stacking, does the Map Info look correct for all files? Does it change after stacking? The mosaic tool will align the images according to the map info - it is here that I think things are changing and it would be good to find the point where the change happens.
|
|
|
|
Deleted User
New Member
Posts: 
| 17 Aug 2010 02:03 PM |
|
|
Thanks for the reply. I've just gone through it all again and I think the coordinates get muddled in the stacking process. I tried to stack ASTER bands 1-4 and 13 (b/c I'm using them to create a cloud mask) and the coordinates are as follows:
pre-stack
89 18'53.44"W, 17 32'13.07N
post-stack
98 35'44.58"W, 17 12'41.65"N
The coordinates for each group of bands are slightly different in the raw data, which I assume is b/c the different bands are at different resolutions (VNIR 15m, SWIR 30m, TIR 90m) and that doesn't seem like it should really matter for the stacking. At least I can't see how coordinates that differ between bands by less than a minute would cause the output coordinates for the whole stack to be so far off. Any suggestions? I'll get in touch with the support people and see what they have to say about it.
On a related note, do you know of anyway to apply a series of band math expressions to a whole stack at once? The key issue is that two variables, related to the particular band being operated on, are different for each band. Here's an example of the band math:
(3.14597*b1*0.981108464293358)/(1848*cos(42.50525*(3.14597/180)))
(3.14597*b2*0.981108464293358)/(1549*cos(42.50525*(3.14597/180)))
(3.14597*b3*0.981108464293358)/(1114*cos(42.50525*(3.14597/180)))
I would like to be able to simple select the VNIR band stack and have the appropriate equation applied to the correct band and for the output to be another stack. Is there a simple way to do this without having to write my own extentions? Seeing as how ENVI can apply, according to the help manual, a single band math equation to a whole stack one would think that applying a set of different equations to a stack would be a logical extension. Thanks,
C
|
|
|
|
MariM
Veteran Member
Posts:2396
| 18 Aug 2010 07:36 AM |
|
|
One of the main reasons the output extents/coordinates will be different is because ASTER data is stored with a rotation from north so that it minimizes the background areas. When you stack and reproject, the rotation will be removed, the data will be oriented north up, and you will have more background area. Perhaps this is the main difference you are seeing. The locations of the actual data pixels should be approximately the same.
With respect to your second question - if the equations are different for each band, then they will need to be applied to each band. You can do it in Band Math, band-by-band, or you could write an IDL routine, or use MATH_DOIT to do something similar.
|
|
|
|
Deleted User
New Member
Posts:
| 20 Aug 2010 08:13 AM |
|
|
Okay, after more work on this issue, I think I've narrowed down the potential problem. After each processing step the coordinates seemed to be accurate when I start from the raw ASTER data. All day yesterday the coordinates for the processed data were correct at each step and the images were overlain properly in the mosaic widget. However, as has happened previously, when I returned this morning and pulled up the processed images (the ones that had correct coordinates yesterday) I found that the Map Info is displaying incorrect coordinates again. Something is occuring that corrupts the headers in ENVI after a reboot of the system, I think. Is anyone else using linux with ENVI and encountering strange behaviour?
Chris
|
|
|
|
Deleted User
New Member
Posts:
| 20 Aug 2010 08:52 AM |
|
|
I've never heard of such a thing in my 15+ years of working with ENVI. I'd bet a lot of money that the software is not changing the map info in your header. And it's hard to imagine how the OS could be corrupting a text file (the header) during a reboot. That would be a huge problem for the OS that would be reported all over the internet.
Have you tried opening the header file in a text editor when it looks good, and then again when it appears to have been corrupted? If it's different, then it's being corrupted, but if it's the same, then something else is going on. Here are the possibilities that occur to me:
1. Maybe you've named your processed images using the same root name as your original HDF ASTER images, so the software is getting confused about which header to use. To remedy this, try renaming each of your processed images (and corresponding headers) to something quite different than any other processed image, and different than the original HDF ASTER file.
2. Maybe you're not really opening the same file before and after the reboot? Could you be getting confused about which file you were looking at?
3. Do you have an enemy in your lab who would be devious enough to mess with your headers when you go home for the day?
|
|
|
|
Deleted User
New Member
Posts:
| 20 Aug 2010 09:21 AM |
|
|
I always rename my files to something approriate so that they can be easily identified. How could a program 'get confused' about which headers to read unless the file names are identical (with the exception of a lousy software engineer that only wanted to read in the first few bytes of a name or something)? Even if I wanted to use identical filenames, ENVI would ask me about overwritting the data - and my OS wouldn't allow there to be two identically named files in the same folder. So, I don't think that it's a file name issue. I'm also certain that I'm not confusing them since, again, my naming convention is pretty clear and there are only 3 stacks to look at in any case in the folder that contains the processed data that I'm talking about . For those reasons, I highly doubt that it's because I'm confusing my own file names (and am a little offended at the suggestion I have to say -- like when the tech support guy at Dell says 'lets try restarting your system' as they go through the moron check list). No one with the exception of me and my supervisor have access to this computer. None of my other software is giving me trouble so again I just don't understand what could possibly be causing the header files to spontaneously change their coordinates by over a hundred degrees (in this case -- other times it's been less than that). I did check the header file and the coordinates are incorrect in there -- just as they are being displayed in the avialable bands list. The only reason I think there might be something happening in the course of terminating ENVI or rebooting, is that before that ocurred the coordiantes were correct and after that occured the coordinates are incorrect. Anyone have any useful critical troubleshooting to offer?
Chris
|
|
|
|
Deleted User
New Member
Posts:
| 20 Aug 2010 09:41 AM |
|
|
I didn't mean to offend. I was more trying to be funny. I don't think one has to be a moron to get mixed up about which file they were looking at. Or at least, I've done it before. I always think it's good to double check the obvious stuff, before moving on to more complicated theories.
I don't know what OS you are on, but on UNIX/Linux, you can have two ENVI image files that have the same root name. Because ENVI identifies the header for a file by matching its rootname (and the filename extension is always .hdr) it can get confused about which header goes with which, because both would be called rootname.hdr. What are your filenames, by the way?
If the OS or ENVI were corrupting the header during shutdown or startup, it's surprising (to me anyway) that the header would end with valid map info in it. I would expect something more along the lines of garbage, in which case, ENVI wouldn't apply any map info to your file when you read it. So, there is some kind of method to this madness.
But that's as far as I can help you. I can't imagine what is going on for you. I truly have never heard of the map info in a header getting changed like this. And I have used ENVI for a long time, on lots of systems, with lots of data files (including ASTER). Sorry.
|
|
|
|
Deleted User
New Member
Posts:
| 20 Aug 2010 09:56 AM |
|
|
I'm still not clear on the rootname issue. Here are the file names of images before they have been stacked and after:
before:
AST_TOA_VNIR_21597_Band1
AST_TOA_VNIR_21597_Band2
AST_TOA_VNIR_21597_Band3
AST_TOA_SWIR_21597_Band4
...
AST_TOA_VNIR_22053_Band1
...
AST_TOA_VNIR_22114_Band1
after:
AST_cloudmsk_stack_21597
AST_cloudmsk_stack_22053
AST_cloudmsk_stack_22114
These names are all quite different than, and in different folders than, the original ASTER data file names, which as I'm sure you know, are great long alphanumeric strings. I'm about to try, again for the millionth time it seems, reprocessing everything from the raw data, restacking the images, and then terminating ENVI. I'll restart ENVI again and see if that's where the change is happening, and then I'll do the same thing with a complete system restart. I'd be surprised if I'm right about exactly when the change is occuring too since this would seem to be a significant bug in either ENVI or Ubuntu 10.4 LTS (my current OS). However, since no other files are having problems then I suspect it's something else too. As for the 'corruption' of the text files, you're right again and I would think that if they are being corrupted then the text file should just look like garbage after the fact. However, in this case it seems that only the coordinates are being changed, which suggests again that it's not a restart thing and might have something to do with when ENVI reads the file information after a restart. It's going to take a while to really locate when the change is occuring, but it's been infuriating to deal with. The analysis should only take a single day, but it's been weeks of restarting the whole process each time the 'bug' crops up. I may have to try running ENVI in wine or move it completely over to a windows machine. As an aside, it would be easier to troubleshoot if it weren't proprietary (just my plug for open source). I do appreciate any time you devote to thinking about this problem,
Chris
|
|
|
|
Deleted User
New Member
Posts:
| 20 Aug 2010 10:12 AM |
|
|
What projection is your data in? Maybe ENVI is able to understand the projection when it first reads your ASTER data, and then passes that along during processing. But then maybe when you start a new session and read the data in, it has lost the ability to correctly understand the projection? I'm grasping at straws here. But I am curious what projection and datum you see ENVI applying to the same data file when you open them before and after things change.
|
|
|
|
Deleted User
New Member
Posts:
| 20 Aug 2010 10:20 AM |
|
|
I've just completed the processing chain again for the 22114 granule so I'm going to save a copy of the header since the coordinates appear correct and then go through the restart. My data is in UTM Z16N WGS-84. I've wondered about that too though so I guess, assuming that the bug actually crops up again (infamous Murphy's Law might mean that I'll look like a complete idiot again and the headers will look the same until I decide I must have been imagining things and it happens again next week....) that we'll see what the differences are in total after the reboot. This will take a couple of minutes and then I'll post the results. Thanks again,
C
|
|
|
|
Deleted User
New Member
Posts:
| 20 Aug 2010 10:45 AM |
|
|
Okay, I've made a bit of progress. After restarting ENVI the UTM coordinates are still correct, so is the projection and datum. However, the geographic coordinates are way off again. I also rebooted my system to see if that would have a furhter imapact and it didn't. I imagine that whatever the issue is it will propagate into further processes completed using the 'bad' data. This is what happened before too - I recall now noticing that the geographic coordinates were off, but I dismissed it as some kind of minor ENVI error since the UTM coordinates were okay and the images still overlapped in the mosaic widget. After more work though, the images stopped overlapping and the UTM coordinates were also wrong though I don't know where along the process that error occured. It seems as though it all starts with this minor error regarding the geographic coordinates. The geographic coordinates should be something like:
-89.something W
16.somthing N
There's something strange overall about the UTM coordinates also since, as far as I know, the easting should be a 6 digit number regardless of the zone. Here is the ASTER .met file container for the spatial information:
GROUP = SPATIALDOMAINCONTAINER
GROUP = HORIZONTALSPATIALDOMAINCONTAINER
GROUP = GPOLYGON
OBJECT = GPOLYGONCONTAINER
CLASS = "1"
GROUP = GRINGPOINT
CLASS = "1"
OBJECT = GRINGPOINTLONGITUDE
NUM_VAL = 4
CLASS = "1"
VALUE = (-89.3983131794813, -88.7049544259472, -88.7929778007792, -89.4842892028109)
END_OBJECT = GRINGPOINTLONGITUDE
OBJECT = GRINGPOINTLATITUDE
NUM_VAL = 4
CLASS = "1"
VALUE = (16.9564202315803, 16.8569288430389, 16.2940237488295, 16.3934545638717)
END_OBJECT = GRINGPOINTLATITUDE
OBJECT = GRINGPOINTSEQUENCENO
NUM_VAL = 4
CLASS = "1"
VALUE = (1, 2, 3, 4)
END_OBJECT = GRINGPOINTSEQUENCENO
END_GROUP = GRINGPOINT
GROUP = GRING
CLASS = "1"
OBJECT = EXCLUSIONGRINGFLAG
NUM_VAL = 1
CLASS = "1"
VALUE = "N"
END_OBJECT = EXCLUSIONGRINGFLAG
END_GROUP = GRING
END_OBJECT = GPOLYGONCONTAINER
END_GROUP = GPOLYGON
END_GROUP = HORIZONTALSPATIALDOMAINCONTAINER
END_GROUP = SPATIALDOMAINCONTAINER
Here's a screenshot to help me explain:
|
|
|
|
Deleted User
New Member
Posts:
| 20 Aug 2010 10:51 AM |
|
|
I couldn't seem to add more text after the screen shot so I'm double posting here - sry. I checked both the copy of the header (seen in the screen shot) and the header of the image stack after I restarted ENVI. Both headers are the same still so nothing has happened to the header yet. If it's not a header issue, as this bout of troubleshooting would suggest, then what could be the problem?
Chris
|
|
|
|
Deleted User
New Member
Posts:
| 24 Aug 2010 10:03 AM |
|
|
Hi All,
I just thought that I should update the forum on this issue. I've been working with a couple of people at the ITT tech support desk and we've determined that there is a real problem here and I'm not just crazy. In short, while operating in Ubuntu 10.04 LTS 64bit, ENVI writes incorrect header information. At first, the Map Info will display correctly because ENVI reads the header info from a raw data source such as NASA .hdf-eos correctly (and presumably any other source headers), but when the file is saved in ENVI format, or processed in any way that involves writing a new header file, the following two issues occur:
1. Earth radius information is written incorrectly and, at least in my case and the machine tested by ITT, the last digit before the proper decimal place is written as another period '.' instead of the correct digit
2. the false easting in the header file is written as 50 000 when it should be 500 000
It's possible that simply editing the headers manually after they are written in order to correct for the issue at every step of your processing chain is a work-around, but I haven't tried that yet. I'm also not certain if it's a Ubuntu 10.04 LTS specific issue, or if the problem is going to recurr with other 64bit OS. The bug does not seem to appear on 32bit OS with either Windows or Linux. If you're using Ubuntu 10.04 LTS 64bit with ENVI be aware that the error will definitely propagate and result in very strange coordinates for all of your processed images. It's uncertain exactly where in the ENVI/Ubuntu interface things have gone awry.
Chris
PS After some brief testing, manually editing the headers seems to work. I'm not sure if this will be a problem for exporting files though. Also, according to ITT tech support the following OSs do not have this problem:
64-bit Ubuntu 10.04 running ENVI 4.7 SP2 in 32-bit mode
32-bit Ubuntu 9.04
64-bit openSUSE 11.3
64-bit CentOS 5.5
The bug still crops up on this one:
64-bit Fedora 12
|
|
|
|
MariM
Veteran Member
Posts:2396
| 24 Aug 2010 12:31 PM |
|
|
It does not seem to be an issue with Ubuntu 9.10 64-bit either. It could be that a newer linux library is causing some sort of issue with the projection engine.
|
|
|
|