INTERNAL: ROC Curves In ENVI
Anonym
Topic:
Receiver Operating Characteristic Curves (ROC Curves) are used to assess the performance of classification algorithms. How does ENVI create ROC Curves and how should they be interpreted?
Discussion:
ROC curves are used to visualize a classification’s performance and to help in selecting a suitable operating point. For example, choosing they optimum distance threshold for a minimum distance classifier.
They are created by plotting the probability of detection versus the probability of false alarms for a series of classifications. The selected operating point then depends on the desired probability of detection (PD) and probability of false alarm (PFA). Consequently, ROC curves may show that in some cases a selected classification method cannot perform to the desired PD and PFA.
The following steps illustrate the processes ENVI uses for calculating a ROC curve:
1. Compute a threshold and classify the rule image based on this threshold
2. Compare the classified image to the ground truth and count the number of pixels classified properly and the number of pixels classified improperly.
3. Compute one of the points for the ROC curve where:
PD = points classified properly / # of points in class
PFA = points classified improperly / # of point not in class
4. Repeat number 1-3 for a number of different threshold values. Each repeated point becomes a point on the ROC curve.
To fully understand the performance of a classification technique a ROC curve should be derived over a threshold range where the probability of detection and false alarm is zero to a point where they are both one. The optional probability of detection verse threshold plot is useful to determine a suitable operating (threshold) value for the classification Increasing the number of points used to generate the ROC curve may help detail areas of interest and the threshold range can also be used to detail areas of interest.
Explanation to customer:
Subject: RE: Incident #: 123949 #303357 ROC Curves
I am concerned that there is a bug in the ROC curve plotting algorithm. I have attached two plots:
1) ROC curve from matched filter output
2) Statistics from the "cin2" class matched filter output
The ROC curve shows very high Pd and low Pfa but the stats for the "cin2" channel 2 show that band 1 has a higher filter output. The filter output for channel 2 (cin2) should be the highest value with a mean of 1. It has a mean of one but it is not the maximum filter output as would be expected from the ROC curves (green curve) ... I am concerned that there may be a bug with the ROC curve algorithm. The observed ROC curve for cin2 (green) could not be possible given that it does not have the highest filter output...
Hi,
Thanks for your patience! ROC questions usually require a fair amount of digging and research, especially when it's a potential bug. The good news is that there doesn't seem to be a bug in this case. What you're seeing is actually a good illustration of why MTMF is superior to Matched Filter. My apologies for the length of reply and if you're already familiar with the following material, but I do promise it leads to an explanation of the curves you're seeing:
A score of more than 1 on Matched Filter is not usually a good thing, it's an indicator of a false positive. MF scores are created by a vector from your dataset mean to your ROI/class mean (or library spectra, if that's the approach you're using). Then every pixel in the scene is collapsed onto that vector. The vector is scaled, using the assumption(s) of the geometric mixing model, such that the mean of the data set is a score of zero, and the mean of your class/roi is one. A score of less than zero means it plots on the vector away from your class/roi, *past* your dataset mean. It's safe to say that the pixel contains none of your material in question if it has a score of zero or less. A score of 0.5 means that, according to the theory and all underlying assumptions of MF, about 50% of the pixel has your class/material/roi of interest. A score of greater than one should be a red flag for a false alarm for the following reason(s).
MF assumes that your ROI/class/material of interest is spectrally unique and unusual, and that most image pixels are not. Therefore, if your ROI/class/material is plotted in n-dimensional space (either spectral space or MNF space, whichever you're using), and so are the image data, then you'll have a big cloud of image data representing the image (with mostly unremarkable mixed pixels) and your class/material/roi plotting somewhere outside of that. Trouble arises when you get another class/material/roi which plots in roughly the same direction as your first class/material/roi, but off to the side, further out, etc. Any pixels from that class/material/roi, *and any from the image containing the same material* will get a relatively high MF score for the first class/material/roi, too.
This is a bit easier to explain as a graphic illustration, so I've attached a quick free-hand drawing to help show what's happening. I've drawn it as an obivously extreme case, to make it easier to interpret, but hopefully it will help. Note that the pixel to be tested in that case would get an MF score of 1.0 for target A, and only a score of 0.8 for target B, even though it's right on top of B's vector. Target B itself would get a score of greater than one for the MF results of target A! Clearly this would be a problem.
MTMF solves this problem by adding another parameter. "Cones" are defined from the target through the variance (standard deviation) contours of the data set. Pixels now get not only a score of where they plot on the vector for a target, but they get an "infeasibility score" determined by which "cone" they fall into. In the case of our test pixel in the illustration, it would still have a 1.0 score for target A, but you can see it would have an extremely high infeasibility score (many, many sigma out for that cone!). However, it's 0.8 score for target B would be accompanied by an exceptionally low infeasibility score, adding real confidence to our assessment of what's in that pixel.
Now, the fun part comes in with ROC analysis. Target A in that case is your CIN2 roi/material/class. Target B is CIN1. ROC analysis in that case would tell you "Yes, CIN1 would get a really high score returned from MF results on CIN2, however, we know that the CIN1 region is *not* CIN2, so it's curve will be skewed by a higher probability of false alarm." And that's exactly what's in your charts: CIN1's curve gets pulled down and to the right of the CIN2 curve in the ROC plot. Again, I'm sorry about the length of this email. It's not a very brief topic, but I hope this helps! Do you think a tech tip on this topic might be useful for other users? If so, please do let me know. We try to put info of interest to our customers in there relatively frequently to keep the content 'fresh'. Thanks!