I would not recommend that approach. If you cannot get your unmixing results to naturally make sense, then you don't want to just force them to look like they make sense. What this result is telling you is that your endmember spectra are not a good fit to your image. You will want to consider whether you might be missing one or more endmember spectra, or whether one or more of the spectra you are including as endmembers are not good examples of the materials in your scene. It's curious that with endmembers that don't fit well with reality you end up with low RMS error. I'm wondering whether you constrained your unmixing results to sum to 1? In general I don't recommend including such a constraint. It makes endmembers that might not be appropriate look like they are. It's better to try to get a set of endmembers that naturally end up with their sums for each pixel being near 1, and the fractions for each pixel being between 0 and 1 for each endmember. Then you know that the unmixing model is truly appropriate for the image, and you can trust the unmixing fractions that you get. But others disagree. That's why ENVI allows you to include constraints on the unmixing. - Peg

Thanks Peg. Actually I performed both, unconstrained and after constrained SU. In the first case the sum is close to 1, in the second, of course, is almost perfectly one. But in both cases the individaul percent covers are out of the right range [0,1]. Any suggestion?

OK, so those individual percent covers are telling you that your set of endmembers are not adequate for your imagery. You will want to try adjusting them. You may be missing some, and/or some of them may not be good examples of the actual materials in the scene. How did you come up with your set of endmembers? - Peg