I haven't done a lot with ENVI 5 as several tools I use didn't make it into the conversion from Classic to 5, but I tried the masking process as a test and was able to complete it. But there were a couple of spots where the process was not intuitive. Just to confirm, you created the vector (polygon?) file in 5, converted it to a ROI using the "Convert Vector to ROI" tool, and then also did a "Save as" on the roi layer to create an actual .roi file? The Convert tool does not seem to actually create a .roi file which is needed if you are going to use the .roi to create the mask. You then need to do the Raster Management > Masking > Build Mask tool. Under Options > Import ROI select the .roi file you created. Also make sure you select whether you want the "Selected Areas" on or off. Name the mask file whatever you would like and create the mask. You then need to apply the mask to the image/bands that you want using the Raster Management > Masking > Apply Mask tool. If you have the bands open in 5 it should give you a dialog box where you can select the image/bands under the "Select Input File", and then you need to be sure you also use the "Select Mask Band" button near the bottom of this dialog box to designate your mask you just created. Name the file and you should be set. The easiest things to miss are the actually saving as a .roi file (as you would assume the tool would do that for you), importing the .roi, and the "Select Mask Band" button steps. If you did all of this and it still doesn't work can you give me/us a bit more detail on the steps you are doing exactly so I can try to figure out where it's going off? Good Luck.