What version of ENVI are you using?
MNF should honor a mask but it does not use the Data ignore value in the header. So the mask you create should mask every pixel you want excluded in the MNF. You may check if the pixels are masked by calculating statistics using your mask.
NaN is still a 'number' in the sense that it exists, It does not exclude those pixels during processing. For MNF, you can only exclude pixels using a supplied mask to the tool.

I used ENVI 5.3 and 5.5, and the problem appeared in both version. The mask is a binary file，the pixels in my interest areas are 1，and the others are 0. I have checked the mask by computing statistics, and it was effective. But when i select this mask in MNF transform，different value i set for the exclude pixels in the original image，would lead to different MNF results.

I used ENVI 5.3 and 5.5, and the problem appeared in both version. The mask is a binary file，the pixels in my interest areas are 1，and the others are 0. I have checked the mask by computing statistics, and it was effective. But when i select this mask in MNF transform，different value i set for the exclude pixels in the original image，would lead to different MNF results.

I am not sure I understand this statement:
But when i select this mask in MNF transform，different value i set for the exclude pixels in the original image，would lead to different MNF results.

What do you mean by 'different value i set for exclude pixels in original image'?

I ran a test calculating statistics on an image with a mask and compared the mean to the mean calculated in MNF and they are the same. I believe it is honoring the mask.

Dear Mari Minari，thanks for your reply.
'different value i set for exclude pixels in original image', I mean that the value i set for mask background value.
I still have the problem, and the data processing flow is as follows:
1> subset data from ROIs (to get my interest areas). I choose the option that mask pixels out of ROI, and set background value to 0 or-9999(these are the different values i mentioned above). Output result was named a.dat(background value is 0) or b.dat(background value is -9999) .
2> calculate MNDWI by Band Math(in order to mask water by MNDWI before MNF transform). Output result was named MNDWI.dat
3> build mask. input files MNDWI.dat → import data range → data max value 0.00001. Output result was named mask.dat. It is a binary file, pixels with value of 1 represent area of interest but exclude water areas, pixels with value of 0 include water areas and areas out of ROI.
4> MNF transform. 1)input file (a.dat)→ select mask band(mask.dat), Output result was named mnf_a.dat.
2)input file (b.dat)→ select mask band(mask.dat), Output result was named mnf_b.dat.
5>I load both of mnf_a.dat and mnf_b.dat in the ENVI view, but the cursor values of each file are quite different.
That is to say, although i have chosen a mask in MNF transform, the background value still participate in the computation of MNF transform. I don't know where did i go wrong. If the problem still exists, could you tell me what is the most reasonable value should i set for the background in step 1. The other question is whether the different MNF results will affect the PPI computation.
Thanks a lot.

If you subtract your masks (mnf_a - mnf_b) , and calculate statistics on the result, is every pixel 0? This would mean the masks are equivalent and should mask the same pixels for the MNF transform. If they are not identical, then you are masking different pixels when using the two different masks and this could result in different MNF results.
The best value to set the background to is one that *does not exist in the data*. So you would need to make sure that 0 or -9999 are not valid values within the MNDWI.dat when building the mask.

I am sorry, perhaps my English is poor, and you couldn't understand me correctly. In my data processing, there is only one mask file which is named mask.dat. In the MNF transform i used the same mask file, so it would mask the same pixels. There is only one difference that is the background value of the images used for MNF transform.The images was named a.dat(background value is 0) and b.dat(background value is -9999) which i mentioned in the last reply. mnf_a.dat and mnf_b.dat are mnf transform result, they are different in my processing, so mnf_a - mnf_b can't be 0 for any pixels. I don't know if I express myself clearly. May I send you an email to upload data to you.

If I apply mask before MNF transform in the version of ENVI 5.3, and there is an option that is mask value. Suppose i put 0 or -9999, and give the result name as c.dat or d.dat respectively. Then i do MNF transform using c.dat or d.dat without selecting mask band, the results are still quite different.

Applying the mask sets a data ignore value in the header but the MNF tool does not honor this. It will only honor the mask supplied in the file input dialog.
You might want to contact your local distributor with this question. I am not yet able to reproduce it.

Thank you all the same！

When performing the MNF with your two images with different background values, can you try setting the 'shift difference' to an area within the image and not the entire image? I believe it is the shift difference that may not use the mask. Make sure the areas for shift-difference is data-only and not background.

Thank for your reply. I had tried to set the shift difference to an area by ROI, but it had no effect at all. ENVI still processed the entire image. There was no difference between the two MNF results of setting shift difference and not setting shift difference. Perhaps the ROI is irregular, and the minimum enclosing rectangle of the ROI is the entire image!
Should I process data as follows:
1.subset the image by rectangle, there will be no background value.
2.perform the MNF
3.subset MNF by ROI,to get my interesting area MNF
but i think the size of the rectangle will influnce the MNF.

'I believe it is the shift difference that may not use the mask'. Yes, you are right. When I set the shift difference to an area within the image by the 'image' button to drag an rectangle, I obtained the same MNF result for different background values. But the premise is that the rectangle for setting the 'shift difference' must be the same for the two different background values images. That is to say, the area of 'shift difference' still influence the MNF results. So, how to setting the 'shift difference', where and how large, it is still a problem puzzled me.

When I tested the MNF transform with the same image but with two different backgrounds (-9999 and 0), I selected the exact same shift difference area for both. Using the same shift difference area that contains all data pixels (and no NoData pixels) and the mask of the background image area with both images, my MNF results were identical.
And yes, the area of shift difference affects the results. This area is used to calculate the noise statistics and typically should be choosen from an area that is relatively homogeneous. So for the qb_boulder_msi sample data, I selected a 100x100 area within the water feature.

Thank you very much！Your reply was very helpful.
And Some researchers suggested that the background values should be set to NaN. But when I changed the background values as NaN, the MNF processing without setting shift difference would be no response at 99%. Does it mean NaN is not allowed in MNF transform? Because it cann’t calculate the covariance matrix with pixels of NaN.

NaN is still a valid value and it is not ignored, so if a NaN is present in the data even with the mask, it will likely cause issues. NaN values should be masked.

With your help I've solve the problem at last. Thank you very much.