ROIs are held in memory in double precision, regardless of the data type of the image. If you are creating a large ROI in double precision, then you can run into memory issues.
Your system may have 64GB of RAM, but a large portion of that is utilized by the OS and it is not *contiguous* memory which is required to store a large array in memory. You can try a reboot and then launch *only* ENVI to limit memory usage and fragmentation and try the process but it is likely you will need to reduce the size of the ROI or image.

Hi Mari,
I tried with the subset and still got the same error (If I go smaller then the current one it'll be too many subsets).
Is there any other way to work around with this? I am stuck on this from past few days, kindly help me!

You could try making a mask using the same thresholds and then create the ROI from the mask band.

Can you kindly guide me how to do that? As I don't have that extensive experience using ENVI.
It would be great if we can talk over email. ppbhatt@mtu.edu

You can submit a support incident through Technical Support. You will be required to provide a license number that is current on maintenance or work through your license administrator at your institution.

I tried emailing on your mail and a reply came back from the Harris team I have given them the installation number. Hope that works!