A couple of things may be going on. First, I believe the TC transform for Landsat-5 data in ENVI is for uncalibrated data in the original DNs. So input of reflectance data may yield incorrect values. The TC transform for Landsat-7 data should be in calibrated reflectance. Second, FLAASH outputs reflectance data scaled by 10000. I am not sure the TC transform is equiped to deal with scaled reflectance so you may have to convert the data to floating point reflectance between 0-1.0 (masking data that are outliers). You could do a division of the data by 10000 using Band Math.
This issue was discussed in another post as well, which might be useful:
http://www.ittvis.com/Use...e/posts/Default.aspx