It is not uncommon to have some pixel values outside of the range of 0-10000 (when using an output scale factor of 10000). This can happen in some areas of very low or high radiance due to water, shadow to clouds and corner reflectors. These areas are poorly modeled and I often suggest they be masked in further processes. However, if there are significant numbers of pixels or areas of pixels that are outside of the expected range, then it is often a sign that a parameter is off or most commonly, the scale factor is not correct for the input data.

The input scale factor is divided into the data to get to the expected unit of microW/(cm^2*sr*nm). I suggest that the data be processed using the Radiometric Calibration tool using 'FLAASH Settings' to ensure the input data are in the correct units. If you use FLAASH Settings in this tool, the input scale factor will be 1 for FLAASH. The typical radiance cube in the correct units will range from about 2-50.

I would also suggest you leave the output scale factor as 10000 to ensure that you have an appropriate dynamic range of output integers for the reflectance cube. Then, if you want the output cube in 'percent' reflectance, you can use Band Math to divide the data by 10000 to get to values of approximately 0-1 (floating point output). Then I would mask pixels that are outside of this range when processing further.

It can also be useful to run Quac on the radiance data for comparison. Quac is developed by the same scientific group that developed FLAASH but uses a different method to determine surface reflectance that is typically within +/-15% of the accuracy of the FLAASH model. It is far easier to use and can be a better option if you are unsure of the parameters to use for FLAASH. Using the wrong parameters in FLAASH can produce very bad results. Quac also outputs a reflectance cube scaled by 10000 to integers. When using this tool, be sure to include a mask that excludes background border pixels for the calculation.