ENVI has no way of knowing what units your image data are in (radiance, reflectance?) so it is up to the user to either scale the data to match the library or scale the library to match the data.  I would probably set the scale factor of the library so that it matches the data in the Endmember collection dialog rather than scaling the image.  The wavelengths of the input spectral library will be resampled to the dataset wavelengths on the fly, so this is taken care of for you as long as the wavelengths and wavelength units are known for both datasets.  This is true for all algorithms available in the Endmember Collection dialog.