You should be able to import your binary file to ENVI directly, or you could output it to something like TIFF and import this to ENVI.  With your ascii file containing the lat/lon for every pixel - this data will need to be gridded to an image of the same dimensions as the image file, with one grid for lat and one for lon.  Then you could use ENVI's Georeference from Input Geometry ->Georeference from IGM tool, which takes an image file, a latitude band and a longitude band (all the same dimensions in samples/lines) and allows you to georeference the image to a map projection using this information.