The Raster to Vector conversion tool is typically used with elevation data to create contours.  I don't think you would want to use it with something as complex as Landsat TM images since it would not be a very efficient way to store the data.  The DN is just the specific pixel value you want to convert to a vector.  For a digital elevation model, you may want to export in 100m increments, so you would perform a series of exports using the DNs (elevations) of interest - e.g. 100, 200, 300, etc.  I am not sure this makes sense with data such as Landsat TM but the process would be the same.  You can use the Cursor Location/Value tool from the Display to find the DN value you want to export and enter this in to the RTV tool.

Sir I have done edge detection in MATLAB of SPOT image. The output I have is in TIFF form and it is binary with '1' representing the boundary line. The problem is that when i am converting it into vector and want to find the area, it is mentioning the area of the boundary line instead of the area contained inside the polygon. The boundary line in too thick. How can i reduce the thickness of this boundary line in ENVI 5 so that only the inside polygon area is considered?