The metadata may not be imported if the data has been 'DRA-Corrected'. It should say this in the IMD:
radiometricEnhancement = "Off";

If it shows that some sort of DRA (dynamic range adjustment) has been applied and the data are 8-bit rather than 16-bit, then ENVI will not extract the metadata because these products are not appropriate for radiometric correction.

Thank you for the quick reply.
The metadata says following:

radiometricLevel = "Corrected"
radiometricEnhancement = "Off"
bitsPerPixel = 16

It doesn't look like the DRA corrected? Below is the full IMD.

Tomoko

version = "28.0";
generationTime = 2016-08-12T19:52:29.000000Z;
productOrderId = "055559477060_01_P001";
productCatalogId = "A0100101F72AC900";
childCatalogId = "20300101F72ACA00";
imageDescriptor = "Standard2A";
bandId = "Multi";
panSharpenAlgorithm = "None";
numRows = 3508;
numColumns = 12367;
productLevel = "LV2A";
productType = "Standard";
numberOfLooks = 1;
radiometricLevel = "Corrected";
radiometricEnhancement = "Off";
bitsPerPixel = 16;
compressionType = "None";
BEGIN_GROUP = BAND_C
ULLon = -162.17626221;
ULLat = 5.90623382;
ULHAE = 16.00;
URLon = -161.99774416;
URLat = 5.90529789;
URHAE = 16.00;
LRLon = -161.99801613;
LRLat = 5.85460708;
LRHAE = 16.00;
LLLon = -162.17651806;
LLLat = 5.85553492;
LLHAE = 16.00;
absCalFactor = 9.295654e-03;
effectiveBandwidth = 4.730000e-02;
TDILevel = 24;
END_GROUP = BAND_C
BEGIN_GROUP = BAND_B
ULLon = -162.17626221;
ULLat = 5.90623382;
ULHAE = 16.00;
URLon = -161.99774416;
URLat = 5.90529789;
URHAE = 16.00;
LRLon = -161.99801613;
LRLat = 5.85460708;
LRHAE = 16.00;
LLLon = -162.17651806;
LLLat = 5.85553492;
LLHAE = 16.00;
absCalFactor = 1.783568e-02;
effectiveBandwidth = 5.430000e-02;
TDILevel = 10;
END_GROUP = BAND_B
BEGIN_GROUP = BAND_G
ULLon = -162.17626221;
ULLat = 5.90623382;
ULHAE = 16.00;
URLon = -161.99774416;
URLat = 5.90529789;
URHAE = 16.00;
LRLon = -161.99801613;
LRLat = 5.85460708;
LRHAE = 16.00;
LLLon = -162.17651806;
LLLat = 5.85553492;
LLHAE = 16.00;
absCalFactor = 1.364197e-02;
effectiveBandwidth = 6.300000e-02;
TDILevel = 10;
END_GROUP = BAND_G
BEGIN_GROUP = BAND_Y
ULLon = -162.17626221;
ULLat = 5.90623382;
ULHAE = 16.00;
URLon = -161.99774416;
URLat = 5.90529789;
URHAE = 16.00;
LRLon = -161.99801613;
LRLat = 5.85460708;
LRHAE = 16.00;
LLLon = -162.17651806;
LLLat = 5.85553492;
LLHAE = 16.00;
absCalFactor = 5.829815e-03;
effectiveBandwidth = 3.740000e-02;
TDILevel = 21;
END_GROUP = BAND_Y
BEGIN_GROUP = BAND_R
ULLon = -162.17626221;
ULLat = 5.90623382;
ULHAE = 16.00;
URLon = -161.99774416;
URLat = 5.90529789;
URHAE = 16.00;
LRLon = -161.99801613;
LRLat = 5.85460708;
LRHAE = 16.00;
LLLon = -162.17651806;
LLLat = 5.85553492;
LLHAE = 16.00;
absCalFactor = 1.103623e-02;
effectiveBandwidth = 5.740000e-02;
TDILevel = 10;
END_GROUP = BAND_R
BEGIN_GROUP = BAND_RE
ULLon = -162.17626221;
ULLat = 5.90623382;
ULHAE = 16.00;
URLon = -161.99774416;
URLat = 5.90529789;
URHAE = 16.00;
LRLon = -161.99801613;
LRLat = 5.85460708;
LRHAE = 16.00;
LLLon = -162.17651806;
LLLat = 5.85553492;
LLHAE = 16.00;
absCalFactor = 5.188136e-03;
effectiveBandwidth = 3.930000e-02;
TDILevel = 21;
END_GROUP = BAND_RE
BEGIN_GROUP = BAND_N
ULLon = -162.17626221;
ULLat = 5.90623382;
ULHAE = 16.00;
URLon = -161.99774416;
URLat = 5.90529789;
URHAE = 16.00;
LRLon = -161.99801613;
LRLat = 5.85460708;
LRHAE = 16.00;
LLLon = -162.17651806;
LLLat = 5.85553492;
LLHAE = 16.00;
absCalFactor = 1.224380e-02;
effectiveBandwidth = 9.890000e-02;
TDILevel = 10;
END_GROUP = BAND_N
BEGIN_GROUP = BAND_N2
ULLon = -162.17626221;
ULLat = 5.90623382;
ULHAE = 16.00;
URLon = -161.99774416;
URLat = 5.90529789;
URHAE = 16.00;
LRLon = -161.99801613;
LRLat = 5.85460708;
LRHAE = 16.00;
LLLon = -162.17651806;
LLLat = 5.85553492;
LLHAE = 16.00;
absCalFactor = 9.042234e-03;
effectiveBandwidth = 9.960000e-02;
TDILevel = 24;
END_GROUP = BAND_N2
outputFormat = "GeoTIFF";
BEGIN_GROUP = IMAGE_1
satId = "WV02";
mode = "FullSwath";
scanDirection = "Forward";
CatId = "103001003849B100";
firstLineTime = 2014-11-14T21:18:58.547262Z;
avgLineRate = 5000.01;
exposureDuration = 0.00020000;
minCollectedRowGSD = 2.105;
maxCollectedRowGSD = 2.106;
meanCollectedRowGSD = 2.106;
minCollectedColGSD = 2.413;
maxCollectedColGSD = 2.415;
meanCollectedColGSD = 2.414;
meanCollectedGSD = 2.255;
rowUncertainty = 36.27;
colUncertainty = 63.47;
minSunAz = 143.5;
maxSunAz = 143.5;
meanSunAz = 143.5;
minSunEl = 59.7;
maxSunEl = 59.7;
meanSunEl = 59.7;
minSatAz = 79.0;
maxSatAz = 79.5;
meanSatAz = 79.3;
minSatEl = 60.0;
maxSatEl = 60.0;
meanSatEl = 60.0;
minInTrackViewAngle = 8.2;
maxInTrackViewAngle = 8.4;
meanInTrackViewAngle = 8.3;
minCrossTrackViewAngle = 25.2;
maxCrossTrackViewAngle = 25.3;
meanCrossTrackViewAngle = 25.3;
minOffNadirViewAngle = 26.5;
maxOffNadirViewAngle = 26.5;
meanOffNadirViewAngle = 26.5;
PNIIRS = 2.8;
cloudCover = 0.000;
resamplingKernel = "NN";
positionKnowledgeSrc = "R";
attitudeKnowledgeSrc = "R";
revNumber = 26766;
END_GROUP = IMAGE_1
BEGIN_GROUP = MAP_PROJECTED_PRODUCT
earliestAcqTime = 2014-11-14T21:19:06.846789Z;
latestAcqTime = 2014-11-14T21:19:06.846789Z;
datumName = "WE";
semiMajorAxis = 6378137.0000;
inverseFlattening = 298.257223563;
datumOffset = (
0.000,
0.000,
0.000 );
mapProjName = "UTM";
mapProjCode = 1;
mapZone = 3;
mapHemi = "N";
mapProjParam = (
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000,
0.000000000 );
productUnits = "M";
originX = 812679.20000145;
originY = 653632.79999594;
orientationAngle = 0.0;
colSpacing = 1.60;
rowSpacing = 1.60;
productGSD = 1.60;
ULX = 812679.20000145;
ULY = 653632.79999594;
ULH = 16.00;
URX = 832464.80000372;
URY = 653632.79999503;
URH = 16.00;
LRX = 832464.80000342;
LRY = 648021.59999506;
LRH = 16.00;
LLX = 812679.20000125;
LLY = 648021.59999597;
LLH = 16.00;
DEMCorrection = "Coarse DEM";
numGCP = 0;
END_GROUP = MAP_PROJECTED_PRODUCT
END;

You are correct, this data looks appropriate for Radiometric Calibration. Are you running ENVI 5.3 SP1? You can check by going to Help->About ENVI.

WHen you open the file by selecting the .TIL (or drag and drop it in the display), then go to View Metadata->Spectral tab, do you see any of the spectral information present? Does the Radiometric Calibration tool warn you that the necessary information is not present in the file?

If necessary, you can manually add in the Spectral metadata by going to View Metadata ->Edit Metadata.and add in the necessary information for Radiometric Calibration.

I am running only ENVI 5.3. I guess I will install the service pack.

I might have totally misunderstand what was shown in the metadata in ENVI compared to IMD file.
Are Radiance Gains (shown in ENVI metadata) = abscalfactor/effective bandwidth (in IMD file)?
I thought the Radiance Gains and Offset are the absolute radiometric calibration factors provided by Digital Globe.
If that's the case, I am assuming the absolute radiometric calibration factors, provided by Digital Globe need to be separately processed in Apply Gain and Offset tool? Or is there any way to do all in together?

Yes these values are used in the gain/offset correction along with some hard-coded values in the code:
absCalFactor = 9.295654e-03;
effectiveBandwidth = 4.730000e-02

ENVI knows how to extract and apply them. If the gains and offsets are populated in the Spectral tab of the Metadata Viewer, you should be set to use Radiometric Calibration.