You cannot pass a mask to the ENVI Classic ENVI_SVM_DOIT routine since they were never supported. However, with ENVI 5.4, there is the new Classification Framework which supports the SVM algorithm. With this API, you can pass a masked raster:
https://www.harrisgeospatial.com/docs...
http://www.harrisgeospatial.com/docs/...