ENVI's SVM implementation uses LibSVM. You might refer to the paper titled LIBSVM: A Library for Support Vector Machines.
You can find a copy here:
http://dns2.asia.edu.tw/~...Sys%20Tec2,%2027.pdf Details of Probability Estimates were described in Section 8 of the paper (p22-24). On Page 22, "after collecting all rij values, Wu et al. [2004] proposed several approaches to obtain probability. In LIBSVM, we consider their second approach to solve the optimization problem.