The RUNNING_COVARIANCE function computes the unbiased sample covariance and correlation between two vectors or for all columns of a two-dimensional array. The function can also combine previously computed values with new data to allow computing covariance and correlation on data sets that are too large to fit into memory.
RUNNING_COVARIANCE uses the Welford "online" algorithm to compute the covariance in a single pass through the data. The routine is faster than the CORRELATE function, and unlike CORRELATE, does not require any additional memory.
Examples
Define the data as vectors:
IDL> A = [1 ,2 ,3 ,4 ,5 ,6 ,7 ,8 ,9 ,10]
IDL> B = [-11 ,12 ,13 ,14 ,15 ,16 ,17 ,18 ,19 ,20]
Compute the [covariance, mean1, mean2, count]:
IDL> result = RUNNING_COVARIANCE(A, B)
IDL> ['covariance','correlation','meanX','varianceX','meanY','varianceY','count'], format='(7a12)'
IDL> result, format='(7f12.5)'
IDL prints:
covariance correlation meanX varianceX meanY varianceY count
20.16667 0.74673 5.50000 9.16667 13.30000 79.56667 10.00000
Define an array with A and B as its columns.
IDL> X = TRANSPOSE([[A],[B]])
Compute the matrix.
IDL> Z = running_covariance(X)
Access the data you are interested in.
IDL> print, Z.covariance
IDL prints:
9.1666667 20.166667
20.166667 79.566667
Syntax
Result = RUNNING_COVARIANCE( X, Y, [, /NAN] [, PREVIOUS=value]
)
Return Value
Returns the covariance of the arrays X and Y in the form of a seven element array of doubles: [covariance, correlation, meanX, varianceX, meanY, varianceY, count].
OR
If X is an M x N array consisting of M variables with N samples each, then the result will be the covariance matrix, which consists of an M x M array of IDL structures containing the following fields: {covariance, correlation, meanX, varianceX, meanY, varianceY, count}. In this case the element i,j in the result corresponds to the computation of the i-th column against the j-th column of the input array.
Arguments
X
A vector or an m x n array of any numeric type other than complex or double complex.
Y
A vector of any numeric type other than complex or double complex.
if X is an M x N array of M variables with N samples each, then Y should not be supplied.
Keywords
NAN
Set this keyword to cause the routine to check for occurrences of the IEEE floating-point values NaN or Infinity in the input data. Elements with the value NaN or Infinity are treated as missing data.
Note: Since the value NaN is treated as missing data, if you set /NAN and X and Y contain only NaN values, the routine will return NaN for all of the returned values, except for the count, which will be zero.
PREVIOUS
Set the PREVIOUS keyword to the result of the last calculation to continue from that value.
-
For vector mode (both X and Y are vectors) set this keyword to a seven-element array containing the [covariance, correlation, meanX, varianceX, meanY, varianceY, count].
-
For matrix mode (X is a 2D array) set this keyword to an M x M array of structures, where M is the number of columns in the input array. Each structure contains seven fields: {covariance, correlation, meanX, varianceX, meanY, varianceY, count}.
See below for examples of chaining together multiple calls to RUNNING_COVARIANCE using the PREVIOUS keyword.
Note: If the count from a previous calculation is zero, then a new calculation is started, regardless of the other values.
Thread Pool Keywords
This routine is written to make use of IDL’s thread pool, which can increase execution speed on systems with multiple CPUs. The values stored in the !CPU system variable control whether IDL uses the thread pool for a given computation. In addition, you can use the thread pool keywords TPOOL_MAX_ELTS, TPOOL_MIN_ELTS, and TPOOL_NOTHREAD to override the defaults established by !CPU for a single invocation of this routine. See Thread Pool Keywords for details.
When computing the covariance for a large number of values, the results will depend upon the order in which the numbers are combined. Since the thread pool will combine values in a different order, you may obtain a different — but equally correct — result than that obtained using the standard non-threaded implementation. This effect occurs because RUNNING_COVARIANCE uses floating point arithmetic, and the mantissa of a floating point value has a fixed number of significant digits. For more information on floating-point numbers, see Accuracy and Floating Point Operations.
Additional Examples
Using the PREVIOUS keyword
IDL> x = [1:10]
IDL> y = [11:20]
IDL> RUNNING_COVARIANCE(x,y)
IDL prints:
9.1666667 1.0000000 5.5000000 9.1666667 15.500000 9.1666667 10.000000
IDL> cov1 = RUNNING_COVARIANCE(x[0:4], y[0:4])
IDL> print, cov1
2.5000000 1.0000000 3.0000000 2.5000000 13.000000 2.5000000 5.0000000
IDL> RUNNING_COVARIANCE(x[5:*], y[5:*], PREVIOUS=cov1)
IDL prints:
9.1666667 1.0000000 5.5000000 9.1666667 15.500000 9.1666667 10.000000
Notice that both methods produce the same result, even though the second one is performed in two steps using the PREVIOUS keyword.
Using Matrix Mode
IDL> seed = 1
IDL> X = randomu(seed,2,100)
IDL> cov = running_covariance(X)
IDL> help, cov
COV STRUCT = -> <Anonymous> Array[2, 2]
IDL> help, cov[0]
** Structure <439538d0>, 7 tags, length=56, data length=56, refs=2:
covariance DOUBLE 0.087547755
correlation DOUBLE 1.0000000
meanX DOUBLE 0.48587793
varianceX DOUBLE 0.087547755
meanY DOUBLE 0.48587793
varianceY DOUBLE 0.087547755
count LONG64 100
IDL> cov.covariance
IDL prints:
0.087547754879168979 -0.0044967843113049967
-0.0044967843113049967 0.082687655455897777
Version History
|
8.8.3 |
Introduced |
|
9.3 |
Added matrix mode
|
See Also
MEAN, MOMENT, RUNNING_STATS, STDDEV, VARIANCE