大协方差矩阵的并行计算

我们需要计算大小为的协方差矩阵 $10000\times10000$ 至 $100000\times100000$。我们可以访问GPU和集群，我们想知道什么是加快这些计算速度的最佳并行方法。

第一件事是认识到您可以使用BLAS执行此操作。如果你的数据矩阵是$X = [x_1 x_2 x_3 ...] \in \mathbb{R}^{m\times n}$ （每 $x$是对应于一个度量的列向量；行是试验），则可以将协方差写为：

$$C_{ij} = E[x_i,x_j] - E[x_i] E[x_j] = \frac{1}{n} \sum_k x_{ik} x_{jk} - \frac{1}{n^2} \left(\sum_k x_{ik} \right) \left(\sum_k x_{jk}\right)$$ 我们可以这样写： $$C = \frac{1}{n} X^T X - \frac{1}{n^2} (1^T X)^T (1^T X)$$ 哪里 $(1^T)$ 是具有所有元素1的行向量，所以 $(1^T X)$ 是列总和的行向量 $X$。这可以完全写成BLAS，其中$X^T X$是GEMM或更好的是SYRK / HERK，您可以$(1^T X) = b$使用GEMV时，$b^T b$再次使用GEMM或SYRK / HERK，以及使用SCAL的前置因子。

您的数据和结果矩阵可能约为64GB，因此您将无法容纳单个节点或一个节点的GPU价值。对于非GPU群集，您可能需要查看PBLAS，就像scalapack一样。对于GPU，还没有多节点库。 岩浆有某种底层的并行BLAS实现，但它可能不是用户友好的。我认为CULA尚不支持多节点，但需要密切注意。 CUBLAS是单节点。

我还建议您强烈考虑自己实现并行性，特别是如果您熟悉MPI并且必须将其连接到现有的代码库中时。这样，您可以轻松地在CPU和GPU BLAS之间进行切换，并以所需的确切位置开始和结束数据。您不应只需要几个MPI_ALLREDUCE调用。

我使用CUBlas和Cuda Thrust实现了@Max Hutchinson给出的公式，并与在线协方差计算工具进行了比较。我的似乎产生了良好的结果。下面的代码计划用于QDA贝叶斯。因此，给定的矩阵可能包含多个类别。因此，计算了多个协方差矩阵。我希望这对某人有用。

//! Calculates one or more than one coVarianceMatrix given data.
//  There can be many classes since many covariance matrixes.
/*!
    \param inMatrix This vector contains matrix data in major storage. 
    Forexample if inMatrix=[1 2 3 4 5 6] and trialSizes=[2] this means matrix we will work on a matrix like :
        |1 4 |
        |2 5 |
        |3 6 | -> 2 Trials, 3 Features. Columns contains feature rows contains trials (samples)
    \param trialSizes There can be many classes since many covariance matrixes. Samples from all classes will be given with inMatrix.
    But we need to know how many trials(samples) we have for each class. 
    For example if inMatrix=[1 2 3 4 5 6 7 8 9 10 11 12] and trialSizes=[2,2] 
    this means matrix we will work on a matrix like :
        |1 4 |  |7 10 |
        |2 5 |  |8 11 |
        |3 6 |  |9 12 |  --> Total number of trials(samples which is total rowCount) 2 + 2 = 4 , 
                             So colSize = inMatrix.size()/4 = 3(feature vector size)
                         --> There is two element in trialSize vec so each vector has to samples
*/
void multiQDACovianceCalculator(std::vector<float>& inMatrix, std::vector<int>& trialSizes)
{
    cublasHandle_t handle; // CUBLAS context
    int classCount = trialSizes.size();
    int rowSize = std::accumulate(trialSizes.begin(), trialSizes.end(), 0);
    int dimensionSize = inMatrix.size() / rowSize;
    float alpha = 1.0f;
    float beta = 0.0f; // bet =1

    thrust::device_vector<float> d_cov1(dimensionSize * dimensionSize);
    thrust::device_vector<float> d_cov2(dimensionSize * dimensionSize);
    thrust::device_vector<float> d_covResult(dimensionSize * dimensionSize);

    thrust::device_vector<float> d_wholeMatrix(inMatrix);
    thrust::device_vector<float> d_meansVec(dimensionSize); // rowVec of means of trials
    float *meanVecPtr = thrust::raw_pointer_cast(d_meansVec.data());
    float *device2DMatrixPtr = thrust::raw_pointer_cast(d_wholeMatrix.data());
    auto maxTrialNumber = *std::max_element(trialSizes.begin(), trialSizes.end());
    thrust::device_vector<float> deviceVector(maxTrialNumber, 1.0f);

    cublasCreate(&handle);
    // Inside of for loop  one covariance matrix calculated each time
    for (int i = 0; i < trialSizes.size(); i++)
    {
        // X*transpose(X) / N
        alpha = 1.0f / trialSizes[i];
        cublasSgemm(handle, CUBLAS_OP_N, CUBLAS_OP_T, dimensionSize, dimensionSize, trialSizes[i], &alpha,
            device2DMatrixPtr, dimensionSize, device2DMatrixPtr, dimensionSize, &beta,
            thrust::raw_pointer_cast(d_cov1.data()), dimensionSize);

        // Mean vector of each column
        alpha = 1.0f;
        cublasSgemv(handle, CUBLAS_OP_N, dimensionSize, trialSizes[i], &alpha, device2DMatrixPtr,
            dimensionSize, thrust::raw_pointer_cast(deviceVector.data()), 1, &beta, meanVecPtr, 1);

        // MeanVec * transpose(MeanVec) / N*N
        alpha = 1.0f / (trialSizes[i] * trialSizes[i]);
        cublasSgemm(handle, CUBLAS_OP_T, CUBLAS_OP_N, dimensionSize, dimensionSize, 1, &alpha,
            meanVecPtr, 1, meanVecPtr, 1, &beta,
            thrust::raw_pointer_cast(d_cov2.data()), dimensionSize);

        alpha = 1.0f;
        beta = -1.0f;
        //  (X*transpose(X) / N) -  (MeanVec * transpose(MeanVec) / N*N)
        cublasSgeam(handle, CUBLAS_OP_N, CUBLAS_OP_N, dimensionSize, dimensionSize, &alpha,
            thrust::raw_pointer_cast(d_cov1.data()), dimensionSize, &beta, thrust::raw_pointer_cast(d_cov2.data()), 
            dimensionSize, thrust::raw_pointer_cast(d_covResult.data()), dimensionSize);

        // Go to other class and calculate its covarianceMatrix
        device2DMatrixPtr += trialSizes[i] * dimensionSize;
    }
    printVector(d_covResult);
    cublasDestroy(handle);
}