From the above images, you can see that the absolute values in the Minimum Covariance Determinant estimator are lower in comparison to the simple Covariance matrix, which means that the algorithm has eliminated some of the outliers in the data and the resulting covariance matrix estimator is a more robust one. Note that this method achieves the best result when the data has outliers in it.

Maximum Likelihood Covariance Estimator (Empirical Covariance)

Maximum Likelihood Estimator of a sample is an unbiased estimator of the corresponding population’s covariance matrix. This estimation works well when the number of observations is big enough in relation to the number of features.

We can implement this algorithm through the empirical_covariance method() in the PortfolioLab library.

# Finding the Empirical Covariance on price data
empirical_cov = risk_estimators.empirical_covariance(stock_prices, price_data=True)

# Transforming Empirical Covariance from np.array to pd.DataFrame
empirical_cov = pd.DataFrame(empirical_cov, index=cov_matrix.index, columns=cov_matrix.columns)

De-Noising Covariance Matrix

The main idea behind de-noising the covariance matrix is to eliminate the eigenvalues of the covariance matrix that are representing noise and not useful information.

Constant Residual Eigenvalue De-noising Method

This is done by determining the maximum theoretical value of the eigenvalue of such matrix as a threshold and then setting all the calculated eigenvalues above the threshold to the same value.

The function provided below for de-noising the covariance works as follows:

• The given covariance matrix is transformed to the correlation matrix.
• The eigenvalues and eigenvectors of the correlation matrix are calculated.
• Using the Kernel Density Estimate algorithm a kernel of the eigenvalues is estimated.
• The Marcenko-Pastur pdf is fitted to the KDE estimate using the variance as the parameter for the optimization.
• From the obtained Marcenko-Pastur distribution, the maximum theoretical eigenvalue is calculated using the formula in the section Instability caused by noise in the following paper – A Robust Estimator of the Efficient Frontier.
• The eigenvalues in the set that are above the theoretical value are all set to their average value. For example, we have a set of 5 sorted eigenvalues two of which are above the maximum theoretical value, then we set \begin{aligned}\lambda^{NEW}_{4} = \lambda^{NEW}_{5} = \frac{\lambda^{OLD}_{4} + \lambda^{OLD}_{5}}{2}\end{aligned}
• Eigenvalues above the maximum theoretical value are left intact. \begin{aligned}\lambda^{NEW}_{1} = \lambda^{OLD}_{1}\end{aligned} \begin{aligned}\lambda^{NEW}_{2} = \lambda^{OLD}_{2}\end{aligned} \begin{aligned}\lambda^{NEW}_{3} = \lambda^{OLD}_{3}\end{aligned}
• The new set of eigenvalues with the set of eigenvectors is used to obtain the new de-noised correlation matrix. is the de-noised correlation matrix, is the eigenvectors matrix, and is the diagonal matrix with new eigenvalues.
• The new correlation matrix is then transformed back to the new de-noised covariance matrix.\begin{aligned}\tilde{C} = W \Lambda W\end{aligned}
• To rescale $\tilde{C}$ so that the main diagonal consists of 1s the following transformation is made. This is how the final $C_{denoised}$ is obtained.\begin{aligned}C_{denoised} = \tilde{C} [(diag[\tilde{C}])^\frac{1}{2}(diag[\tilde{C}])^{\frac{1}{2}’}]^{-1}\end{aligned}
• The new correlation matrix is then transformed back to the new de-noised covariance matrix.

The process of de-noising the covariance matrix is described in a paper by Potter M., J.P. Bouchaud, L. Laloux – Financial Applications of Random Matrix Theory: Old Laces and New Pieces.

# Setting the required parameters for de-noising

# Relation of number of observations T to the number of variables N (T/N)
tn_relation = stock_prices.shape[0] / stock_prices.shape[1]

# The bandwidth of the KDE kernel
kde_bwidth = 0.25

# Finding the Вe-noised Сovariance matrix
cov_matrix_denoised = risk_estimators.denoise_covariance(cov_matrix, tn_relation, kde_bwidth)

# Transforming De-noised Covariance from np.array to pd.DataFrame
cov_matrix_denoised = pd.DataFrame(cov_matrix_denoised, index=cov_matrix.index, columns=cov_matrix.columns)