理解covariance协方差 vs correlation coefficient相关系数

• covariance

From wikipedia, in probability theory and statistics, covariance is a measure of the joint variability of two random variables.

If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the lesser values, the covariance is positive.

In the opposite case, when the greater values of one variable mainly correspond to the lesser values of the other, the covariance is negative.

The sign of the covariance therefore shows the tendency in the linear relationship between the variables.

The magnitude of the covariance is not easy to interpret because it is not normalized and hence depends on the magnitudes of the variables.

The normalized version of the covariance, the correlation coefficient, however, shows by its magnitude the strength of the linear relation.

• Covariance and correlation

From wikipedia, in probability theory and statictics, the mathematical concepts of covariance and correlation are very similar.

Both describe the degree to which two random variables or sets of random variables tend to deviate from their expected values in similar ways.

If X and Y are two random variables, with means (expected values) ${\mu }_X$and${\mu }_Y$and standard deviations ${\sigma }_X$and${\sigma }_Y$,respectively, then their covariance and correlation are as follows:
$$\begin{gathered} covariance: \\ co{v}_{XY}={\sigma }_{XY}=E[(X-{\mu }_X)(Y-{\mu }_Y)] \end{gathered}$$

$$\begin{gathered} correlation: \\ cor{r}_{XY}={\rho }_{XY}=\frac{E[(X-{\mu }_X)(Y-{\mu }_Y)]}{{\sigma }_X{\sigma }_Y} \end{gathered}$$

so that:
$${\rho }_{XY}=\frac{{\sigma }_{XY}}{{\sigma }_X{\sigma }_Y}$$

• Demystifying the terms

Both terms measure the relationship and the dependency between two variables.

“Covariance” indicates the direction of the linear relationship between variables.

“Correlation” on the other hand measures both the strength and direction of the linear relationship between two variables.

Correlation is a function of the variance. What sets them apart is the fact that correlation values are standardized whereas, covariance values are not.

You can obtain the correlation coefficient of two variables by dividing the covariance of these variables by the product of the standard deviations of the same values.

If we revisit the definition of Standard Deviation, it essentially measures the absolute variability of a dataset’s distribution.

When you divide the covariance values by the standard deviation, it essentially scales the value down to a limited range of -1 to +1. This is precisely the range of the correlation values.

• Defining the Terms Mathematically

1. Covariance

   Covariance compares two variables in terms of the deviations from their mean (or expected) value.

   The covariance of two variables (x and y) can be represented as cov(x,y).

   If E[x] is the expected value or mean of a sample ‘x’, then cov(x,y) can be represented in the following way:
   $$\begin{gathered} cov(x,y)=E[(x-{\mu }_x)(y-{\mu }_y)] \\ =E[xy]-E[x]E[y] \\ =E[xy]-{\mu }_x{\mu }_y \end{gathered}$$

   $$\begin{gathered} cov(y,y)=E[(y-{\mu }_y)(y-{\mu }_y)] \\ var(y)=E[(y-{\mu }_y{)}^2]=s^2 \end{gathered}$$

   $$s^2=cov(x,x)=\frac{{\sum }_{i=1}^n(x_i-{\mu }_x{)}^2}{n-1}$$

   $$cov(x,y)=\frac{{\sum }_{i=1}^n(x_i-{\mu }_x)(y_i-{\mu }_y)}{n-1}$$

   numerator of (6) is called the sum of squared deviations;

   numerator of (7) is called the sum of cross products.

   $n$ is the number of samples in the data set.

   $n-1$ indicates the degree of freedom.

   E is the expectation and $\mu$ is the mean.

   degrees of freedom is the number of independent data point that went into calculating the estimate.

   A set with number n=3, and $\mu$ is 10, 给定2个数字，第三个数字是可以推断出来的，第三个是确定的不能随机自由的。因此自由度是2.

   A covariance of 0 indicates that two variables are totally unrelated.

   If the covariance is positive, the variables increase in the same direction, and if the covariance is negative, the variables change in oppsote directions.

2. Correlation

   Correlation is a normalization of covariance by the standard deviation of each variable.

   $$Corr(X,Y)=E[\frac{(X-{\mu }_x)}{{\sigma }_X}\frac{(Y-{\mu }_y)}{{\sigma }_Y}]$$

   $\sigma$ is the standard deviarion.

   The correlation coefficient is also known as the Pearson product-moment correlation coefficient, or Pearson’s correlation coefficient.

   It is obtained by dividing the covariance of the two variables by the product of their standard deviations.
   $$\begin{gathered} corr(x,y)=\frac{cov(x,y)}{s_x{s}_y}=\frac{E[(x-{\mu }_x)(y-{\mu }_y)]}{s_x{s}_y} \\ =\frac{E[(x-{\mu }_x)(y-{\mu }_y)]}{{\sigma }_x{\sigma }_y} \end{gathered}$$
   The values of the correlation coefficient can range from -1 to +1. The closer it is to +1 or -1, the more closely are the two variables are related.

   The positive sign signifies the direction of the correlation i.e. if one of the variables increase, and other variables is also supposed to increase.

• Data-matrix representation of Covariance and Correlation

Correlation is dimensionless, it is a unit-free measure of the relationship between variables. This is because we divide the value of covariance by the product of standard deviations which have the same units.

While correlation coefficients lie between -1 and +1, covariance can take any value between -uninfinite and +uninfinite.

The unit of covariance is a product of the units of the two variables.

Correlation is simply a normalized form of covariance and not affected by scale. Both covariance and correlation measure the linear relationship between variables but cannot be used interchangeably.

• Applications in Analytics

Now that we are done with mathematical theory, let us explore how and where it can be applied in the field of data analytics.

Correlation analysis, as a lot of analysts would know is a vital tool for feature selection and multivariate analysis in data preprocessing and exploration. Correlation helps us investigate and establish relationships between variabels. This is employed in feature selection before any kind of statistical modelling or data analysis.

You are advised to use the covariance matrix when the variable are on similar scales and the correlation matrix when the scales of the variables differ.

Covariance is used to determine how much two random variables vary together; whereas correlation is used to determine when a change in one variable can result in a change in another.

Covariance 只是用来表示两家的不同，没有情绪；

Correlation就是竞争了，看看我家买了新电视后，他家会不会换个新摩托车。For positive, A 买奥迪，B买宝马；For negative, A买奥迪，B卖宝马。Without relation, A买奥迪，B约妹子。

前者重在区别，后者重在联系。

When the correlation coefficient is positive, an increase in one variable also result in an increase in the other.

If we are comparing the relationship among three or more variables, it is better to use correlation because the value ranges or unit may cause false assumptions.

If we want to measure the relationship between X-Y and X-Z, using Covariance first, maybe get the result which show the covariance of X and Y is bigger than X and Z, but this may because the data range. Therefore, we need to use correlation to eliminate the effect of different value ranges.

Covariance 只能用来查看两个变量之间的deviation，不用用来对比两组数据之间的covariance值，因为值是unit相关的。

当对比三个或以上变量之间的区别，也就是存在至少两组Cov值的对比，就需要用Corr来实现无量纲化对比。

Cov是两个变量的对比，Corr是三个及以上变量（which means两个及以上Cov）

• Python Application

np.cov() returns the covariance matrix.

array([[0.07707877, 0.11350354],
	   [0.11350354, 0.18856815]])

The value at position [0,0] shows the covariance of X with itself,which means the variance;

The value at position [1,1] shows the covariance of Y with itself;

The covariance of X and Y is 0.11.

For the negative covariance example like :

array([[0.07707877, -0.11350354],
	   [-0.11350354, 0.18856815]])

For the independent variables :

array([[0.07707877, 0.00350354],
	   [0.00350354, 0.18856815]])

which has a covariance value that is close to zero.

This figure indicates the notion, Covariance describes how similarly two random variables deviate from their mean.

• References

1. Baffled by Covariance and Correlation??? Get the Math and the Application in Analytics for both the terms…
2. Covariance and Correlation
3. Covariance vs Correlation — Explained (Recommand)