--- ### Covariance >[!quote] In a Nutshell >Captures the relationship between coupled [[Random Variable|random variables]] / dimensions of a multidimensional random variable. The sign of each entry encodes the linear relationship between the two components. A positive sign signals that greater / lesser values of the one variable tend to correspond to greater / lesser values in the other. A negative sign indicates that they behave opposed to each other, i.e. greater values for one tend to correspond to lesser values in the other. ![[Pasted image 20240516093856.png|center|400]] >[!info] Definition >The covariance is a measure of the joint variability of two or more [[Random Variable|random variables]]. It is greater than zero if on average, larger values of one variable correspond to larger values of the other one v.v..$\mathrm{Cov}[X,Y] = E[(X-E[X])(Y-E[Y])].$With more than two variables, the covariance is computed by the sum of all pairs. **Properties** - The total [[Covariance and Variance|variance]] of two [[Random Variable|random variables]] is $\text{Cov}[X+Y]={\text{Var}}[X]+\text{Var}[Y]+2\text{Cov}[X,Y].$ - The Covariance of two [[Statistical Independence|i.i.d.]] variables is zero $\text{Cov}[X,Y]=0.$However, if the covariance is zero the variables are not necessarily independent. --- >[!info] Covariance and Precision Matrix >The [[Matrices|matrix]] equivalent of the above is computed via $\boldsymbol{\Sigma}=\mathbb{E}[(\mathbf{x}-\mathbb{E}[\mathbf{x}])(\mathbf{x}-\mathbf{E}[\mathbf{x}])^{H}],$yielding a **symmetric [[Matrices|positive semidefinite matrix]]**. If the underlying [[Probability Distribution|distribution]] has zero mean, the formula is simplified to an expected outer product : $\mathbf{Q}=\mathbb{E}[\mathbf{xx^{H}}]=\mathbb{E}\Big[\begin{pmatrix}x_{1}x_{1}^{*} & x_{1}x_{2}^{*} & \dots & x_{1}x_{K}^{*} \\ x_{2}x_{1}^{*} & x_{2}x_{2}^{*} & \dots & \vdots \\ \vdots &\vdots & \ddots & \vdots \\ x_{K}x_{1}^{*} & x_{K}x_{2}^{*} & \dots & x_{K}x_{K}^{*} \end{pmatrix} \Big]$The inverse of the above covariance matrix is denoted the **precision matrix** $\mathbf{P}=\boldsymbol{\Sigma}^{-1}.$ --- ### Variance >[!info] Variance >When only considering a single [[Random Variable|random variable]], the variance is a measure of spread of a [[Probability Distribution|distribution]] around its [[Expectations|mean]]. It is defined as $\mathrm{Var}[X] = E[(X-E[X])^{2}] = E[X^{2}]-E[X]^{2}.$In the continuous case, it can be computed using $\text{Var}[X]= \int_\mathbb{R}x²\text{dF}(x)- \mu²$ - The **standard deviation** $\sigma$ is the square-root of the variance$\sigma = \sqrt{\text{Var}(X)}.$ --- #### Sample (Co-) Variance and Bessel's Correction ... needed for [[Statistic and Estimator|unbiased estimator]]