PROB_L7

Expectation

Suppose $X$ is any discrete random variable having possible values $x_1,x_2,\dots$. We would like to define the expectation of $X$ as

$$EX=\sum _{j=1}^{\infty }{x}_{j}f(x_j).$$

If the support of $X$ is finite, we are good. In the general discrete case, the definition is valid only if the sum ${\sum }_j|x_j|f(x_j)$ is defined (basically, the sequence $(x_{j}f(x_j))$ must converge absolutely, to avoid Riemann rearrangement shenanigans). This leads to the following.

Definition 1(Definition).

Let $X$ be a discrete random variable having density $f$. If ${\sum }_j|x_j|f(x_j)<\infty$, we say that $X$ has finite expectation and we define its expectation by

$$EX=\sum _{j=1}^{\infty }{x}_{j}f(x_j).$$

On the other hand if ${\sum }_j|x_j|f(x_j)=\infty$, then we say $X$ does not have finite expectation and $EX$ is undefined.

It should be evident that $X$ has finite expectation iff $|X|$ has finite expectation.

Properties of expectation

Theorem 2(Property 1 (LOTUS)).

Let $\mathbf{X}$ be a discrete $r$-dimensional random vector having density $f$ and let $\phi$ be a real valued function on ${\mathbb{R}}^r$. Then, the random variable $Z=\phi (\mathbf{X})$ has finite expectation iff

$$\sum _{\mathbf{x}}|\phi (\mathbf{x})|f(\mathbf{x})<\infty .$$

If $Z$ has finite expectation,

$$EZ=\sum _{\mathbf{x}}\phi (\mathbf{x})f(\mathbf{x}).$$

Proof
Clearly,

$$\sum _j|z_j|f_Z(z_j)=\sum _{\mathbf{x}}|\phi (\mathbf{x})|f_{\mathbf{X}}(\mathbf{x}).$$

Thus, $Z$ has finite expectation iff ${\sum }_{\mathbf{x}}|\phi (\mathbf{x})|f(\mathbf{x})<\infty$. Similarly, it is clear that

$$\sum _j{z}_j{f}_Z(z_j)=\sum _{\mathbf{x}}\phi (\mathbf{x})f_{\mathbf{X}}(\mathbf{x}),$$

and the second part follows.

Theorem 3(Property 2).

Let $X$ and $Y$ be two random variables having finite expectation.

1. If $c$ is a constant and $P(X=c)=1$, then $EX=c$.
2. If $c$ is a constant, then $cX$ has finite expectation and $E(cX)=cEX$.
3. $X+Y$ has finite expectation and $E(X+Y)=EX+EY$.
4. Suppose $P(X\ge Y)=1$. Then $EX\ge EY$; moreover $EX=EY$ iff $P(X=Y)=1$.
5. $|EX|\le E|X|$.

Proof of 3

Let $\mathbf{W}=(X,Y)$. Let $\phi :{\mathbb{R}}^2\to \mathbb{R}$ be defined by $\phi (x,y)=x+y$. Then, $X+Y=\phi (\mathbf{W})$ has finite expectation if

$$\begin{array}{r}\sum _{\mathbf{w}}|\phi (\mathbf{w})|f_{\mathbf{W}}(\mathbf{w})\le \infty \end{array}$$

Now,

$$\begin{array}{rl}\sum _{x,y}|x+y|f_{\mathbf{W}}(x,y)& \le \sum _{x,y}|x|f_{\mathbf{W}}(x,y)+\sum _{x,y}|y|f(x,y)\\ & =\sum _x|x|\sum _y{f}_{\mathbf{W}}(x,y)+\sum _y|y|\sum _x{f}_{\mathbf{W}}(x,y)\\ & =\sum _x|x|f_{\mathbf{x}}(x)+\sum _y|y|f_{\mathbf{Y}}(y)\le \infty .\end{array}$$

Hence, $X+Y$ has finite expectation. From the definition of expectation, we get

$$\begin{array}{rl}E(X+Y)& =\sum _{x,y}(x+y)f(x,y)\\ & =\sum _{x,y}xf(x,y)+\sum _{x,y}yf(x,y)\\ & =EX+EY.\end{array}$$

Proof of 4
Let $Z=X-Y$.

$$\sum _{z}z{f}_Z(z)=EZ=E(X-Y)=EX-EY.$$

Since $P(Z\ge 0)=P(X\ge Y)=1$, $z>0$ for all $z$ in the support of $Z$. Thus, $EX-EY\ge 0$. If $EX=EY$, all $z$ in the support of $Z$ are forced to be $0$, so we get $P(Z=0)=P(X=Y)=1$.

Theorem 4(Property 3).

Let $X$ be a random variable such that for some constant $M$, $P(|X|\le M)=1$. Then $X$ has finite expectation and $|EX|\le M$.

Theorem 5(Property 4).

Let $X$ and $Y$ be two independent random variables having finite expectations. Then $XY$ has finite expectation and $E(XY)=(EX)(EY)$.

Proof
Since $X$ and $Y$ are independent, the joint density of $X$ and $Y$ is $f_X(x)f_Y(y)$. Thus

$$\begin{array}{rl}\sum _{x,y}|xy|f(x,y)& =\sum _{x,y}|x||y|f_X(x)f_Y(y)\\ & =\left(\sum _x|x|f_X(x)\right)\left(\sum _y|y|f_Y(y)\right)\le \infty .\end{array}$$

So, $XY$ has finite expectation. Using Property $1$, we can conclude that $E(XY)=(EX)(EY)$.

Theorem 6(Property 5).

Let $X$ be a nonnegative integer-valued random variable. Then $X$ has finite expectation iff the series ${\sum }_{x=1}^{\infty }P(X\ge x)$ converges. If the series does converge, then $EX={\sum }_{x=1}^{\infty }P(X\ge x)$.

Proof
Follows from the fact that

$$\sum _{x=1}^{\infty }xP(X=x)=\sum _{x=1}^{\infty }P(X\ge x).$$

---

Moments

Definition 7(Definition).

Let $X$ be a discrete random variable, and let $r\ge 0$ be an integer. We say that $X$ has a moment of order $r$ if $X^r$ has finite expectation. In that case we define the $r$th moment of $X$ as $E{X}^r$. If $X$ has a moment of order $r$, then the $r$th moment of $X-\mu$, where $\mu =EX$, is called the $r$th central moment of $X$.

$$E{X}^r=\sum _x{x}^{r}f(x);$$ $$E(X-\mu {)}^r=\sum _x(x-\mu {)}^{r}f(x).$$

Properties of Moments

Theorem 8(Property 1).

If $X$ has a moment of order $r$, then $X$ has a moment of order $k$ for all $k\le r$.

Proof
Let $k\le r$. For any $x$ in the support of $X$, it is always true that $|x{|}^k\le |x{|}^r+1$. Since ${\sum }_x(|x{|}^r+1)f(x)=E{X}^r+1$, it follows from the comparison test that $\sum |x{|}^{k}f(x)$ converges. Thus, $X^k$ has finite expectation.

Example 9(A random variable which does not have finite first moment).

Let $f$ be the function defined on $\mathbb{R}$ by

$$f(x)=\{\begin{array}{ll}\frac{1}{x(1+x)}& x=1,2,\dots \\ 0& \text{otherwise.}\end{array}$$

$f$ clearly satisfies the properties of a discrete density function. However, $f$ does not have finite expectation because ${\sum }_x|x|f(x)={\sum }_x\frac{1}{1+x}$ diverges.

Example 10(A random variable which has a finite moment of order$r$ but no higher finite moment).

We know that the series ${\sum }_{x\in \mathbb{N}}\frac{1}{x^k}$ converges for $k\ge 2$. Let the series ${\sum }_{x\in \mathbb{N}}\frac{1}{x^{r+2}}$ converge to $\mathcal{C}$. Let $X$ be a random variable with density

$$f(x)=\{\begin{array}{ll}\frac{1}{\mathcal{C}{x}^{r+2}}& x=1,2,\dots \\ 0& \text{otherwise.}\end{array}$$

Then,

$$E{X}^r=\sum _{x\in \mathbb{N}}\frac{1}{\mathcal{C}{x}^2}=\frac{{\pi }^2}{6\mathcal{C}}.$$

But, $E{X}^{r+1}$ and all higher moments are not finite.

We know that the $r$th central moment of $X$ exists given $X$ has a moment of order $r$ because of the following theorem.

Theorem 11(Property 2).

If the random variables $X$ and $Y$ have moments of order $r$, then $X+Y$ also has a moment of order $r$.

Variance

Definition 12(Definition).

If $X$ is a random variable having finite second moment, then the variance of $X$, denoted by $V(X)$, is defined to be the second central moment of $X$:

$$V(X)=E[(X-EX{)}^2].$$

The nonnegative number $\sigma =\sqrt{V(X)}$ is called the standard deviation of $X$ or $f_X$.

By expanding the right hand side, we get

$$V(X)=E{X}^2-(EX{)}^2.$$

Note that $\text{Var}(X)=0$ iff $X$ is a constant.

$EX$ and $\text{Var}(X)$ can be computed using the PGF for $X$:

$$EX={\Phi }_X^{\prime }(1),$$ $$\text{Var}(X)={\Phi }_X^{\prime \prime }(1)+{\Phi }_X^{\prime }(1)-({\Phi }_X^{\prime }(1){)}^2.$$

---

Variance of a sum

Let $X$ and $Y$ be two random variables each having finite second moment. Then $X+Y$ has finite second moment and hence finite variance. Also,

$$\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)+2E[(X-EX)(Y-EY)].$$

The quantity $E[(X-EX)(Y-EY)]$ is called the covariance of $X$ and $Y$. Thus,

$$\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)+2\text{Cov}(X,Y).$$

Further,

$$\text{Cov}(X,Y)=E(XY)-(EX)(EY).$$

From this, it is clear that $\text{Cov}(X,Y)=0$ when $X$ and $Y$ are independent (the converse is not true). So, if $X$ and $Y$ are independent, $\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)$.

More generally, if $X_1,\dots ,X_n$ are $n$ random variables each having a finite second moment, then

$$\text{Var}\left(\sum _{i=1}^n{X}_i\right)=\sum _{i=1}^n\text{Var}(X_i)+2\sum _{i=1}^{n-1}\sum _{j=i+1}^n\text{Cov}(X_i,X_j).$$

In particular, if $X_1,\dots ,X_n$ are mutually independent, then

$$\text{Var}\left(\sum _{i=1}^n{X}_i\right)=\sum _{i=1}^n\text{Var}(X_i).$$

In particular, if $X_1,\dots ,X_n$ are mutually independent random variables having a common variance ${\sigma }^2$ (for example, if they had the same density), then

$$\text{Var}(X_1+\cdots +X_n)=n\text{Var}(X_1)=n{\sigma }^2.$$

Another useful fact is that $\text{Var}(aX)=a^2\text{Var}(X)$.

The Schwarz inequality

Theorem 13(Theorem).

Let $X$ and $Y$ have finite second moments. Then,

$$[E(XY){]}^2\le (E{X}^2)(E{Y}^2).$$

Equality holds iff either $P(Y=0)=1$ or $P(X=aY)=1$ for some constant $a$.

Proof
If $P(Y=0)=1$ or $P(X=aY)=1$ equality holds trivially.

Sorry, I'm stupid (Not you, I am stupid)

If $P(X=aY)=1$, then

$$\begin{array}{rl}E(XY)& =\sum _{x,y}xyP(X=x,Y=y)\\ & =\sum _{y}y\sum _{x}xP(X=x,Y=y)\\ & =\sum _{y}y\left(ayP(X=ay,Y=y)+\sum _{x\ne ay}xP(X=x,Y=y)\right)\end{array}$$

Now, $\{X=x,Y=y\}$ where $x\ne ay$ is a subset of $\{X\ne aY\}$. $1=P(\{X=aY\}\bigsqcup \{X\ne aY\})=P(X=aY)+P(X\ne aY)$ so $P(X\ne aY)=0$. Now, $P(X=x,Y=y)\le P(X\ne aY)$ when $x\ne ay$ so it follows that $P(X=x,Y=y)=0$ when $x\ne ay$. Therefore,

$$\begin{array}{rl}E(XY)& =\sum _{y}a{y}^{2}P(X=ay,Y=y)\end{array}$$

Now, $P(Y=y)=P(Y=y,X=ay)+P(Y=y,X\ne ay)=P(Y=y,X=ay)$. Thus,

$$E(XY)=\sum _{y}a{y}^{2}P(Y=y)=aE(Y^2).$$

Also,

$$\begin{array}{rl}E(X^2)& =\sum _x{x}^{2}P(X=x)\\ & =\sum _{ay}(ay{)}^{2}P(X=ay)\\ & =\sum _{ay}(ay{)}^{2}P(Y=y)\\ & =a^{2}E(Y^2)\end{array}$$

The correlation coefficient

Applying the Schwarz inequality to the random variables $(X-EX)$ and $(Y-EY)$, we see that

$$(E[(X-EX)(Y-EY)]{)}^2\le [E(X-EX{)}^2][E(Y-EY{)}^2],$$

that is,

$$[\text{Cov}(X,Y){]}^2\le \text{Var}(X)\text{Var}(Y);$$ $$\text{Cov}(X,Y)\le \sqrt{\text{Var}(X)\text{Var}(Y)}.$$

For two random variables $X$ and $Y$ having finite non-zero variances, the quantity

$$\rho (X,Y)\equiv \frac{\text{Cov}(X,Y)}{\sqrt{\text{Var(X)}\text{Var}(Y)}}$$

is called the correlation coefficient of $X$ and $Y$. Clearly, $-1\le \rho (X,Y)\le 1$.

---

Chebyshev’s inequality

Let $X$ be a nonnegative random variable having finite expectation, and let $t$ be a positive real number. Definite the random variable $Y$ by setting $Y=0$ if $X<t$ and $Y=t$ if $X\ge t$. Then,

$$EY=tP(Y=t)+0P(Y=0)=tP(Y=t)=tP(X\ge t).$$

Now clearly, $X\ge Y$ and hence $EX\ge EY$. Thus
$EX\ge EY=tP(X\ge t)$ or

$$P(X\ge t)\le \frac{EX}{t}.$$

If we apply the the above inequality to the random variable $(X-\mu {)}^2$ and the number $t^2$, we get Chebyshev’s inequality:

$$P((X-\mu {)}^2\ge t^2)\le \frac{E(X-\mu {)}^2}{t^2}=\frac{\text{Var}(X)}{t^2}.$$

Theorem 14(Chebyshev's inequality).

Let $X$ be a random variable with mean $\mu$ and finite variance. Then for any real number $t>0$

$$P(|X-\mu |\ge t)\le \frac{\text{Var}(X)}{t^2}.$$

Chebyshev’s inequality gives an upper bound in terms of $\text{Var}(X)$ and $t$ for the probability that $X$ deviates from its mean by more than $t$ units.

---

Laws of large numbers

Weak law of large numbers

Let $X_1,\dots ,X_n$ be $n$ independent random variables having the same distribution. These random variables may be thought of as $n$ independent measurements of some quantity that is distributed according to their common distribution. Suppose that the common distribution of these random variables has finite mean $\mu$. Then for $n$ sufficiently large we would expect that their arithmetic mean $S_n/n=(X_1+\cdots +X_n)/n$ should be close to $\mu$. If the $X_i$ also have finite variance, then

$$\text{Var}\left(\frac{S_n}{n}\right)=\frac{n{\sigma }^2}{n^2}=\frac{{\sigma }^2}{n}$$

and thus $\text{Var}(S_n/n)\to 0$ as $n\to \infty$, that is, as $n$ gets large, the distribution $S_n/n$ becomes more concentrated about its mean $\mu$. More precisely, by applying Chebyshev’s inequality to $S_n/n$ we obtain

$$P\left(\left|\frac{S_n}{n}-\mu \right|\ge \delta \right)\le \frac{\text{Var}(S_n/n)}{{\delta }^2}=\frac{{\sigma }^2}{n{\delta }^2}.$$

It follows that for any $\delta >0$,

$$\underset{n\to \infty }{\lim }P\left(\left|\frac{S_n}{n}-\mu \right|\ge \delta \right)=0.$$

The number $\delta$ can be thought of as the desired accuracy in the approximation of $\mu$ by $S_n/n$. The above equation assures us that no matter how small $\delta$ may be, the probability that $S_n/n$ approximates $\mu$ to within this accuracy converges to $1$ as the number of observations gets large. This is called the Weak Law of Large Numbers.

Theorem 15(Weak law of large numbers).

Let $X_1,X_2,\dots ,X_n$ be independent random variables having a common distribution with finite expectation $\mu$ and let $S_n=X_1+\cdots +X_n$. Then for any $\delta >0$,

$$\underset{n\to \infty }{\lim }P\left(\left|\frac{S_n}{n}-\mu \right|\ge \delta \right)=0.$$

Strong law of large numbers

Definition 16(Definition).

We say that a sequence of random variables $\{Y_n{\}}_{n=1}^{\infty }$ converges in probability to a random variable $Y$ if for all $ϵ>0$, ${\lim }_{n\to \infty }P(|Y_n-Y|>ϵ)=0$.

Denote $S_n/n$ by ${\overline{S}}_n$. Note that the weak law states that the sequence of events $\{{\overline{S}}_n\}$ converges in probability to the constant random variable $\mu$.

Definition 17(Definition).

We say that a sequence of random variables $\{Y_n{\}}_{n=1}^{\infty }$ converges almost surely to a random variable $Y$ if ${\lim }_{n\to \infty }{Y}_n(\omega )=Y(\omega )$ for almost every $\omega$, that is, $P(\{\omega |{\lim }_{n\to \infty }{Y}_n(\omega )=Y(\omega )\})=1$.

For discrete random variables, this reduces to ${\lim }_{n\to \infty }{Y}_n(\omega )=Y(\omega )$ for all $\omega \in \Omega$.