PROB_L9

Expectation of continuous random variables

Definition 1(Definition).

Let $X$ be a continuous random variable with density $f$. $X$ is said to have finite expectation if

$${\int }_{-\infty }^{\infty }|t|f(t)dt<\infty ,$$

in which case $E(X)$ is defined to be

$$E(X)={\int }_{-\infty }^{\infty }tf(t)dt.$$

Theorem 2(Proposition).

Let $X$ be a positive continuous random variable with distribution $F$ and density $f$. Then $X$ has finite expectation iff ${\int }_0^{\infty }P(X\ge x)dx<\infty$, in which case,

$$\begin{array}{rl}E(X)& ={\int }_0^{\infty }P(X\ge x)dx\\ & ={\int }_0^{\infty }(1-F(x))dx\end{array}$$

Proof.

This proof is not analytically rigorous.

$$\begin{array}{rl}{\int }_0^{\infty }(1-F(x))dx& ={\int }_0^{\infty }\left({\int }_x^{\infty }f(t)dt\right)dx\end{array}$$

You can think of the integral on the right as integrating the function $f(t,x)=f(t)$ on the region $\{(x,t):0\le x\le \infty ,x\le t\le \infty \}$. This region can also be expressed as $\{(x,t):0\le t\le \infty ,0\le x\le t\}$. Thus, the integral becomes

$$\begin{array}{r}{\int }_0^{\infty }\left({\int }_0^{t}f(t)dx\right)dt\end{array}={\int }_0^{\infty }tf(t)dt=E(X).$$ □

More generally, if $X$ is not positive,

$$EX={\int }_0^{\infty }(1-F(x))dx-{\int }_{-\infty }^{0}F(x)dx$$

(Yes, the sign ahead of the second integral in negative, not positive. Check your integration limits.)

Theorem 3(LOTUS).

Let $X$ be a continuous random variable with density $f$. Let $g:\mathbb{R}\to [0,\infty )$ be such that $g(X)$ is a continuous random variable. Then,

$$E(g(X))={\int }_{-\infty }^{\infty }g(x)f(x)dx.$$

Proof.

$$\begin{array}{rlr}E(g(X))& ={\int }_0^{\infty }(1-F_{g(X)}(u))du& \\ & ={\int }_0^{\infty }P(g(X)>u)du& \\ & ={\int }_0^{\infty }{\int }_B{f}_X(y)dydu& B=\{y|g(y)>u\}\\ & ={\int }_{-\infty }^{\infty }{\int }_0^{g(y)}{f}_X(y)dudy& \\ & ={\int }_{-\infty }^{\infty }g(y)f_X(y)dy& \end{array}$$ □

Vasanth's proof is incorrect!

Vasanth stated the theorem for a general function $g:\mathbb{R}\to \mathbb{R}$, and supplied the following proof:

Proof.

$$E(g(X))={\int }_0^{\infty }(1-F_{g(X)}(u))du-{\int }_{-\infty }^0{F}_{g(X)}(u)du$$ $$\begin{array}{rlr}& {\int }_0^{\infty }(1-F_{g(X)}(u))du& \\ & ={\int }_0^{\infty }P(g(X)>u)du& \\ & ={\int }_0^{\infty }{\int }_B{f}_X(y)dydu& B=\{y|g(y)>u\}\\ & =\boxed{{\int }_0^{\infty }{\int }_0^{g(y)}{f}_X(y)dudy}& \\ & ={\int }_0^{\infty }g(y)f_X(y)dy& \\ & & \\ & {\int }_{-\infty }^0{F}_{g(X)}(u)du& \\ & ={\int }_{-\infty }^{0}P(g(X)\le u)du& \\ & ={\int }_{-\infty }^0{\int }_B{f}_X(y)dydu& B=\{y|g(y)<u\}\\ & =\boxed{{\int }_{-\infty }^0{\int }_{g(y)}^0{f}_X(y)dudy}& \\ & =-{\int }_{-\infty }^{0}g(y)f_X(y)dy& \end{array}$$ □

The boxed steps are incorrect; the limits of the first integral in both cases must be $-\infty$ and $\infty$. The way to remedy this is to prove the theorem for a positive function $g$, and then to use the result $E(X)=E(X^{+})-E(X^{-})$ to prove the theorem for general $g$.

Note that Vasanth erred here on another point; if $g$ is form $\mathbb{R}$ to $[0,\infty )$, the integration bounds still remain $-\infty$ and $\infty$.

Example 4(Example).

Let $X\sim n(0,1)$. Clearly,

$${\int }_{-\infty }^{\infty }|t|e^{-t^2/2}dt=2<\infty .$$

Thus,

$$E(X)=\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }t{e}^{-t^2/2}dt=0,$$

since the integrand is an odd function.

Variance

We can now define variance to be $E[(X-\mu {)}^2]$, which simplifies to $E(X^2)-{\mu }^2$ (assume linearity of expectation for the moment).

Example 5(Example).

Let $X\sim n(0,1)$. $\text{Var}(X)=E(X^2)-\mu =E(X^2)$. Note that

$$\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }|t^2|e^{-t^2/2}dt=1<\infty .$$

Using the LOTUS property with $\phi (x)=x^2$ yields

$$\begin{array}{r}E(X^2)=\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }{t}^2{e}^{-t^2/2}dt=1.\end{array}$$

---

Moments of continuous random variables

Defined analogously to moments of discrete random variables.

Example 6(Examples).

If $X\sim \Gamma (\alpha ,\lambda )$,

$$\begin{array}{rl}E{X}^n& =\frac{{\lambda }^{\alpha }}{\Gamma (\alpha )}{\int }_0^{\infty }{x}^{n+\alpha -1}{e}^{-\lambda x}dx\\ & =\frac{{\lambda }^{\alpha }\Gamma (n+\alpha )}{\Gamma (\alpha ){\lambda }^{n+\alpha }}\\ & =\frac{(n+\alpha -1)!}{{\lambda }^n(\alpha -1)!}\end{array}$$

If $X\sim \text{Exp}(\lambda )$, $X\sim \Gamma (1,\lambda )$, so

$$E{X}^n=\frac{n!}{{\lambda }^n}.$$

If $X\sim n(0,{\sigma }^2)$, $X^2\sim \Gamma (1/2,1/2{\sigma }^2)$. Thus, all even moments of $X$ exist, and we can compute them via the gamma density. In a bit, we will show that If a continuous random variable has a moment of order $r$, then it has a moment of order $k$ for all $k\le r$, as we did for discrete random variables. It’ll then follow that all odd moments of $X$ also exist, and subsequently must be zero due to an odd integrand:

$${\int }_{-\infty }^{\infty }\stackrel{\text{odd}}{\overbrace{x^n\underset{\text{even}}{\underbrace{f_X(x)}}}}dx=0.$$

---

Joint distributions

We say the random vector $(X,Y)$ has density $f_{X,Y}$ if

$$F_{X,Y}(x,y)={\int }_{-\infty }^x{\int }_{-\infty }^y{f}_{X,Y}(x,y)dydx.$$

and

$${\iint }_{-\infty }^{\infty }{f}_{X,Y}(x,y)dydx=1.$$

Also, if $f_{X,Y}$ is continuous at $(x,y)$, then $F_{X,Y}$ is differentiable at $(x,y)$, and

$$\begin{array}{r}\left(\frac{\partial F_{X,Y}}{\partial x}\right)(x,y)={\int }_{-\infty }^y{f}_{X,Y}(x,y)dy,\\ \\ \left(\frac{\partial F_{X,Y}}{\partial y}\right)(x,y)={\int }_{-\infty }^x{f}_{X,Y}(x,y)dx,\\ \\ \left(\frac{{\partial }^2{F}_{X,Y}}{\partial x\partial y}\right)(x,y)=f_{X,Y}(x,y).\end{array}$$

Also note that

$$\begin{array}{r}\underset{x\to \infty }{\lim }{F}_{X,Y}(x,y)=F_Y(y),\\ \underset{y\to \infty }{\lim }{F}_{X,Y}(x,y)=F_X(x),\end{array}$$

and

$$\begin{array}{r}{\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)dx=f_Y(y),\\ {\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)dy=f_X(x),\end{array}$$

Example 7(Example).

Here’s now you compute normalization factors:

$$\begin{array}{rl}& {\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{e}^{-(x^2-xy+y^2)/2}dxdy\\ & ={\int }_{-\infty }^{\infty }{e}^{-3y^2/8}{\int }_{-\infty }^{\infty }{e}^{-(x-y/2{)}^2/2}dxdy\\ & =\sqrt{2\pi }{\int }_{-\infty }^{\infty }{e}^{-(\sqrt{3}y/2{)}^2/2}dy\\ & =\frac{4\pi }{\sqrt{3}}.\end{array}$$

Theorem 8(Theorem).

Let $X,Y$ be continuous random variables with joint distribution $f$. Let $g:{\mathbb{R}}^2\to [0,\infty )$ such that $g(X,Y)$ is a continuous random variable. Then,

$$E(g(X,Y))={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }g(x,y)f(x,y)dxdy.$$

Proof.

Let$F\equiv F_{g(X,Y)}$ and $f\equiv f_{X,Y}$, the joint density of $X$ and $Y$.

$$\begin{array}{rl}E(g(X,Y))& ={\int }_0^{\infty }(1-F(z))dz\\ & ={\int }_0^{\infty }P(g(X,Y)\ge z)dz\\ & ={\int }_0^{\infty }{\iint }_{B}f(x,y)dxdydz,B=\{(x,y)|g(x,y)\ge z\}\\ & ={\iint }_{{\mathbb{R}}^2}{\int }_0^{g(x,y)}f(x,y)dzdxdy\\ & ={\iint }_{{\mathbb{R}}^2}g(x,y)f(x,y)dxdy.\end{array}$$ □

We will now prove linearity of expectation for a specific case. The result will be used to prove linearity of expectation generally.

Theorem 9(Theorem).

Let $X$ be a continuous random variable with finite expectation. Define $X^{+}\equiv \max (X,0)$ and $X^{-}\equiv \max (-X,0)$ (Clearly, $X=X^{+}-X^{-}$ and $|X|=X^{+}+X^{-}$). Then, $E(X)=E(X^{+})-E(X^{-})$.

Proof.

Note that$X^{+}$ and $X^{-}$ are not continuous random variables - they have non zero density at $0$. However, since $0$ does not contribute to the expectation, we are (sort of) justified in writing

$$\begin{array}{rl}E(X^{+})& ={\int }_0^{\infty }P(X^{+}\ge x)dx\\ & ={\int }_0^{\infty }P(X\ge x)dx\\ & ={\int }_0^{\infty }(1-F_X(x))dx\\ & \\ & \\ E(X^{-})& ={\int }_0^{\infty }P(X^{-}\ge x)dx\\ & ={\int }_0^{\infty }P(X\le -x)dx\\ & ={\int }_0^{\infty }{F}_X(-x)dx\\ & ={\int }_{-\infty }^0{F}_X(x)dx\end{array}$$

It follows that

$$\begin{array}{rl}E(X^{+})-E(X^{-})& =E(X).\end{array}$$ □

So, if $g:{\mathbb{R}}^2\to \mathbb{R}$, we can write $E(g(X,Y))=E(g^{+}(X,Y))-E(g^{-}(X,Y))$, where $g^{+}\equiv \max (g,0)$ and $g^{-}\equiv \max (-g,0)$ are both non negative functions, and apply the previous theorem to obtain the general LOTUS property:

$$\begin{array}{rl}E(g(X,Y))& ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{g}^{+}(x,y)f(x,y)dxdy-{\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{g}^{-}(x,y)f(x,y)dxdy\\ & ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }g(x,y)f(x,y)dxdy.\end{array}$$

This allows us to prove the linearity of expectation.

Theorem 10(Linearity of expectation).

Let $X$ and $Y$ be continuous random variables with finite expectation. Then, $E(X+Y)=E(X)+E(Y)$.

Proof.

Let$g:{\mathbb{R}}^2\to \mathbb{R}$ be defined by $(x,y)\mapsto x+y$. Let $f$ be the joint density of $X$ and $Y$. Then,

$$\begin{array}{rl}& E(g(X,Y))=E(X+Y)\\ & ={\iint }_{{\mathbb{R}}^2}(x+y)f(x,y)dxdy\\ & ={\int }_{-\infty }^{\infty }x{\int }_{-\infty }^{\infty }f(x,y)dydx+{\int }_{-\infty }^{\infty }y{\int }_{-\infty }^{\infty }f(x,y)dxdy\\ & ={\int }_{-\infty }^{\infty }x{f}_X(x)dx+{\int }_{-\infty }^{\infty }y{f}_Y(y)dy\\ & =E(X)+E(Y).\end{array}$$ □

---

Independent continuous random variables

Definition 11(Definition).

Random variables $X$ and $Y$ are independent if the events $\{X\le x\}$ and $\{Y\le y\}$ are independent for all $x$ and $y$, that is, $F(x,y)=F_X(x)F_Y(y)$.

It follows from the definition that $X$ and $Y$ are independent iff $f_{X,Y}(x,y)=f_X(x)f_Y(y)$.

If $X$ and $Y$ are independent, it follows that

$$P(X\in [a,b],Y\in [c,d])=P(X\in [a,b])P(Y\in [c,d]).$$

This implies, that for any two “reasonable” (I suppose measurable) sets $A$ and $B$,

$$P(X\in A,Y\in B)=P(X\in A)P(Y\in B).$$

Example 12(Example).

Let the joint distribution of $(X,Y)$ be given by

$$f_{(X,Y)}(x,y)=\{\begin{array}{ll}1& (x,y)\in [0,1]\times [0,1]\\ 0& \text{otherwise.}\end{array}$$

We can compute $f_X$ and $f_Y$:

$$\begin{array}{rl}{f}_X(x)& ={\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)dy\\ & =\{\begin{array}{ll}1& x\in [0,1]\\ 0& \text{otherwise.}\end{array}\\ & \\ f_Y(y)& ={\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)dx\\ & =\{\begin{array}{ll}1& y\in [0,1]\\ 0& \text{otherwise.}\end{array}\end{array}$$

Notice that $f_{X,Y}(x,y)=f_X(x)f_Y(y)$. Thus, $X$ and $Y$ are independent.

Example 13(Example).

Let the joint distribution of $(X,Y)$ be given by

$$f_{(X,Y)}(x,y)=\{\begin{array}{ll}2& 0\le x\le 1,0\le y\le 1,x+y\le 1\\ 0& \text{otherwise.}\end{array}$$

Verify that this is indeed a density. Again, we can compute the marginals:

$$\begin{array}{rl}{f}_X(x)& =\{\begin{array}{ll}2(1-x)& x\in [0,1]\\ 0& \text{otherwise.}\end{array}\\ & \\ f_Y(y)& ={\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)dx\\ & =\{\begin{array}{ll}2(1-y)& y\in [0,1]\\ 0& \text{otherwise.}\end{array}\end{array}$$

Notice that $f_{X,Y}(x,y)\ne f_X(x)f_Y(y)$. $X$ and $Y$ are not independent.

Example 14(Example:$n$ dimensional standard Gaussian).

The two dimensional standard Gaussian (which just means normal, btw), which is denoted by $\mathcal{N}(0,I_{2\times 2})$, is defined like so:

$$\begin{array}{r}f(x,y)=\frac{1}{2\pi }{e}^{-(x^2+y^2)/2},(x,y)\in {\mathbb{R}}^{{}^2}.\end{array}$$

Its marginal densities are

$$\begin{array}{r}{f}_Y(y)=\frac{1}{2\pi }{e}^{-y^2/2}{\int }_{-oo}^{\infty }{e}^{-x^2/2}dx=\frac{1}{\sqrt{2\pi }}{e}^{-y^2/2}\sim n(0,1),\\ f_X(x)=\frac{1}{2\pi }{e}^{-x^2/2}{\int }_{-oo}^{\infty }{e}^{-y^2/2}dx=\frac{1}{\sqrt{2\pi }}{e}^{-x^2/2}\sim n(0,1).\end{array}$$

Note that $f(x,y)=f_X(x)f_Y(y)$, so if $(X,Y)\sim \mathcal{N}(0,I_{2\times 2})$, $X$ and $Y$ are independent.

Example 15(Example).

Let $X\sim \text{Exp}(\lambda )$, $Y\sim \text{Exp}(\mu )$, $X$ and $Y$ are independent. Find $P(X\le Y)$.

$$\begin{array}{rl}& P(X\le Y)=E(1_{\{X\le Y\}})\\ & ={\iint }_{\{(x,y)|x\le y\}}f(x,y)dydx\\ & ={\int }_0^{\infty }{\int }_0^{y}f(x,y)dxdy\\ & ={\int }_0^{\infty }f(y){\int }_0^{y}f(x)dxdy\\ & =\lambda \mu {\int }_0^{\infty }{e}^{-\lambda y}{\int }_0^y{e}^{-\mu x}dxdy\\ & =\lambda {\int }_0^{\infty }{e}^{-\lambda y}(1-e^{-\mu y})dy\\ & =\frac{\lambda }{\lambda +\mu }.\end{array}$$

If $X$ and $Y$ are independent and $f$ and $g$ are functions form $\mathbb{R}$ to $\mathbb{R}$, then $f(X)$ and $g(X)$ are independent.

Expectation of independent random variables

Theorem 16(Theorem).

$X$ and $Y$ are independent iff $E(XY)=E(X)(Y)$.

Proof.

If$X$ and $Y$ are independent, then

$$\begin{array}{rl}E(XY)& ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }xy{f}_{X,Y}(x,y)dxdy\\ & ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }xy{f}_X(X)f_Y(y)dxdy\\ & =\left({\int }_{-\infty }^{\infty }x{f}_X(x)dx\right)\left({\int }_{-\infty }^{\infty }y{f}_Y(y)dy\right)\\ & =E(X)E(Y).\end{array}$$

Conversely, if $E(XY)=E(X)(Y)$, observe that

$$\begin{array}{rl}{F}_{X,Y}(x,y)& =P(X\le x,Y\le y)\\ & =E{1}_{\{X\le x,Y\le y\}}\\ & =E(1_{\{X\le x\}}{1}_{\{Y\le y\}})\\ & =(E{1}_{\{X\le x\}})(E{1}_{\{Y\le y\}})\\ & =F_X(x)F_Y(y).\end{array}$$ □