LEC PROB 10

Distributions of sums and quotients

Distribution of X+Y

Let $X,Y$ be random variables (not necessarily independent) with joint distribution $f_{X,Y}$. We want to find $F_{X+Y}(z)$.

$$\begin{array}{rl}{F}_{X+Y}(z)& =P(X+Y\le z)\\ & ={\iint }_{\{x+y\le z\}}f(x,y)dxdy\\ & ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{z-x}f(x,y)dydx.\\ & ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{z}f(x,u-x)dudx\\ & ={\int }_{-\infty }^z\left({\int }_{-\infty }^{\infty }f(x,u-x)dx\right)du.\end{array}$$

Thus,

$$f_{X+Y}(u)={\int }_{-\infty }^{\infty }f(t,u-t)dt.$$

Additionally, if $X$ and $Y$ are independent, we have

$$\begin{array}{rl}{f}_{X+Y}(u)& ={\int }_{-\infty }^{\infty }{f}_X(t)f_Y(u-t)dt\\ & ={\int }_{-\infty }^{\infty }{f}_Y(t)f_X(u-t)dt.\end{array}$$

The above expression is analogous to the convolution product defined for densities of discrete random variables.

Example 238.1(Example).

Let $X,Y\sim \text{Exp}(\lambda )$.

$$\begin{array}{rl}{f}_{X+Y}(u)& ={\int }_0^u\lambda e^{-\lambda t}\lambda e^{-\lambda (u-t)}dt\\ & ={\lambda }^2{\int }_0^u{e}^{-\lambda u}dt\\ & ={\lambda }^{2}u{e}^{-\lambda u}.\end{array}$$

---

Tools

Dominated convergence theorem

We will use the DCT frequently in the upcoming proofs.

Theorem 238.2(DCT for sequences of sequences).

Let $f_n:\mathbb{N}\to \mathbb{R}$ be a sequence for $n\in N$. Assume a summable positive sequence $r:\mathbb{N}\to {\mathbb{R}}_{\ge 0}$ exists such that $|f_n(i)|\le r(i)$ for all $n$ and $i$, that is, $|f_n|\le r$ for all $n$. Let the sequence of sequences $(f_n)$ converge to a sequence $f$ pointwise, that is $f_n(i)\to f(i)$ for all $i$. Then, each $f_n$ is summable, $f$ is summable, and

$$\underset{n\to \infty }{\lim }\sum _{i=1}^{\infty }{f}_n(i)=\sum _{i=1}^{\infty }\underset{n\to \infty }{\lim }{f}_n(i)=\sum _{i=1}^{\infty }f(i).$$

0988fa

To put it simply, if a sequence of sequences is bounded by a summable sequence and converges pointwise to a sequence, then the limit of its sum is the sum of its limit. Here, summable means absolutely convergent. Note that the conclusion that each $f_n$ is summable follows from the hypothesis that it is bounded by a summable sequence.

Theorem 238.3(DCT for sequences of functions).

Let $f_n:\mathbb{R}\to \mathbb{R}$ be a measurable function for $n\in \mathbb{N}$. Assume an integrable positive function $r:\mathbb{R}\to [0,\infty )$ exists such that $|f_n|\le r$ for all $n$. Let the sequence of functions $(f_n)$ converge to a function $f$ pointwise. Then, $f_n$ is integrable, $f$ is integrable, and

$$\underset{n\to \infty }{\lim }{\int }_{-\infty }^{\infty }{f}_n(\mu )d\mu ={\int }_{-\infty }^{\infty }\underset{n\to \infty }{\lim }{f}_n(\mu )d\mu ={\int }_{-\infty }^{\infty }f(\mu )d\mu .$$

f84359

Here, integrable means Lebesgue integrable. Any measurable function that is absolutely dominated by an integrable function is integrable¹ (thus, the conclusion that each $f_n$ is integrable follows from the hypothesis that it is measurable and bounded by an integrable function).

Theorem 238.4(DCT for sequences of random variables).

Let $(X_n)$ be a sequence of random variables. Let $X$ be a random variable such that for every $\omega \in \Omega$, we have $X_n(\omega )\to X(\omega )$, that is, $(X_n)$ converges to $X$ pointwise. Assume there is an integrable random variable $Y$ such that $|X_n|\le Y$. Then,

$$E{X}_n\to EX.$$

Proof.

Treat each$X_n$, $X$ and $Y$ as measurable functions (they are measurable by the definition of a random variable)

$$X_n,X,Y:(\Omega ,\mathcal{F})\to (\mathbb{R},\mathcal{B}),$$

where $\mathcal{B}$ is the Borel $\sigma$-algebra on $\mathbb{R}$. The probability measure $P$ on $\Omega$ plays the role of the Lebesgue measure. We are given that $|X_n|\le Y$, and that

$$EY={\int }_{\Omega }Y(\omega )dP(\omega )<\infty .$$

So, we have all the hypotheses of the DCT, which enables us to write

$$\underset{n\to \infty }{\lim }E{X}_n=\underset{n\to \infty }{\lim }{\int }_{\Omega }{X}_n(\omega )dP(\omega )={\int }_{\Omega }X(\omega )dP(\omega )=EX.$$ □

A random variable is said to be integrable if it has finite expectation.

Fubini’s Theorem

Used to justify swapping integrals.

Theorem 238.5(Fubini).

For a function $g(t,y)$ defined on $\mathbb{R}\times \mathbb{R}$, if

$${\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }|g(t,y)|dydt<\infty$$

then the double integral equals the iterated integrals in either order.

---

Characteristic functions

Definition 238.6(Definition).

$X:\Omega \to \mathbb{C}$ is a complex random variable if $\mathrm{Re}X$ and $\mathrm{Im}X$ are both real random variables.

Definition 238.7(Definition).

Let $X$ be a complex random variable. $X$ has finite expectation if $\mathrm{Re}X$ and $\mathrm{Im}X$ have finite expectation, in which case we define

$$E(X)=E(\mathrm{Re}X)+iE(\mathrm{Im}X).$$

Note the following facts for real random variables:

1. $|E(X)|\le E|X|$.
2. If $X\le Y$, then $E(X)\le E(Y)$.

It is easy to verify that $E(\alpha X+Y)=\alpha E(X)+E(Y)$ for complex random variables $X,Y$ and $\alpha \in \mathbb{C}$.

Theorem 238.8(Theorem).

Let $X$ be a complex random variable. Then, $|E(X)|\le E(|X|)$.

Proof.

Since $E(X)=e^{i\theta }|E(X)|$ for some $\theta$, we have

$$\begin{array}{rl}|E(X)|& =e^{-i\theta }E(X)\\ & =\mathrm{Re}(e^{-i\theta }E(X))\\ & =\mathrm{Re}(E(e^{-i\theta }X))\\ & =E(\mathrm{Re}(e^{-i\theta }X))\end{array}$$

Now, $\mathrm{Re}(e^{-i\theta }X)\le |e^{-i\theta }X|=|X|$. Thus, we have

$$E(\mathrm{Re}(e^{-i\theta }X))\le E(|X|).$$ □

Note the following facts for all $z\in \mathbb{C}$, which are also easy to verify (just use the Taylor expansion for $e^z$):

1. $\frac{d}{dt}{e}^{zt}=z{e}^{zt}$,
2. $\int e^{zt}dt=e^{zt}/z$.

Definition 238.9(Definition).

Let $X:\Omega \to \mathbb{R}$ be a random variable. Define the characteristic function of $X$ by

$${\phi }_X(t)\equiv E{e}^{itX},t\in \mathbb{R}.$$

If $X$ is continuous, we have

$${\phi }_X(t)={\int }_{-\infty }^{\infty }{e}^{itx}{f}_X(x)dx.$$

It is clear that $|{\phi }_X|\le 1$.

Characteristic functions of common distributions

Let $X\sim \text{Unif}(a,b)$.

$$\begin{array}{rl}{\phi }_X(t)& ={\int }_a^b{e}^{itx}{f}_X(x)dx\\ & =\frac{e^{itb}-e^{itz}}{(b-a)it}.\end{array}$$

Let $X\sim \text{Exp}(\lambda )$.

$$\begin{array}{rl}{\phi }_X(t)& =\lambda {\int }_0^{\infty }{e}^{itx}{e}^{-\lambda x}dx\\ & =\frac{\lambda }{\lambda -it}.\end{array}$$

Let $X\sim n(0,1)$.

$$\begin{array}{rl}{\phi }_X(t)& =\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }{e}^{itx}{e}^{-x^2/2}dx\\ & =\frac{1}{\sqrt{2\pi }}\left({\int }_{-\infty }^{\infty }\cos (tx)e^{-x^2/2}dx+i\underset{=0}{\underbrace{{\int }_{-\infty }^{\infty }\sin (tx)e^{-x^2/2}dx}}\right)\\ & =\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }\cos (tx)e^{-x^2/2}dx\end{array}$$

From the dominated convergence theorem (one can take the dominating function $g$ to be $|x|e^{-x^2/2}$), we have

$$\begin{array}{rl}\frac{d}{dt}{\phi }_X(t)& =\frac{-1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }\sin (tx)x{e}^{-x^2/2}dx.\\ & =\frac{-1}{\sqrt{2\pi }}\left([-\sin (tx)e^{-x^2/2}{]}_{-\infty }^{\infty }+t{\int }_{-\infty }^{\infty }\cos (tx)e^{-x^2/2}dx\right)\\ & =\frac{-t}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }\cos (tx)e^{-x^2/2}dx.\end{array}$$

This yields a simple differential equation.

$$\begin{array}{rl}\frac{d}{dt}{\phi }_X(t)& =-t{\phi }_X(t)\\ \int \frac{1}{{\phi }_X(t)}d{\phi }_X(t)& =-\int tdt\\ {\phi }_X(t)& =c{e}^{-t^2/2}\end{array}$$

${\phi }_X(0)=1$ tells us that $c=1$. Thus,

$${\phi }_X(t)=e^{-t^2/2}.$$

If $Y=\mu +\sigma X$, then $Y\sim n(\mu ,{\sigma }^2)$.

$$\begin{array}{rl}{\phi }_Y(t)& =E{e}^{it(\mu +\sigma X)}\\ & =e^{\mu it}E{e}^{it\sigma X}\\ & =e^{\mu it}{\phi }_X(\sigma t)\\ & =e^{\mu it}{e}^{-{\sigma }^2{t}^2/2}\end{array}$$

Characteristic function of sum of independent random variables

Theorem 238.10(Theorem).

If $X$ and $Y$ are independent random variables, then

$${\phi }_{X+Y}(t)={\phi }_X(t){\phi }_Y(t)$$

Proof.

$$\begin{array}{rl}{\phi }_{X+Y}(t)& =E{e}^{it(X+Y)}\\ & =E{e}^{itX}{e}^{itY}\\ & =E{e}^{itX}E{e}^{itY}\\ & ={\phi }_X(t){\phi }_Y(t).\end{array}$$ □

Properties of characteristic functions

Property 0: ${\phi }_X(0)=1$, $|{\phi }_X(t)|\le 1$.

Theorem 238.11(Property 1).

A characteristic function is uniformly continuous.

Proof.

Let $\phi$ be the characteristic function of $X$.

$$\begin{array}{rl}|\phi (t+h)-\phi (t)|& =|E{e}^{itX}(e^{ihX}-1)|\\ & \le E|e^{itX}(e^{ihX}-1)|\\ & =E(|e^{itX}||e^{ihX}-1|)\\ & =E|e^{ihX}-1|\end{array}$$

From the dominated convergence theorem,

$$\begin{array}{rl}\underset{h\to 0}{\lim }E|e^{ihX}-1|& =\underset{h\to 0}{\lim }{\int }_{-\infty }^{\infty }|e^{ihX}-1|f_X(x)dx\\ & ={\int }_{-\infty }^{\infty }\underset{h\to 0}{\lim }|e^{ihX}-1|f_X(x)dx\\ & =0.\end{array}$$

Since $e^{ihX}\to 0$ as $h\to 0$, given $ϵ$, we can choose $\delta$ such that $E|e^{ihX}-1|<ϵ$ if $|h|<\delta$.□

Definition 238.12(Definition).

A function $\phi :\mathbb{R}\to \mathbb{C}$ is called positive definite if for all $z_1,\dots ,z_k\in \mathbb{C}$ and $t_1,\dots ,t_k\in \mathbb{R}$, $k\in \mathbb{N}$,

$$\sum _{i,j=1}^n{z}_i\overline{z_j}\phi (t_i-t_j)\ge 0.$$

Theorem 238.13(Property 2).

A characteristic function is positive definite.

Proof.

$$\begin{array}{rl}\sum _{i,j=1}^n{z}_i\overline{z_j}\phi (t_i-t_j)& =\sum _{i,j=1}^n{z}_i\overline{z_j}E{e}^{i{t}_{i}X-i{t}_{j}X}\\ & =\sum _{i,j=1}^n{z}_i\overline{z_j}E{e}^{i{t}_{i}X}\overline{e^{i{t}_{j}X}}\\ & =E\left(\sum _{i,j=1}^n{z}_i\overline{z_j}{e}^{i{t}_{i}X}\overline{e^{i{t}_{j}X}}\right)\\ & =E{\left|\sum _{i=1}^n{z}_i{e}^{i{t}_{i}X}\right|}^2\\ & \ge 0.\end{array}$$ □

Bochner’s Theorem

Note that for any distribution function $F$, there exists a random variable with distribution $F$.

Bochner’s Theorem claims that the properties listed in the previous section completely characterize characteristic functions.

Theorem 238.14(Bochner's Theorem).

If $\phi :\mathbb{R}\to \mathbb{C}$ satisfies

1. $\phi (0)=1$, $|\phi |\le 1$;
2. $\phi$ is continuous;
3. $\phi$ is positive definite;

Then there exists a distribution function $F$ such that if $X$ is a random variable with distribution $F$, ${\phi }_X=\phi$.

(continuity and positive definiteness together apparently imply uniform continuity.)

In other words, there exists a surjective map from the space of all distribution functions to the space of all functions satisfying the three listed properties (called characteristic functions from now on).

We will now prove that this map is injective.

Inverse theorem

Inverse theorem for integer valued random variables

Theorem 238.15(Theorem).

Let $X$ be an integer valued random variable. Let $f_X$ be the mass function of $X$, and let ${\phi }_X$ be the characteristic function of $X$. Then,

$$f_X(k)=\frac{1}{2\pi }{\int }_{-\pi }^{\pi }{e}^{-itk}{\phi }_X(t)dt.$$

Proof.

Compute:

$$\begin{array}{rl}& \frac{1}{2\pi }{\int }_{-\pi }^{\pi }{e}^{-itk}\left[\sum _{j=-\infty }^{\infty }{e}^{ijt}{f}_X(j)\right]dt\end{array}$$ $$\begin{array}{rl}& =\frac{1}{2\pi }\sum _{j=-\infty }^{\infty }{f}_X(j){\int }_{-\pi }^{\pi }{e}^{it(j-k)}dt\\ & =f_X(k)+\frac{1}{2\pi }\sum _{j\ne k}{f}_X(j)\underset{=0}{\underbrace{{\int }_{-\pi }^{\pi }{e}^{it(j-k)}dt}}\\ & =f_X(k).\end{array}$$ □

Justifying swapping the sum and integral

Now, we need to justify swapping the sum and the integral (see here for more). From Tonelli’s theorem, we have $\int \sum f_n=\sum \int f_n$ if $f_n\ge 0$ for all $n,x$, without any further conditions needed. Then Fubini’s theorem says that for general $f_n$, if $\sum \int |f_n|<\infty$ or $\int \sum |f_n|<\infty$ (by Tonelli the two are equivalent), then $\int \sum f_n=\sum \int f_n$. This can also be proven using DCT: Consider the functions

$$\begin{array}{rl}{f}_n(t)& =\sum _{j=-n}^n{e}^{it(j-k)}{f}_X(j)\text{ for }n\in \mathbb{N},\\ & \\ f(t)& =\sum _{j=-\infty }^{\infty }{e}^{it(j-k)}{f}_X(j).\end{array}$$

Clearly, $f_n\to f$ pointwise. Now,

$$\begin{array}{r}|f_n|\le \sum _{j=-n}^n|e^{-itk}{e}^{ijt}{f}_X(j)|=\sum _{j=-n}^n{f}_X(j)<1.\end{array}$$

The constant function $1$ is integrable on the bounded interval $[-\pi ,\pi ]$. Thus,

$$\begin{array}{rl}& {\int }_{-\pi }^{\pi }\sum _{j=-\infty }^{\infty }{e}^{it(j-k)}{f}_X(j)dt\\ & ={\int }_{-\pi }^{\pi }\underset{n\to \infty }{\lim }\sum _{j=-n}^n{e}^{it(j-k)}{f}_X(j)dt\\ & =\underset{n\to \infty }{\lim }{\int }_{-\pi }^{\pi }\sum _{j=-n}^n{e}^{it(j-k)}{f}_X(j)dt\\ & =\underset{n\to \infty }{\lim }\sum _{j=-n}^n{\int }_{-\pi }^{\pi }{e}^{it(j-k)}{f}_X(j)dt\\ & =\sum _{j=-\infty }^{\infty }{\int }_{-\pi }^{\pi }{e}^{it(j-k)}{f}_X(j)dt.\end{array}$$

Inverse theorem for discrete random variables

Theorem 238.16(Theorem).

Let $X$ be a discrete random variable with density $f_X$ and characteristic function ${\phi }_X$. Then,

$$f_X(x)=\underset{T\to \infty }{\lim }\frac{1}{2T}{\int }_{-T}^T{e}^{-itx}{\phi }_X(t)dt.$$

Proof.

Note that the support of $X$ is countable.

$$\begin{array}{rl}& \underset{T\to \infty }{\lim }\frac{1}{2T}{\int }_{-T}^T{e}^{-itx}{\phi }_X(t)dt\\ & =\underset{T\to \infty }{\lim }\frac{1}{2T}{\int }_{-T}^T{e}^{-itx}\left[\sum _{y\in \mathbb{R}}{e}^{ity}{f}_X(y)\right]dt\\ & =\underset{T\to \infty }{\lim }\frac{1}{2T}\sum _{y\in \mathbb{R}}{f}_X(y){\int }_{-T}^T{e}^{it(y-x)}dt\\ & =f_X(x)+\underset{T\to \infty }{\lim }\frac{1}{2T}\sum _{y\ne x}{f}_X(y){\int }_{-T}^T{e}^{it(y-x)}dt\\ & =f_X(x)+\underset{T\to \infty }{\lim }\sum _{y\ne x}\underset{|\cdot |\le f_X}{\underbrace{f_X(y)\frac{\sin (T(y-x))}{T(y-x)}}}\\ & =f_X(x)+\sum _{y\ne x}{f}_X(y)\underset{T\to \infty }{\lim }\frac{\sin (T(y-x))}{T(y-x)}\\ & =f_X(x).\end{array}$$ □

Inverse theorem for continuous random variables

Theorem 238.17(Theorem).

Let $X$ be a continuous random variable with continuous density $f$ and integrable characteristic function $\phi$ (${\int }_{-\infty }^{\infty }|\phi (t)|dt<\infty$). Then,

$$f(x)=\frac{1}{2\pi }{\int }_{-\infty }^{\infty }{e}^{-itx}\phi (t)dt.$$

Proof.

Notice that

$$\begin{array}{r}\underset{ϵ\to 0}{\lim }{\int }_{-\infty }^{\infty }\underset{|\cdot |\le |\phi (t)|}{\underbrace{e^{-{ϵ}^2{t}^2/2}{e}^{-itx}\phi (t)}}dt={\int }_{-\infty }^{\infty }{e}^{-itx}\phi (t)dt.\end{array}$$

Let’s compute the limit on the left.

$$\begin{array}{rl}& \underset{ϵ\to 0}{\lim }{\int }_{-\infty }^{\infty }{e}^{-{ϵ}^2{t}^2/2}{e}^{-itx}\left({\int }_{-\infty }^{\infty }{e}^{iyt}f(y)dy\right)dt\\ & =\underset{ϵ\to 0}{\lim }{\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{e}^{-{ϵ}^2{t}^2/2}{e}^{-itx}{e}^{iyt}f(y)dydt\end{array}$$

To use Fubini’s Theorem to swap the integrals, we must show that the integrand is absolutely integrable.

$$\begin{array}{rl}& {\iint }_{{\mathbb{R}}^2}|e^{-{ϵ}^2{t}^2/2}{e}^{-itx}{e}^{-iyt}f(y)|dydt\\ & ={\iint }_{{\mathbb{R}}^2}{e}^{-{ϵ}^2{t}^2/2}f(y)dydt\\ & =\left({\int }_{-\infty }^{\infty }{e}^{-{ϵ}^2{t}^2/2}dt\right)\left({\int }_{-\infty }^{\infty }f(y)dy\right)\\ & =\frac{\sqrt{2\pi }}{ϵ}.\end{array}$$

Now, we can swap those pesky integrals:

$$\begin{array}{r}\underset{ϵ\to 0}{\lim }{\int }_{-\infty }^{\infty }f(y){\int }_{-\infty }^{\infty }{e}^{-{ϵ}^2{t}^2/2}{e}^{it(y-x)}dtdy.\end{array}$$

Substitute $u=ϵt$.

$$\begin{array}{r}\underset{ϵ\to 0}{\lim }\frac{1}{ϵ}{\int }_{-\infty }^{\infty }f(y){\int }_{-\infty }^{\infty }{e}^{-u^2/2}{e}^{iu(y-x)/ϵ}dudy.\end{array}$$

If we dress the expression nicely, we’ll see that the inner integral is the characteristic function of the normal distribution evaluated at $(y-x)/ϵ$, which we have already computed:

$$\begin{array}{rl}& \underset{ϵ\to 0}{\lim }\frac{\sqrt{2\pi }}{ϵ}{\int }_{-\infty }^{\infty }f(y){\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{i\left(\frac{y-x}{ϵ}\right)u}{e}^{-u^2/2}dudy\\ & =\underset{ϵ\to 0}{\lim }\frac{\sqrt{2\pi }}{ϵ}{\int }_{-\infty }^{\infty }f(y){\phi }_{n(0,1)}\left(\frac{y-x}{ϵ}\right)dy\\ & =\underset{ϵ\to 0}{\lim }\frac{\sqrt{2\pi }}{ϵ}{\int }_{-\infty }^{\infty }f(y)e^{-(y-x{)}^2/2{ϵ}^2}dy.\end{array}$$

Substitute $v=(y-x)/ϵ$.

$$\begin{array}{rl}& \underset{ϵ\to 0}{\lim }\sqrt{2\pi }{\int }_{-\infty }^{\infty }f(ϵv+x)e^{-v^2/2}dv.\end{array}$$

Passing the limit inside is tricky; since $f$ may not be bounded and $ϵ$ does not vanish on taking absolute value, using the DCT directly is difficult. Instead, we use the DCT on an compact interval $[-R,R]$ (where $f$ is bounded) to show that in $[-R,R]$, we can take the limit inside. We then show that as we increase $R$, the integral on $[-R,R{]}^c$ vanishes (requires justification I don’t have time for now).

$$\begin{array}{r}\sqrt{2\pi }\underset{f\text{ bounded; DCT applicable}}{\underbrace{{\int }_{-R}^R\underset{ϵ\to 0}{\lim }f(ϵv+x)e^{-v^2/2}dv}}+\underset{\to 0\text{ as }R\to \infty }{\underbrace{\underset{ϵ\to 0}{\lim }{\int }_{[-R,R{]}^c}f(ϵv+x)e^{-v^2/2}dv}}.\end{array}$$

Thus, the expression becomes

$$\begin{array}{rl}& f(x)\sqrt{2\pi }{\int }_{-\infty }^{\infty }{e}^{-v^2/2}dv=f(x).\end{array}$$ □

3092aa

Footnotes

1. Note that there does not exist a similar property for Riemann integrals, that is, being absolutely dominated by a Riemann integrable function does not imply Riemann integrability. Even if we assume Riemann integrability in the hypothesis, we cannot conclude that the limit is Riemann integrable (here’s an example). The closest analogue of the DCT in Riemann land does away with the dominating function and requires the sequence of functions to converge uniformly instead. ↩