PROB_L10

Distributions of sums and quotients

Distribution of X+Y

Let $X,Y$ be random variables (not necessarily independent) with joint distribution $f_{X,Y}$. We want to find $F_{X+Y}(z)$.

$$\begin{array}{rl}{F}_{X+Y}(z)& =P(X+Y\le z)\\ & ={\iint }_{\{x+y\le z\}}f(x,y)dxdy\\ & ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{z-x}f(x,y)dydx.\\ & ={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{z}f(x,u-x)dudx\\ & ={\int }_{-\infty }^z\left({\int }_{-\infty }^{\infty }f(x,u-x)dx\right)du.\end{array}$$

Thus,

$$f_{X+Y}(u)={\int }_{-\infty }^{\infty }f(t,u-t)dt.$$

Additionally, if $X$ and $Y$ are independent, we have

$$\begin{array}{rl}{f}_{X+Y}(u)& ={\int }_{-\infty }^{\infty }{f}_X(t)f_Y(u-t)dt\\ & ={\int }_{-\infty }^{\infty }{f}_Y(t)f_X(u-t)dt.\end{array}$$

The above expression is analogous to the convolution product defined for densities of discrete random variables.

Example 1(Example).

Let $X,Y\sim \text{Exp}(\lambda )$.

$$\begin{array}{rl}{f}_{X+Y}(u)& ={\int }_0^u\lambda e^{-\lambda t}\lambda e^{-\lambda (u-t)}dt\\ & ={\lambda }^2{\int }_0^u{e}^{-\lambda u}dt\\ & ={\lambda }^{2}u{e}^{-\lambda u}.\end{array}$$

---

Tools

The dominated convergence theorem

We will use the DCT frequently in the coming proofs.

Theorem 2(DCT for sequences of sequences).

Let $f_n:\mathbb{N}\to \mathbb{R}$ be a sequence for $n\in N$. Assume a summable positive sequence $r:\mathbb{N}\to {\mathbb{R}}_{\ge 0}$ exists such that $|f_n(i)|\le r(i)$ for all $n$ and $i$, that is, $|f_n|\le r$ for all $n$. Let the sequence of sequences $(f_n)$ converge to a sequence $f$ pointwise, that is $f_n(i)\to f(i)$ for all $i$. Then, each $f_n$ is summable, $f$ is summable, and

$$\underset{n\to \infty }{\lim }\sum _{i=1}^{\infty }{f}_n(i)=\sum _{i=1}^{\infty }\underset{n\to \infty }{\lim }{f}_n(i)=\sum _{i=1}^{\infty }f(i).$$

To put it simply, if a sequence of sequences is bounded by a summable sequence and converges pointwise to a sequence, then the limit of its sum is the sum of its limit. Here, summable means absolutely convergent. Note that the conclusion that each $f_n$ is summable follows from the hypothesis that it is bounded by a summable sequence.

Theorem 3(DCT for sequences of functions).

Let $f_n:\mathbb{R}\to \mathbb{R}$ be a measurable function for $n\in \mathbb{N}$. Assume an integrable positive function $r:\mathbb{R}\to [0,\infty )$ exists such that $|f_n|\le r$ for all $n$. Let the sequence of functions $(f_n)$ converge to a function $f$ pointwise. Then, $f_n$ is integrable, $f$ is integrable, and

$$\underset{n\to \infty }{\lim }{\int }_{-\infty }^{\infty }{f}_n(\mu )d\mu ={\int }_{-\infty }^{\infty }\underset{n\to \infty }{\lim }{f}_n(\mu )d\mu ={\int }_{-\infty }^{\infty }f(\mu )d\mu .$$

Here, integrable means Lebesgue integrable. Any measurable function that is absolutely dominated by an integrable function is integrable¹ (thus, the conclusion that each $f_n$ is integrable follows from the hypothesis that it is measurable and bounded by an integrable function).

Theorem 4(DCT for sequences of random variables).

Let $(X_n)$ be a sequence of random variables. Let $X$ be a random variable such that for every $\omega \in \Omega$, we have $X_n(\omega )\to X(\omega )$, that is, $(X_n)$ converges to $X$ pointwise. Assume there is an integrable random variable $Y$ such that $|X_n|\le Y$. Then,

$$E{X}_n\to EX.$$

Proof.

Treat each$X_n$, $X$ and $Y$ as measurable functions (they are measurable by the definition of a random variable)

$$X_n,X,Y:(\Omega ,\mathcal{F})\to (\mathbb{R},\mathcal{B}),$$

where $\mathcal{B}$ is the Borel $\sigma$-algebra on $\mathbb{R}$. The probability measure $P$ on $\Omega$ plays the role of the Lebesgue measure. We are given that $|X_n|\le Y$, and that

$$EY={\int }_{\Omega }Y(\omega )dP(\omega )<\infty .$$

So, we have all the hypotheses of the DCT, which enables us to write

$$\underset{n\to \infty }{\lim }E{X}_n=\underset{n\to \infty }{\lim }{\int }_{\Omega }{X}_n(\omega )dP(\omega )={\int }_{\Omega }X(\omega )dP(\omega )=EX.$$ □

A random variable is said to be integrable if it has finite expectation.

Fubini’s Theorem

Used to justify swapping integrals.

Theorem 5(Fubini's Theorem).

For a function $g(t,y)$ defined on $\mathbb{R}\times \mathbb{R}$, if

$${\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }|g(t,y)|dydt<\infty$$

then the double integral equals the iterated integrals in either order.

---

Characteristic functions

Definition 6(Definition).

$X:\Omega \to \mathbb{C}$ is a complex random variable if $\mathrm{Re}X$ and $\mathrm{Im}X$ are both real random variables.

Definition 7(Definition).

Let $X$ be a complex random variable. $X$ has finite expectation if $\mathrm{Re}X$ and $\mathrm{Im}X$ have finite expectation, in which case we define

$$E(X)=E(\mathrm{Re}X)+iE(\mathrm{Im}X).$$

Note the following facts for real random variables:

1. $|E(X)|\le E|X|$.
2. If $X\le Y$, then $E(X)\le E(Y)$.

It is easy to verify that $E(\alpha X+Y)=\alpha E(X)+E(Y)$ for complex random variables $X,Y$ and $\alpha \in \mathbb{C}$.

Theorem 8(Theorem).

Let $X$ be a complex random variable. Then, $|E(X)|\le E(|X|)$.

Proof.

Since $E(X)=e^{i\theta }|E(X)|$ for some $\theta$, we have

$$\begin{array}{rl}|E(X)|& =e^{-i\theta }E(X)\\ & =\mathrm{Re}(e^{-i\theta }E(X))\\ & =\mathrm{Re}(E(e^{-i\theta }X))\\ & =E(\mathrm{Re}(e^{-i\theta }X))\end{array}$$

Now, $\mathrm{Re}(e^{-i\theta }X)\le |e^{-i\theta }X|=|X|$. Thus, we have

$$E(\mathrm{Re}(e^{-i\theta }X))\le E(|X|).$$ □

Note the following facts for all $z\in \mathbb{C}$, which are also easy to verify (just use the Taylor expansion for $e^z$):

1. $\frac{d}{dt}{e}^{zt}=z{e}^{zt}$,
2. $\int e^{zt}dt=e^{zt}/z$.

Definition 9(Definition).

Let $X:\Omega \to \mathbb{R}$ be a random variable. Define the characteristic function of $X$ by

$${\phi }_X(t)\equiv E{e}^{itX},t\in \mathbb{R}.$$

If $X$ is continuous, we have

$${\phi }_X(t)={\int }_{-\infty }^{\infty }{e}^{itx}{f}_X(x)dx.$$

It is clear that $|{\phi }_X|\le 1$.

Characteristic functions of common distributions

Let $X\sim \text{Unif}(a,b)$.

$$\begin{array}{rl}{\phi }_X(t)& ={\int }_a^b{e}^{itx}{f}_X(x)dx\\ & =\frac{e^{itb}-e^{itz}}{(b-a)it}.\end{array}$$

Let $X\sim \text{Exp}(\lambda )$.

$$\begin{array}{rl}{\phi }_X(t)& =\lambda {\int }_0^{\infty }{e}^{itx}{e}^{-\lambda x}dx\\ & =\frac{\lambda }{\lambda -it}.\end{array}$$

Let $X\sim n(0,1)$.

$$\begin{array}{rl}{\phi }_X(t)& =\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }{e}^{itx}{e}^{-x^2/2}dx\\ & =\frac{1}{\sqrt{2\pi }}\left({\int }_{-\infty }^{\infty }\cos (tx)e^{-x^2/2}dx+i\underset{=0}{\underbrace{{\int }_{-\infty }^{\infty }\sin (tx)e^{-x^2/2}dx}}\right)\\ & =\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }\cos (tx)e^{-x^2/2}dx\end{array}$$

From the dominated convergence theorem (one can take the dominating function $g$ to be $|x|e^{-x^2/2}$), we have

$$\begin{array}{rl}\frac{d}{dt}{\phi }_X(t)& =\frac{-1}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }\sin (tx)x{e}^{-x^2/2}dx.\\ & =\frac{-1}{\sqrt{2\pi }}\left([-\sin (tx)e^{-x^2/2}{]}_{-\infty }^{\infty }+t{\int }_{-\infty }^{\infty }\cos (tx)e^{-x^2/2}dx\right)\\ & =\frac{-t}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }\cos (tx)e^{-x^2/2}dx.\end{array}$$

This yields a simple differential equation.

$$\begin{array}{rl}\frac{d}{dt}{\phi }_X(t)& =-t{\phi }_X(t)\\ \int \frac{1}{{\phi }_X(t)}d{\phi }_X(t)& =-\int tdt\\ {\phi }_X(t)& =c{e}^{-t^2/2}\end{array}$$

${\phi }_X(0)=1$ tells us that $c=1$. Thus,

$${\phi }_X(t)=e^{-t^2/2}.$$

If $Y=\mu +\sigma X$, then $Y\sim n(\mu ,{\sigma }^2)$.

$$\begin{array}{rl}{\phi }_Y(t)& =E{e}^{it(\mu +\sigma X)}\\ & =e^{\mu it}E{e}^{it\sigma X}\\ & =e^{\mu it}{\phi }_X(\sigma t)\\ & =e^{\mu it}{e}^{-{\sigma }^2{t}^2/2}\end{array}$$

Characteristic function of sum of independent random variables

Theorem 10(Theorem).

If $X$ and $Y$ are independent random variables, then

$${\phi }_{X+Y}(t)={\phi }_X(t){\phi }_Y(t)$$

Proof.

$$\begin{array}{rl}{\phi }_{X+Y}(t)& =E{e}^{it(X+Y)}\\ & =E{e}^{itX}{e}^{itY}\\ & =E{e}^{itX}E{e}^{itY}\\ & ={\phi }_X(t){\phi }_Y(t).\end{array}$$ □

Properties of characteristic functions

Property 0: ${\phi }_X(0)=1$, $|{\phi }_X(t)|\le 1$.

Theorem 11(Property 1).

A characteristic function is uniformly continuous.

Proof.

Let $\phi$ be the characteristic function of $X$.

$$\begin{array}{rl}|\phi (t+h)-\phi (t)|& =|E{e}^{itX}(e^{ihX}-1)|\\ & \le E|e^{itX}(e^{ihX}-1)|\\ & =E(|e^{itX}||e^{ihX}-1|)\\ & =E|e^{ihX}-1|\end{array}$$

From the dominated convergence theorem,

$$\begin{array}{rl}\underset{h\to 0}{\lim }E|e^{ihX}-1|& =\underset{h\to 0}{\lim }{\int }_{-\infty }^{\infty }|e^{ihX}-1|f_X(x)dx\\ & ={\int }_{-\infty }^{\infty }\underset{h\to 0}{\lim }|e^{ihX}-1|f_X(x)dx\\ & =0.\end{array}$$

Since $e^{ihX}\to 0$ as $h\to 0$, given $ϵ$, we can choose $\delta$ such that $E|e^{ihX}-1|<ϵ$ if $|h|<\delta$.□

Definition 12(Definition).

A function $\phi :\mathbb{R}\to \mathbb{C}$ is called positive definite if for all $z_1,\dots ,z_k\in \mathbb{C}$ and $t_1,\dots ,t_k\in \mathbb{R}$, $k\in \mathbb{N}$,

$$\sum _{i,j=1}^n{z}_i\overline{z_j}\phi (t_i-t_j)\ge 0.$$

Theorem 13(Property 2).

A characteristic function is positive definite.

Proof.

$$\begin{array}{rl}\sum _{i,j=1}^n{z}_i\overline{z_j}\phi (t_i-t_j)& =\sum _{i,j=1}^n{z}_i\overline{z_j}E{e}^{i{t}_{i}X-i{t}_{j}X}\\ & =\sum _{i,j=1}^n{z}_i\overline{z_j}E{e}^{i{t}_{i}X}\overline{e^{i{t}_{j}X}}\\ & =E\left(\sum _{i,j=1}^n{z}_i\overline{z_j}{e}^{i{t}_{i}X}\overline{e^{i{t}_{j}X}}\right)\\ & =E{\left|\sum _{i=1}^n{z}_i{e}^{i{t}_{i}X}\right|}^2\\ & \ge 0.\end{array}$$ □

Bochner’s Theorem

Note that for any distribution function $F$, there exists a random variable with distribution $F$.

Bochner’s Theorem claims that the properties listed in the previous section completely characterize characteristic functions.

Theorem 14(Bochner's Theorem).

If $\phi :\mathbb{R}\to \mathbb{C}$ satisfies

1. $\phi (0)=1$, $|\phi |\le 1$;
2. $\phi$ is continuous;
3. $\phi$ is positive definite;

Then there exists a distribution function $F$ such that if $X$ is a random variable with distribution $F$, ${\phi }_X=\phi$.

(continuity and positive definiteness together apparently imply uniform continuity.)

In other words, there exists a surjective map from the space of all distribution functions to the space of all functions satisfying the three listed properties (called characteristic functions from now on).

We will now prove that this map is injective.

Inverse theorem

Inverse theorem for integer valued random variables

Theorem 15(Theorem).

Let $X$ be an integer valued random variable. Let $f_X$ be the mass function of $X$, and let ${\phi }_X$ be the characteristic function of $X$. Then,

$$f_X(k)=\frac{1}{2\pi }{\int }_{-\pi }^{\pi }{e}^{-itk}{\phi }_X(t)dt.$$

Proof.

Compute:

$$\begin{array}{rl}& \frac{1}{2\pi }{\int }_{-\pi }^{\pi }{e}^{-itk}\left[\sum _{j=-\infty }^{\infty }{e}^{ijt}{f}_X(j)\right]dt\end{array}$$ $$\begin{array}{rl}& =\frac{1}{2\pi }\sum _{j=-\infty }^{\infty }{f}_X(j){\int }_{-\pi }^{\pi }{e}^{it(j-k)}dt\\ & =f_X(k)+\frac{1}{2\pi }\sum _{j\ne k}{f}_X(j)\underset{=0}{\underbrace{{\int }_{-\pi }^{\pi }{e}^{it(j-k)}dt}}\\ & =f_X(k).\end{array}$$ □

Justifying swapping the sum and integral

Now, we need to justify swapping the sum and the integral (see here for more). From Tonelli’s theorem, we have $\int \sum f_n=\sum \int f_n$ if $f_n\ge 0$ for all $n,x$, without any further conditions needed. Then Fubini’s theorem says that for general $f_n$, if $\sum \int |f_n|<\infty$ or $\int \sum |f_n|<\infty$ (by Tonelli the two are equivalent), then $\int \sum f_n=\sum \int f_n$. This can also be proven using DCT: Consider the functions

$$\begin{array}{rl}{f}_n(t)& =\sum _{j=-n}^n{e}^{it(j-k)}{f}_X(j)\text{ for }n\in \mathbb{N},\\ & \\ f(t)& =\sum _{j=-\infty }^{\infty }{e}^{it(j-k)}{f}_X(j).\end{array}$$

Clearly, $f_n\to f$ pointwise. Now,

$$\begin{array}{r}|f_n|\le \sum _{j=-n}^n|e^{-itk}{e}^{ijt}{f}_X(j)|=\sum _{j=-n}^n{f}_X(j)<1.\end{array}$$

The constant function $1$ is integrable on the bounded interval $[-\pi ,\pi ]$. Thus,

$$\begin{array}{rl}& {\int }_{-\pi }^{\pi }\sum _{j=-\infty }^{\infty }{e}^{it(j-k)}{f}_X(j)dt\\ & ={\int }_{-\pi }^{\pi }\underset{n\to \infty }{\lim }\sum _{j=-n}^n{e}^{it(j-k)}{f}_X(j)dt\\ & =\underset{n\to \infty }{\lim }{\int }_{-\pi }^{\pi }\sum _{j=-n}^n{e}^{it(j-k)}{f}_X(j)dt\\ & =\underset{n\to \infty }{\lim }\sum _{j=-n}^n{\int }_{-\pi }^{\pi }{e}^{it(j-k)}{f}_X(j)dt\\ & =\sum _{j=-\infty }^{\infty }{\int }_{-\pi }^{\pi }{e}^{it(j-k)}{f}_X(j)dt.\end{array}$$

Inverse theorem for discrete random variables

Theorem 16(Theorem).

Let $X$ be a discrete random variable with density $f_X$ and characteristic function ${\phi }_X$. Then,

$$f_X(x)=\underset{T\to \infty }{\lim }\frac{1}{2T}{\int }_{-T}^T{e}^{-itx}{\phi }_X(t)dt.$$

Proof.

Note that the support of $X$ is countable.

$$\begin{array}{rl}& \underset{T\to \infty }{\lim }\frac{1}{2T}{\int }_{-T}^T{e}^{-itx}{\phi }_X(t)dt\\ & =\underset{T\to \infty }{\lim }\frac{1}{2T}{\int }_{-T}^T{e}^{-itx}\left[\sum _{y\in \mathbb{R}}{e}^{ity}{f}_X(y)\right]dt\\ & =\underset{T\to \infty }{\lim }\frac{1}{2T}\sum _{y\in \mathbb{R}}{f}_X(y){\int }_{-T}^T{e}^{it(y-x)}dt\\ & =f_X(x)+\underset{T\to \infty }{\lim }\frac{1}{2T}\sum _{y\ne x}{f}_X(y){\int }_{-T}^T{e}^{it(y-x)}dt\\ & =f_X(x)+\underset{T\to \infty }{\lim }\sum _{y\ne x}\underset{|\cdot |\le f_X}{\underbrace{f_X(y)\frac{\sin (T(y-x))}{T(y-x)}}}\\ & =f_X(x)+\sum _{y\ne x}{f}_X(y)\underset{T\to \infty }{\lim }\frac{\sin (T(y-x))}{T(y-x)}\\ & =f_X(x).\end{array}$$ □

Inverse theorem for continuous random variables

Theorem 17(Theorem).

Let $X$ be a continuous random variable with continuous density $f$ and integrable characteristic function $\phi$ (${\int }_{-\infty }^{\infty }|\phi (t)|dt<\infty$). Then,

$$f(x)=\frac{1}{2\pi }{\int }_{-\infty }^{\infty }{e}^{-itx}\phi (t)dt.$$

Proof.

Notice that

$$\begin{array}{r}\underset{ϵ\to 0}{\lim }{\int }_{-\infty }^{\infty }\underset{|\cdot |\le |\phi (t)|}{\underbrace{e^{-{ϵ}^2{t}^2/2}{e}^{-itx}\phi (t)}}dt={\int }_{-\infty }^{\infty }{e}^{-itx}\phi (t)dt.\end{array}$$

Let’s compute the limit on the left.

$$\begin{array}{rl}& \underset{ϵ\to 0}{\lim }{\int }_{-\infty }^{\infty }{e}^{-{ϵ}^2{t}^2/2}{e}^{-itx}\left({\int }_{-\infty }^{\infty }{e}^{iyt}f(y)dy\right)dt\\ & =\underset{ϵ\to 0}{\lim }{\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{e}^{-{ϵ}^2{t}^2/2}{e}^{-itx}{e}^{iyt}f(y)dydt\end{array}$$

To use Fubini’s Theorem to swap the integrals, we must show that the integrand is absolutely integrable.

$$\begin{array}{rl}& {\iint }_{{\mathbb{R}}^2}|e^{-{ϵ}^2{t}^2/2}{e}^{-itx}{e}^{-iyt}f(y)|dydt\\ & ={\iint }_{{\mathbb{R}}^2}{e}^{-{ϵ}^2{t}^2/2}f(y)dydt\\ & =\left({\int }_{-\infty }^{\infty }{e}^{-{ϵ}^2{t}^2/2}dt\right)\left({\int }_{-\infty }^{\infty }f(y)dy\right)\\ & =\frac{\sqrt{2\pi }}{ϵ}.\end{array}$$

Now, we can swap those pesky integrals:

$$\begin{array}{r}\underset{ϵ\to 0}{\lim }{\int }_{-\infty }^{\infty }f(y){\int }_{-\infty }^{\infty }{e}^{-{ϵ}^2{t}^2/2}{e}^{it(y-x)}dtdy.\end{array}$$

Substitute $u=ϵt$.

$$\begin{array}{r}\underset{ϵ\to 0}{\lim }\frac{1}{ϵ}{\int }_{-\infty }^{\infty }f(y){\int }_{-\infty }^{\infty }{e}^{-u^2/2}{e}^{iu(y-x)/ϵ}dudy.\end{array}$$

If we dress the expression nicely, we’ll see that the inner integral is the characteristic function of the normal distribution evaluated at $(y-x)/ϵ$, which we have already computed:

$$\begin{array}{rl}& \underset{ϵ\to 0}{\lim }\frac{\sqrt{2\pi }}{ϵ}{\int }_{-\infty }^{\infty }f(y){\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{i\left(\frac{y-x}{ϵ}\right)u}{e}^{-u^2/2}dudy\\ & =\underset{ϵ\to 0}{\lim }\frac{\sqrt{2\pi }}{ϵ}{\int }_{-\infty }^{\infty }f(y){\phi }_{n(0,1)}\left(\frac{y-x}{ϵ}\right)dy\\ & =\underset{ϵ\to 0}{\lim }\frac{\sqrt{2\pi }}{ϵ}{\int }_{-\infty }^{\infty }f(y)e^{-(y-x{)}^2/2{ϵ}^2}dy.\end{array}$$

Substitute $v=(y-x)/ϵ$.

$$\begin{array}{rl}& \underset{ϵ\to 0}{\lim }\sqrt{2\pi }{\int }_{-\infty }^{\infty }f(ϵv+x)e^{-v^2/2}dv.\end{array}$$

Passing the limit inside is tricky; since $f$ may not be bounded and $ϵ$ does not vanish on taking absolute value, using the DCT directly is difficult. Instead, we use the DCT on an compact interval $[-R,R]$ (where $f$ is bounded) to show that in $[-R,R]$, we can take the limit inside. We then show that as we increase $R$, the integral on $[-R,R{]}^c$ vanishes (requires justification I don’t have time for now).

$$\begin{array}{r}\sqrt{2\pi }\underset{f\text{ bounded; DCT applicable}}{\underbrace{{\int }_{-R}^R\underset{ϵ\to 0}{\lim }f(ϵv+x)e^{-v^2/2}dv}}+\underset{\to 0\text{ as }R\to \infty }{\underbrace{\underset{ϵ\to 0}{\lim }{\int }_{[-R,R{]}^c}f(ϵv+x)e^{-v^2/2}dv}}.\end{array}$$

Thus, the expression becomes

$$\begin{array}{rl}& f(x)\sqrt{2\pi }{\int }_{-\infty }^{\infty }{e}^{-v^2/2}dv=f(x).\end{array}$$ □

Footnotes

1. Note that there does not exist a similar property for Riemann integrals, that is, being absolutely dominated by a Riemann integrable function does not imply Riemann integrability. Even if we assume Riemann integrability in the hypothesis, we cannot conclude that the limit is Riemann integrable (here’s an example). The closest analogue of the DCT in Riemann land does away with the dominating function and requires the sequence of functions to converge uniformly instead. ↩