PROB_L11

Convergence

Definition 1(Definition).

A sequence of random variables $\{X_n{\}}_{n=1}^{\infty }$ converges almost surely to a random variable $X$ if

$$P\{\omega |X_n(\omega )\to X(\omega )\}=1.$$

Denoted by $X_n\stackrel{\text{a.s}}{\to }X$.

Definition 2(Definition).

A sequence of random variables $\{X_n{\}}_{n=1}^{\infty }$ converges in probability to a random variable $X$ if for all $ϵ>0$,

$$\underset{n\to \infty }{\lim }P(|X_n-X|>ϵ)=0.$$

Denoted by $X_n\stackrel{\text{p}}{\to }X$.

Definition 3(Definition).

A sequence $\{X_n{\}}_{n=1}^{\infty }$ of random variables with distribution functions $\{F_n{\}}_{n=1}^{\infty }$ converges in distribution to a random variable $X$ with distribution function $F$ if for all $x\in \mathbb{R}$ at which $F$ is continuous,

$$\underset{n\to \infty }{\lim }{F}_n(x)=F(x).$$

Denoted by $X_n\stackrel{\text{d}}{\to }X$.

The limit of a sequence of random variables is almost surely (that is, bar a set of measure zero) unique for almost sure convergence and for convergence in probability. In other words, we have $X_n\stackrel{\text{a.s}}{\to }X$ or $X_n\stackrel{\text{p}}{\to }X$, then $X(w)$ is uniquely determined for all but a set of measure zero. This is not the case for convergence in distribution; take an iid sequence $\{X_n{\}}_{n=1}^{\infty }$ for example:

$$(X_n)\stackrel{\text{d}}{\to }{X}_k\text{for all }k\in \mathbb{N}.$$

Theorem 4(Theorem).

Almost sure convergence implies convergence in probability.

Proof.

If $\{X_n\}$ converges to $X$ almost surely, it means that the event $O=\{\omega |{\lim }_{n\to \infty }{X}_n(\omega )\ne X(\omega )\}$ has probability $0$. Fix $ϵ>0$. Consider the sequence of sets

$$A_n=\bigcup _{m\ge n}\{|X_m-X|>ϵ\}.$$

This sequence of sets is decreasing ($A_n\supseteq A_{n+1}\supseteq \dots$) towards the set

$$A_{\infty }=\bigcap _{n\ge 1}{A}_n.$$

Thus, we have

$$\underset{n\to \infty }{\lim }P(A_n)=P(A_{\infty }).$$

We shall now show that $P(A_{\infty })$ is $0$. For any point $\omega$ outside of $O$, we have ${\lim }_{n\to \infty }{X}_n=X(\omega )$, which implies that $|X_n(\omega )-X(\omega )|<ϵ$ for all $n\ge N$ for some $N$. In particular, for such $n$, the point $\omega$ will not lie in $A_n$, and hence won’t lie in $A_{\infty }$. Therefore, $A_{\infty }\subseteq O$, and so $P(A_{\infty })=0$.

Finally,

$$P(|X_n-X|>ϵ)\le P(A_n)\to 0,$$

which by definition means $X_n$ converges in probability to $X$.□

Theorem 5(Theorem).

Convergence in probability implies convergence in distribution.

Proof.

We want to show that

$$P(X_n\le x)\to P(X\le x).$$

Let $ϵ>0$.

$$\begin{array}{rl}P(X_n\le x)& =P(X_n\le x,X\le x+ϵ)+P(X_n\le x,X>x+ϵ)\\ & \le P(X\le x+ϵ)+P(|X_n-X|\ge ϵ)\\ & \\ P(X\le x-ϵ)& =P(X\le x-ϵ,X_n\le x)+P(X\le x-ϵ,X_n>x)\\ & \le P(X_n\le x)+P(|X_n-X|\ge ϵ)\end{array}$$

Combining the two inequalities, we get

$$P(X\le x-ϵ)-P(|X_n-X|\ge ϵ)\le P(X_n\le x)\le P(X\le x+ϵ)+P(|X_n-X|\ge ϵ).$$

Take limsup:

$$P(X\le x-ϵ)\le \underset{n\to \infty }{\mathrm{limsup}}P(X_n\le x)\le P(X\le x+ϵ).$$

Take liminf:

$$P(X\le x-ϵ)\le \underset{n\to \infty }{\mathrm{liminf}}P(X_n\le x)\le P(X\le x+ϵ).$$

Thus,

$$F(x-ϵ)\le \underset{n\to \infty }{\mathrm{liminf}}P(X_n\le x)\le \underset{n\to \infty }{\mathrm{limsup}}P(X_n\le x)\le F(x+ϵ).$$

If $F$ is continuous at $x$, ${\lim }_{ϵ\to 0}F(x-ϵ)={\lim }_{ϵ\to 0}F(x+ϵ)=F(x)$. Then, it follows that

$$\underset{n\to \infty }{\lim }P(X_n\le x)=F(x).$$ □

Note that $(F_n)\to F$, where $F_n$ and $F$ are distributions, does not imply $(f_n)\to f$. The converse is true, however.

Theorem 6(Theorem).

Let $F_n,F$ be distribution functions with densities $f_n$ and $f$. If $f_n\to f$ pointwise, then $F_n\to F$ pointwise.

Proof.

We want to prove that

$${\int }_{-\infty }^x{f}_{n}dx\to {\int }_{-\infty }^{x}fdx.$$

Now,

$$\left|{\int }_{-\infty }^x{f}_{n}dx-{\int }_{-\infty }^{x}fdx.\right|\le {\int }_{-\infty }^{\infty }|f-f_n|dx,$$

so it suffices to prove that the integral on the right converges to $0$ as $n\to \infty$. Since

$${\int }_{-\infty }^{\infty }{f}_{n}dx={\int }_{-\infty }^{\infty }fdx=1,$$

we have

$${\int }_{-\infty }^{\infty }(f-f_n)dx=0.$$

So,

$${\int }_{-\infty }^{\infty }(f-f_n{)}^{+}dx={\int }_{-\infty }^{\infty }(f-f_n{)}^{-}dx$$

and

$${\int }_{-\infty }^{\infty }|f-f_n|dx=2{\int }_{-\infty }^{\infty }(f-f_n{)}^{+}dx.$$

Now, note that $f-f_n\le f$, and $f$ is integrable. Thus, using DCT, we have

$$\begin{array}{r}\underset{n\to \infty }{\lim }2{\int }_{-\infty }^{\infty }(f-f_n{)}^{+}dx=0.\end{array}$$ □

Theorem 7(Lemma).

If $X_n^k\stackrel{\text{d}}{\to }{Y}^k$, $Y^k\stackrel{\text{d}}{\to }X$, and

$$\underset{k\to \infty }{\lim }\underset{n\to \infty }{\mathrm{limsup}}P(|X_n^k-X_n|\ge ϵ)=0,$$

then $X_n\stackrel{\text{d}}{\to }X$.

Proof.

We want to show that

$$P(X_n\le x)\to P(X\le x).$$

As before, we can bound:

$$F_n^k(x-ϵ)-P(|X_n^k-X_n|\ge ϵ)\le F_n(x)\le F_n^k(x+ϵ)+P(|X_n^k-X_n|\ge ϵ).$$

Take liminf on the left, limsup on the right as $n\to \infty$:

$$\begin{array}{rl}& \underset{n\to \infty }{\lim }{F}_n^k(x-ϵ)-\underset{n\to \infty }{\mathrm{limsup}}P(|X_n^k-X_n|\ge ϵ)\\ \le & \underset{n\to \infty }{\mathrm{liminf}}{F}_n(x)\le \underset{n\to \infty }{\mathrm{limsup}}{F}_n(x)\\ \le & \underset{n\to \infty }{\lim }{F}_n^k(x+ϵ)+\underset{n\to \infty }{\mathrm{limsup}}P(|X_n^k-X_n|\ge ϵ).\end{array}$$

Take limit as $k\to \infty$.

$$\begin{array}{rl}& F(x-ϵ)-\underset{k\to \infty }{\lim }\underset{n\to \infty }{\mathrm{limsup}}P(|X_n^k-X_n|\ge ϵ)\\ \le & \underset{n\to \infty }{\mathrm{liminf}}{F}_n(x)\le \underset{n\to \infty }{\mathrm{limsup}}{F}_n(x)\\ \le & F(x+ϵ)+\underset{k\to \infty }{\lim }\underset{n\to \infty }{\mathrm{limsup}}P(|X_n^k-X_n|\ge ϵ).\end{array}$$

Using the hypothesis,

$$\begin{array}{rl}& F(x-ϵ)\le \underset{n\to \infty }{\mathrm{liminf}}{F}_n(x)\le \underset{n\to \infty }{\mathrm{limsup}}{F}_n(x)\le F(x+ϵ).\end{array}$$

It follows that at all continuous points of $F$, ${\lim }_{n\to \infty }{F}_n(x)=F(x)$.□

Theorem 8(Theorem).

Let $(X_n)$ be a sequence of random variables with characteristic functions ${\phi }_n$. Then $X_n\stackrel{d}{\to }X$ iff ${\phi }_n\to \phi$ pointwise.

Proof.

Let$Z\sim n(0,1)$ be a random variable independent from $X_n$ and $X$. Let $Z_k=Z/k$. Then, $Z_k\sim n(0,\frac{1}{k^2})$. Define

$$\begin{array}{rl}{X}_n^k& =X_n+Z_k\\ Y^k& =X+Z_k\end{array}$$

Let $f_n^k$ be the density of $X_n^k$, and $f^k$ be the density of $Y^k$. Then, we have

$$\begin{array}{rl}{f}_n^k(x)& =\frac{1}{2\pi }{\int }_{-\infty }^{\infty }{e}^{-itx}{\phi }_n(t)e^{-t^2/2k^2}dt\\ f^k& =\frac{1}{2\pi }{\int }_{-\infty }^{\infty }{e}^{-itx}\phi (t)e^{-t^2/2k^2}dt\end{array}$$

If these densities are not continuous, we can arrive at these expressions using the convolution product:

$$\begin{array}{rl}{f}_n^k(x)& =\frac{k}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }{f}_n(y)e^{-(x-y{)}^2{k}^2/2}dy\\ & =\frac{k}{\sqrt{2\pi }}{\int }_{-\infty }^{\infty }{f}_n(y){\phi }_{kX}(y-x)dy\\ & =\frac{1}{2\pi }{\int }_{-\infty }^{\infty }{f}_n(y){\int }_{-\infty }^{\infty }{e}^{i(y-x)u}{e}^{-u^2/2k^2}dudy\\ & =\frac{1}{2\pi }{\int }_{-\infty }^{\infty }{e}^{-ixu}{e}^{-u^2/2k^2}{\int }_{-\infty }^{\infty }{e}^{iyu}{f}_n(y)dydu\\ & =\frac{1}{2\pi }{\int }_{-\infty }^{\infty }{e}^{-ixu}{e}^{-u^2/2k^2}{\phi }_n(u)du.\end{array}$$

From the dominated convergence theorem, we have $f_n^k\to f^k$ pointwise for each $k$ (the dominating function being $e^{-t^2/2k^2}$). We have shown that this implies $F_n^k\to F^k$ pointwise for each $k$, which by definition means $X_n^k\stackrel{\text{d}}{\to }{Y}^k$. Now,

$$\begin{array}{rl}\underset{k\to \infty }{\lim }P(|Y^k-X|\ge ϵ)& =\underset{k\to \infty }{\lim }P(|Z_k|\ge ϵ)\\ & =\underset{k\to \infty }{\lim }P(|Z|\ge kϵ)\\ & =0.\end{array}$$

Thus, $Y^k\stackrel{\text{p}}{\to }X$ as $k\to \infty$. This implies $Y^k\stackrel{\text{d}}{\to }X$.

Here, $P(|X_n^k-X_n|\ge ϵ)=P(|Z^k|\ge ϵ)=P(|Z|>kϵ)$. Thus,

$$\begin{array}{rlr}& \underset{k\to \infty }{\lim }\underset{n\to \infty }{\mathrm{limsup}}P(|X_n^k-X_n|\ge ϵ)& \\ & =\underset{k\to \infty }{\lim }P(|Z|\ge kϵ)=0.& \end{array}$$

The lemma is applicable, and we have $X_n\stackrel{\text{d}}{\to }X$.□