PROB_L8

Continuous random variables

What follows is the general definition of a random variable.

Definition 1(Definition).

Let $(\Omega ,\mathcal{A},P)$ be a probability space. A function $X:\Omega \to \mathbb{R}$ is a random variable if $\{X\le x\}\in \mathcal{A}$ for all $x\in \mathbb{R}$.

It follows from the definition that for all intervals of the form $(a,b]$, $a<b$, $X^{-1}(a,b]$ is in $\mathcal{A}$. Since $\mathcal{A}$ is closed under countable intersections, it follows that the preimages of all intervals of all types are in $\mathcal{A}$ (including singletons). Additionally, since every open set in $\mathbb{R}$ is an at most countable disjoint union of open intervals, the preimage of every open set is in $\mathcal{A}$. It follows that the preimage of every closed set is also in $\mathcal{A}$.

Note that this definition is compatible with our earlier definition of a discrete random variable.

Recall our definition of the distribution function $F$ of a random variable $X$ and its properties. In particular, we showed that $F(x+)-F(x-)=P(X=x)$. We motivate the definition of a continuous random variable by our desire for the distribution function to be continuous:

Definition 2(Definition).

A random variable $X$ is called a continuous random variable if $P(X=x)=0$ for all $x\in \mathbb{R}$.

Observe that $X$ is a continuous random variable iff its distribution function is continuous at every $x$.

---

Densities of continuous random variables

Definition 3(Definition).

A density function with respect to integration is a nonnegative integrable function $f$ such that

$${\int }_{-\infty }^{\infty }f(x)dx=1.$$

Note that if $f$ is a density function, then the function $F$ defined by

$$\begin{array}{r}F(x)={\int }_{-\infty }^{x}f(y)dy,x\in \mathbb{R}(1)\end{array}$$

is a continuous function (follows from a generalization of this theorem) satisfying all the properties of a distribution function. Thus, $F$ is a continuous distribution function. We say that this distribution function has density $f$. Further, if $f$ is continuous at a point $x_0\in \mathbb{R}$, then $F$ is differentiable at $x_0$, and $F^{\prime }(x_0)=f(x_0)$. In other words, $F^{\prime }=f$ where ever $f$ is continuous.

Given a distribution function $F$, we say that it admits a density if there exists a function $f:\mathbb{R}\to \mathbb{R}$ satisfying $(1)$. Note that if $F$ admits a density, $(1)$ does not uniquely determine $f$, as one can always change the value of $f$ at finitely many points and not affect the integral.

Given a distribution function $F$, consider the case when $F$ is differentiable. Then, $f=F^{\prime }$ satisfies $(1)$ due to the fundamental theorem of calculus (But isn’t that only on finite intervals?) (How do we know that $f$ is integrable here?).

Not all continuous distribution functions have densities. A continuous distribution function $F$ admits a density iff $F$ is absolutely continuous (ANA 3 stuff). (So does differentiability imply absolute continuity?)

If $X$ is a random variable having density $f$, then

$$P(a\le X\le b)={\int }_a^{b}f(x)dxa\le b.$$

Densities of functions of continuous random variables

Example 4(Example).

Let $X$ be a continuous random variable having density $f$. Let $Y=X^2$. What is the density of $Y$?

Let $F$ and $G$ denote the distributions of $X$ and $Y$. Then $G(y)=0$ for $y\le 0$. For $y>0$,

$$\begin{array}{rl}G(y)& =P(Y\le y)=P(X^2\le y)\\ & =P(-\sqrt{y}\le X\le \sqrt{y})\\ & =F(\sqrt{y})-F(-\sqrt{y}).\end{array}$$

Everything till this point is totally rigorous. Now, differentiate.

$$G^{\prime }(y)=\frac{1}{2\sqrt{y}}(F^{\prime }(\sqrt{y})+F^{\prime }(-\sqrt{y})).$$

Thus $Y=X^2$ has density $g$ given by

$$g=\{\begin{array}{ll}\frac{f(\sqrt{y})+f(-\sqrt{y})}{2\sqrt{y}}& y>0\\ 0& y\le 0.\end{array}$$

Note that $G$ and $F$ may not be differentiable at all points. To rigorously establish the validity of the above result, define $g$ as above, and integrate it to obtain $F(\sqrt{y})-F(-\sqrt{y})$ (subtleties: show that $g$ is a density function: non negativity and integrability. Showing that ${\int }_{-\infty }^{y}g(x)dx=F(\sqrt{y})-F(-\sqrt{y})$ shows that ${\int }_{-\infty }^{\infty }g(x)dx=1$ and that $g$ is indeed a density of $G$).

Example 5(Example).

Take $Y=\sigma X+\mu$, $\sigma ,\mu \in \mathbb{R}$ , $\sigma >0$ in the previous example. If we let $F$ and $G$ denote the distributions of $X$ and $Y$,

$$\begin{array}{rl}G(y)& =P(Y\le y)\\ & =P(\sigma X+\mu \le y)\\ & =P\left(X\le \frac{y-\mu }{\sigma }\right)\\ & =F\left(\frac{y-\mu }{\sigma }\right).\end{array}$$

Differentiate.

$$G^{\prime }(y)=\frac{1}{\sigma }{F}^{\prime }\left(\frac{y-\mu }{\sigma }\right).$$

The following theorem provides a general solution for some functions of $X$.

Theorem 6(Theorem).

Let $\varphi$ be a differentiable and strictly monotonic function on an interval $I$. Let $X$ be a continuous random variable having density $f$ such that $f(x)=0$ for $x\notin I$. Then $Y=\varphi (X)$ has density $g$ given by $g(y)=0$ for $y\notin \varphi (I)$ and

$$g(y)=f(x)\left|\frac{dx}{dy}\right|y\in \varphi (I)\text{and}x={\varphi }^{-1}(y).$$

proof

Common density functions

Symmetric densities

Definition 7(Definition).

A density function $f$ is called symmetric if $f(x)=f(-x)$ for all $x$.
A random variable $X$ is called symmetric if $X$ and $-X$ have the same distribution function.

Theorem 8(Theorem).

Let $X$ be a random variable that has a density. Then $X$ has a symmetric density iff $X$ is a symmetric random vairable.

We prove this for continuous random variables; the proof for discrete random variables is similar.

Proof of $⟹$
Let $X$ have a symmetric density $f$. Then,

$$\begin{array}{rl}P(-X\le x)& =P(X\ge -x)\\ & ={\int }_{-x}^{\infty }f(t)dt\\ & ={\int }_{-\infty }^{x}f(-t)dt\\ & ={\int }_{-\infty }^{x}f(t)dt\\ & =P(X\le x).\end{array}$$

Thus, $F_{-X}=F_X$.

Proof of $⟸$
Let $X$ be a symmetric random variable, that is, $X$ and $-X$ have the same distribution function. Now, if $g$ is a density of $X$, it follows that $g$ is also a density of $-X$. From the previous theorem, we have $g(x)=g(-x)$.

If a continuous distribution function $F$ has a symmetric density $f$, then $f(0)=1/2$. The values of negative $x$s can be calculated using the values of positive $x$s : $F(-x)=1-F(x)$.

Uniform density

Definition 9(Definition).

Let a and $b$ be constants with $a\le b$. The uniform density on the interval $(a,b)$ is the density $f$ defined by

$$f(x)=\{\begin{array}{ll}(b-a{)}^{-1}& a<x<b\\ 0& \text{otherwise}.\end{array}$$

Note that proving the existence of a uniformly distributed random variable requires measure theory.

Theorem 10(Proposition).

Let $X$ be a continuous random variable with differentiable strictly increasing distribution $F$ and density $f=F^{\prime }$. Then $F(X)$ and $1-F(X)$ have the uniform distribution $[0,1]$.

Proof
$f_{F(X)}(y)=0$ for $y$ not in $(0,1)$. If $y\in (0,1)$,

$$\begin{array}{rl}{f}_{F(X)}(y)& =f(F^{-1}(y))\left|\frac{d}{dy}{F}^{-1}(y)\right|\\ & =F^{\prime }(F^{-1}(y))\left|\frac{d}{dy}{F}^{-1}(y)\right|\\ & =\frac{d}{dy}F(F^{-1}(y))\\ & =1.\end{array}$$

The fact that $F^{-1}$ is an increasing function has been used.

This can also be proved without using the density (and thus without assuming $F$ is differentiable): for $y\in (0,1)$,

$$\begin{array}{rl}{F}_{F(X)}(y)& =P(F(X)\le y)\\ & =P(X\le F^{-1}(y))\\ & =F(F^{-1}(y))\\ & =y.\end{array}$$

It follows from monotonicity that if $y\le 0$, $F_{F(X)}(y)=0$, and if $y\ge 1$, $F_{F(X)}(y)=1$.

Next, let $F$ again be a continuous strictly increasing distribution function, and let $Y\sim \text{Unif}(0,1)$. Then, $F^{-1}(Y)$ is a random variable with distribution function $F$: for all $x\in \mathbb{R}$,

$$\begin{array}{rl}{F}_{F^{-1}(Y)}(x)& =P(F^{-1}(Y)\le x)\\ & =P(Y\le F(x))\\ & =F(x).\end{array}$$

Thus, given any continuous strictly increasing distribution function, there exists a random variable with that distribution.

Normal density

Definition 11(Definition).

The standard normal density is usually denoted by $\phi$, and is defined by

$$\phi (x)=\frac{1}{\sqrt{2\pi }}{e}^{-x^2/2}.$$

Its distribution is denoted by $\Phi$.

If $g$ is any non-negative function such that

$$0<{\int }_{-\infty }^{\infty }g(t)dt<\infty ,$$

Then $g$ can be normalized by dividing by the value of the above integral to yield a density function. For example, if $g(x)=e^{-x^2/2}$,

$$\begin{array}{rl}{\int }_{-\infty }^{\infty }{e}^{-x^2/2}dx& =\sqrt{{\int }_{-\infty }^{\infty }{e}^{-x^2/2}dx{\int }_{-\infty }^{\infty }{e}^{-y^2/2}dy}\\ & =\sqrt{{\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }{e}^{-(x^2+y^2)/2}dxdy}\\ & =\sqrt{{\int }_0^{\infty }{\int }_{-\pi }^{\pi }{e}^{-r^2/2}rdrd\theta }\\ & =\sqrt{2\pi {\int }_0^{\infty }r{e}^{-r^2/2}dr}\\ & =\sqrt{2\pi },\end{array}$$

so the function $e^{-x^2/2}/\sqrt{2\pi }$ is a density function. (Nobody seems to know a way to evaluate the above integral besides this whacky trick).

Let $X$ be a random variable having the standard normal density $\phi$ and let $Y=\mu +\sigma X$, where $\sigma >0$. Then, by the preceding example, $Y$ has the density $g$ given by

$$g(y)=\frac{1}{\sigma }\phi \left(\frac{y-\mu }{\sigma }\right)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{{}^{-(y-\mu {)}^2/2{\sigma }^2}}.$$

$g$ is called the normal density with mean $\mu$ and variance ${\sigma }^2$, and is denoted by $n(\mu ,{\sigma }^2)$. From the same example, we know that the distribution function of $Y$ is given by

$$P(Y\le y)=\Phi \left(\frac{y-\mu }{\sigma }\right).$$

So, If $Y$ is distributed as $n(\mu ,{\sigma }^2)$ and $a\le b$, then

$$P(a\le Y\le b)=\Phi \left(\frac{b-\mu }{\sigma }\right)-\Phi \left(\frac{a-\mu }{\sigma }\right).$$

If a random variable $Y$ is distributed as $n(\mu ,{\sigma }^2)$, then the random variable $a+bY$, $b\ne 0$ is distributed as

$$a+b(\mu +\sigma X)=(a+b\mu )+b\sigma X\sim n(a+b\mu ,b^2{\sigma }^2)$$

where $X\sim n(0,1)$.

Exponential density

Definition 12(Definition).

The exponential density with parameter $\lambda$ is the density $f$ defined by

$$f(x)=\{\begin{array}{ll}\lambda e^{-\lambda x}& x\ge 0\\ 0& x<0.\end{array}$$

The corresponding distribution function is

$$F(x)=\{\begin{array}{ll}1-e^{-\lambda x}& x\ge 0\\ 0& x<0.\end{array}$$

An important property of exponentially distributed random variables is the following:

$$P(X>a)P(X>b)=P(X>a+b),a\ge 0\text{ and }b\ge 0,$$

or, equivalently,

$$P(X\ge a+b|X>a)=P(X>b),a\ge 0\text{ and }b\ge 0.$$

This result is similar to the one obtained here for geometrically distributed random variables.

The above property actually characterizes the family of exponential distributions:

Theorem 13(Theorem).

Let $X$ be random variable such that $P(X>a)P(X>b)=P(X>a+b)$ holds for all $a,b\in \mathbb{R}$. Then either $P(X\ge 0)=0$ or $X$ is exponentially distributed.

proof

Cauchy density

Definition 14(Definition).

The Cauchy density is the density $f$ given by

$$f(x)=\frac{1}{\pi (1+x^2)},x\in \mathbb{R}.$$

Theorem 15(Proposition).

If $X\sim \text{Cauchy}$, then $\frac{1}{X}\sim \text{Cauchy}$.

It can be easily shown that if $X\sim \text{Cauchy}$, $X$ does not have finite expectation.

Gamma density

Consider functions $g$ of the form

$$g(x)=\{\begin{array}{ll}{x}^{\alpha -1}{e}^{-\lambda x}& x>0\\ 0& x\le 0.\end{array}$$

Here, we require $\alpha >0$ and $\lambda >0$ in order that $g$ be integrable (how do I show that $g$ is integrable under these conditions?). In normalizing $g$ to make it a density we must evaluate

$$c={\int }_0^{\infty }{x}^{\alpha -1}{e}^{-\lambda x}dx.$$

On making the change of variable $y=\lambda x$, we get

$$c=\frac{1}{{\lambda }^{\alpha }}{\int }_0^{\infty }{y}^{\alpha -1}{e}^{-y}dy.$$

There is no simple formula for this integral. Instead, it is used to define what is called the Gamma function:

Definition 16(Definition).

$$\Gamma (\alpha )\equiv {\int }_0^{\infty }{x}^{\alpha -1}{e}^{-x}dx,\alpha >0.$$

So, $c=\Gamma (\alpha )/{\lambda }^{\alpha }$. Note the formula

$${\int }_0^{\infty }{x}^{\alpha -1}{e}^{-\lambda x}dx=\frac{\Gamma (\alpha )}{{\lambda }^{\alpha }}.$$

The normalized function $g/c$ is called the gamma density with parameters $\alpha$ and $\lambda$, denoted by $\Gamma (\alpha ,\lambda )$.

Definition 17(Definition).

The gamma density is defined as follows.

$$\Gamma (x;\alpha ,\lambda )\equiv \{\begin{array}{ll}\frac{{\lambda }^{\alpha }}{\Gamma (\alpha )}{x}^{\alpha -1}{e}^{-\lambda x}& x>0\\ 0& x\le 0.\end{array}$$

Important

The exponential density is a special case of the gamma density. Specifically, the exponential density with parameter $\lambda$ is the same as the gamma density $\Gamma (1,\lambda )$.

Properties of the gamma function

$$\begin{array}{rl}& 1.\Gamma (1)=1\\ & 2.\Gamma (1/2)=\sqrt{\pi }\\ & 3.\Gamma (\alpha +1)=\alpha \Gamma (\alpha )\\ & 4.\Gamma (n)=(n-1)!,n\in \mathbb{N}\\ & 5.\Gamma \left(\frac{n}{2}\right)=\frac{\sqrt{\pi }(n-1)!}{2^{n-1}\left(\frac{n-1}{2}\right)!},n\in \mathbb{N},n\text{ is odd}\end{array}$$

---

Let $X\sim n(0,1)$. If $d$ is the density of $X^2$, using the formula we derived previously, we get $d$ to be

$$d(x)=\frac{1}{\sqrt{2\pi x}}{e}^{-x/2},x>0$$

Note that

$$d(x)=\frac{1}{\sqrt{2\pi }}g(x),$$

with $\alpha =1/2$ and $\lambda =1/2$. Since $d$ is a density, it follows that the normalization constant for $g$ is equal to $\sqrt{2\pi }$. Thus,

$$\begin{array}{rl}& \frac{1}{\sqrt{2\pi }}=\frac{{\lambda }^{\alpha }}{\Gamma (\alpha )}=\frac{1}{\Gamma (1/2)\sqrt{2}}\\ ⟹& \Gamma (1/2)=\sqrt{\pi }.\end{array}$$

---

The third property can be derived by integrating by parts:

$$\begin{array}{rl}\Gamma (\alpha +1)& ={\int }_0^{\infty }{x}^{\alpha }{e}^{-x}dx\\ & =-x^{\alpha }{e}^{-x}{|}_0^{\infty }+{\int }_0^{\infty }\alpha x^{\alpha -1}{e}^{-x}dx\\ & =\alpha \Gamma (\alpha ).\end{array}$$

There is no simple formula for the distribution function corresponding to $\Gamma (\alpha ,\lambda )$ except when $\alpha =m$ is a positive integer $\ge 2$, in which case

$$\begin{array}{rl}{\int }_0^x\frac{{\lambda }^m}{(m-1)!}{y}^{m-1}{e}^{-\lambda y}dy& ={\left[-\frac{(\lambda y{)}^{m-1}{e}^{-\lambda y}}{(m-1)!}\right]}_0^x+{\int }_0^x\frac{{\lambda }^{m-1}{y}^{m-2}{e}^{-\lambda y}}{(m-2)!}dy\\ & ={\int }_0^x\frac{{\lambda }^{m-1}{y}^{m-2}{e}^{-\lambda y}}{(m-2)!}dy-\frac{(\lambda x{)}^{m-1}{e}^{-\lambda x}}{(m-1)!}.\end{array}$$

On repeating $m-1$ times, we get

$${\int }_0^x\frac{{\lambda }^m}{(m-1)!}{y}^{m-1}{e}^{-\lambda y}dy=1-e^{-\lambda x}\sum _{i=0}^{m-1}\frac{(\lambda x{)}^i}{i!}.$$

This formula provides a connection between a random variable $X\sim \Gamma (m,\lambda )$ and a random variable $Y\sim \text{Poisn}(\lambda x)$ having a Poisson distribution with parameter $\lambda x$:

$$P(X\le x)=P(Y\ge m).$$

Expectation of the gamma distribution

Let $X\sim \Gamma (\alpha ,\lambda )$. Then,

$$\begin{array}{rl}\frac{{\lambda }^{\alpha }}{\Gamma (\alpha )}{\int }_0^{\infty }{x}^{\alpha }{e}^{-\lambda x}dx& =\frac{{\lambda }^{\alpha }}{\Gamma (\alpha )}\frac{\Gamma (\alpha +1)}{{\lambda }^{\alpha +1}}\\ & =\frac{\alpha }{\lambda }<\infty .\end{array}$$

Thus, $X$ has finite expectation, and $E(X)=\alpha /\lambda$.