CAL1_L20

Higher order derivatives

Let $f:{\mathbb{R}}^n\to \mathbb{R}$. Then, $f^{\prime }$ is a map from ${\mathbb{R}}^n$ to $\mathcal{L}({\mathbb{R}}^n,\mathbb{R})$:

$$\mathbf{x}\stackrel{f^{\prime }}{\mapsto }(f^{\prime }(\mathbf{x}):\mathbf{t}\stackrel{f^{\prime }(\mathbf{x})}{\mapsto }{f}^{\prime }(\mathbf{x})(\mathbf{t})).$$

Since $\mathcal{L}({\mathbb{R}}^n,\mathbb{R})\cong {\mathbb{R}}^n$, $f^{\prime }$ can be thought of as a map from ${\mathbb{R}}^n$ to ${\mathbb{R}}^n$. Now, let $\mathbf{g}\equiv f^{\prime }$. Then, ${\mathbf{g}}^{\prime }$ would be a map from ${\mathbb{R}}^n$ to $\mathcal{L}({\mathbb{R}}^n,{\mathbb{R}}^n)$:

$${\mathbf{g}}^{\prime }:\underset{\in {\mathbb{R}}^n}{x}\mapsto \left(\underset{\in \mathcal{L}({\mathbb{R}}^n,\mathcal{L}({\mathbb{R}}^n,\mathbb{R}))}{{\mathbf{g}}^{\prime }(\mathbf{x})}:\underset{\in {\mathbb{R}}^n}{{\mathbf{t}}_1}\mapsto \left(\underset{\in \mathcal{L}({\mathbb{R}}^n,\mathbb{R})}{{\mathbf{g}}^{\prime }(\mathbf{x})({\mathbf{t}}_1)}:\underset{\in {\mathbb{R}}^n}{{\mathbf{t}}_2}\mapsto \underset{\in \mathbb{R}}{{\mathbf{g}}^{\prime }(\mathbf{x})({\mathbf{t}}_1)({\mathbf{t}}_2)}\right)\right).$$

Remember that $\mathbf{g}(\mathbf{x}+{\mathbf{t}}_1)=\mathbf{g}(\mathbf{x})+{\mathbf{g}}^{\prime }(\mathbf{x})({\mathbf{t}}_1)+\Vert {\mathbf{t}}_1\Vert \mathbf{E}({\mathbf{t}}_1)$, so ${\mathbf{g}}^{\prime }(\mathbf{x})({\mathbf{t}}_1)$ is a linear approximation of the linear transformation $\mathbf{g}(\mathbf{x}+{\mathbf{t}}_1)-\mathbf{g}(\mathbf{x})$.

Definition 1(Definition).

Let $f:U\to \mathbb{R}$, where $U$ is an open subset of ${\mathbb{R}}^n$. Let $\mathbf{x}\in U$, and assume that $f^{\prime }(\mathbf{x})$ exists. Further, let $\mathbf{g}\equiv f^{\prime }$ and assume that ${\mathbf{g}}^{\prime }(\mathbf{x})$ exists. Then, the second order directional derivative $f^{\prime \prime }(\mathbf{x};\mathbf{t})$, $\mathbf{t}\in {\mathbb{R}}^n$, is defined to be ${\mathbf{g}}^{\prime }(\mathbf{x})(\mathbf{t})(\mathbf{t})$.

Lets obtain the expression for $f^{\prime \prime }(\mathbf{x},\mathbf{t})$. First, recall that

$$\mathbf{x}\stackrel{f^{\prime }}{\mapsto }\left[\begin{array}{ccc}{D}_{1}f(\mathbf{x})& \dots & D_{n}f(\mathbf{x})\end{array}\right].$$

Taking the derivative, we obtain

$$\begin{array}{r}{\mathbf{g}}^{\prime }:\mathbf{x}\mapsto \left({\mathbf{g}}^{\prime }(\mathbf{x}):\mathbf{t}\mapsto {\left(\left[\begin{array}{cccc}{D}_{1,1}f(\mathbf{x})& D_{2,1}f(\mathbf{x})& \dots & D_{n,1}f(\mathbf{x})\\ D_{1,2}f(\mathbf{x})& D_{2,2}f(\mathbf{x})& \dots & D_{n,2}f(\mathbf{x})\\ ⋮& ⋮& \ddots & ⋮\\ D_{1,n}f(\mathbf{x})& D_{2,n}f(\mathbf{x})& \dots & D_{n,n}f(\mathbf{x})\end{array}\right]\mathbf{t}\right)}^T\right)\end{array}$$

So,

$$\begin{array}{r}{\mathbf{g}}^{\prime }(\mathbf{x})(\mathbf{t})={\left[\begin{array}{c}{D}_{1,1}f(\mathbf{x})t_1+D_{2,1}f(\mathbf{x})t_2+\cdots +D_{n,1}f(\mathbf{x})t_n\\ D_{1,2}f(\mathbf{x})t_1+D_{2,2}f(\mathbf{x})t_2+\cdots +D_{n,2}f(\mathbf{x})t_n\\ ⋮\\ D_{1,n}f(\mathbf{x})t_1+D_{2,n}f(\mathbf{x})t_2+\cdots +D_{n,n}f(\mathbf{x})t_n\end{array}\right]}^T.\end{array}$$

It follows that

$$\begin{array}{r}{f}^{\prime \prime }(\mathbf{x};\mathbf{t})={\mathbf{g}}^{\prime }(\mathbf{x})(\mathbf{t})(\mathbf{t})=\sum _{j=1}^n\sum _{i=1}^n{D}_{i,j}f(\mathbf{x})t_i{t}_j.\end{array}$$

The third order directional derivative is similarly defined to the second order one, and the same procedure yields

$$\begin{array}{r}{f}^{\prime \prime \prime }(\mathbf{x};\mathbf{t})=\sum _{k=1}^n\sum _{j=1}^n\sum _{i=1}^n{D}_{i,j,k}f(\mathbf{x})t_i{t}_j{t}_k.\end{array}$$

The symbol $f^{(m)}(\mathbf{x},\mathbf{t})$ is similarly defined. Often, $f^{(m)}(\mathbf{x},\mathbf{t})$ is used to represent the corresponding algebraic expression when all the $m$th order partial derivatives are defined, even if $f^{(m-1)}$ is not differentiable at $\mathbf{x}$.

Taylor’s formula for functions from Rn to R

Theorem 2(Theorem).

Assume that $f$ and all its partial derivatives of order $<m$ are differentiable at each point of an open set $S\subseteq {\mathbb{R}}^n$. If $\mathbf{a}$ and $\mathbf{b}$ are two points of $S$ such that $L(\mathbf{a},\mathbf{b})\subseteq S$, then there is a point $\mathbf{z}$ on the line segment $L(\mathbf{a},\mathbf{b})$ such that

$$\begin{array}{r}f(\mathbf{b})=f(\mathbf{a})+\sum _{k=1}^{m-1}\frac{1}{k!}{f}^{(k)}(\mathbf{a};\mathbf{b}-\mathbf{a})+\frac{1}{m!}{f}^{(m)}(\mathbf{z};\mathbf{b}-\mathbf{a}).\end{array}$$

Proof
Since $S$ is open, there is a $\delta >0$ such that $\mathbf{a}+t(\mathbf{b}-\mathbf{a})\in S$ for all $t\in (-\delta ,1+\delta )$. Define $g$ on $(-\delta ,1+\delta )$ by

$$g(t)\equiv f(\mathbf{a}+t(\mathbf{b}-\mathbf{a})).$$

Then $f(\mathbf{b})-f(\mathbf{a})=g(1)-g(0)$. Note that $g^{(1)},\dots ,g^{(m-1)}$ are continuous on $(-\delta ,1+\delta )$, and $g^m$ is defined on $(-\delta ,1+\delta )$. By applying the one dimensional Taylor formula to $g$, we get

$$g(1)-g(0)=\sum _{k=1}^{m-1}\frac{1}{k!}{g}^{(k)}(0)+\frac{1}{m!}{g}^{(m)}(\theta ),(\ast )$$

where $0<\theta <1$.

Now $g$ is a composite function given by $g(t)=f(\mathbf{p}(t))$, where $\mathbf{p}(t)=\mathbf{a}+t(\mathbf{b}-\mathbf{a})$. From the chain rule, we have

$$\begin{array}{rl}{g}^{\prime }(t)& =\nabla f(\mathbf{p}(t))\cdot \mathbf{Dp}(t)\\ & =\sum _{j=1}^n{D}_{j}f(\mathbf{p}(t))(b_j-a_j)\\ & =f^{\prime }(\mathbf{p}(t);\mathbf{b}-\mathbf{a}).\end{array}$$

Again applying the chain rule,

$$\begin{array}{rl}{g}^{\prime \prime }(t)& =\nabla f^{\prime }(\mathbf{p}(t);\mathbf{b}-\mathbf{a})\cdot \mathbf{Dp}(t)\\ & =\sum _{i=1}^n\sum _{j=1}^n{D}_{i,j}f(\mathbf{p}(t))(b_j-a_j)(b_i-a_i)\\ & =f^{\prime \prime }(\mathbf{p}(t);\mathbf{b}-\mathbf{a}).\end{array}$$

Similarly, we find that $g^{(m)}(t)=f^{(m)}(\mathbf{p}(t);\mathbf{b}-\mathbf{a})$. Plugging these into $(\ast )$ proves the theorem, since $\mathbf{p}(\theta )=\mathbf{a}+\theta (\mathbf{b}-\mathbf{a})\in L(\mathbf{a},\mathbf{b})$.