CAL1_L16

Total derivatives

Let $\mathbf{f}:S\to {\mathbb{R}}^m$ be defined on $S\subseteq {\mathbb{R}}^n$ and let $\mathbf{c}\in S^{\circ }$.

Definition 1(Definition).

$\mathbf{f}$ is differentiable at $\mathbf{c}$ if there exists a linear transformation ${\mathbf{T}}_{\mathbf{c}}:{\mathbb{R}}^n\to {\mathbb{R}}^m$ such that

$$\underset{\mathbf{v}\to 0}{\lim }\frac{1}{\Vert \mathbf{v}\Vert }\left(\mathbf{f}(\mathbf{c}+\mathbf{v})-\mathbf{f}(\mathbf{c})-{\mathbf{T}}_{\mathbf{c}}(\mathbf{v})\right)=0,$$

in which case the total derivative of $\mathbf{f}$ at $\mathbf{c}$ is defined to be ${\mathbf{T}}_{\mathbf{c}}$. ${\mathbf{T}}_{\mathbf{c}}$ is also denoted by ${\mathbf{f}}^{\prime }(\mathbf{c})$.

Essentially, $\mathbf{f}$ is differentiable at $\mathbf{c}$ if there exists a linear function ${\mathbf{T}}_{\mathbf{c}}:{\mathbb{R}}^n\to {\mathbb{R}}^m$ such that

$$\mathbf{f}(\mathbf{c}+\mathbf{v})=\mathbf{f}(\mathbf{c})+{\mathbf{T}}_{\mathbf{c}}(\mathbf{v})+\Vert \mathbf{v}\Vert {\mathbf{E}}_{\mathbf{c}}(\mathbf{v}),$$

where ${\mathbf{E}}_{\mathbf{c}}(\mathbf{v})\to 0$ as $\mathbf{v}\to 0$. The above equation is called a first order Taylor formula.

Theorem 2(Theorem).

If $\mathbf{f}$ is differentiable at $\mathbf{c}$, then ${\mathbf{T}}_{\mathbf{c}}$ is uniquely determined.

Proof
Let $\mathbf{A}$ and ${\mathbf{A}}^{\prime }$ both satisfy the equation in the above definition. Then,

$$\underset{\mathbf{v}\to 0}{\lim }\frac{1}{\Vert \mathbf{v}\Vert }(\mathbf{A}-{\mathbf{A}}^{\prime })(\mathbf{v})=0.$$

For a fixed $\mathbf{v}$,

$$\underset{t\to 0}{\lim }\frac{1}{\Vert t\mathbf{v}\Vert }(\mathbf{A}-{\mathbf{A}}^{\prime })(t\mathbf{v})=\underset{t\to 0}{\lim }\frac{1}{\Vert \mathbf{v}\Vert }(\mathbf{A}-{\mathbf{A}}^{\prime })(\mathbf{v})=0,$$

so $(\mathbf{A}-{\mathbf{A}}^{\prime })(\mathbf{v})=0$ for every $\mathbf{v}$. It follows that $\mathbf{A}={\mathbf{A}}^{\prime }$.

Theorem 3(Theorem).

Assume $\mathbf{f}$ is differentiable at $\mathbf{c}$ with total derivative ${\mathbf{T}}_{\mathbf{c}}$. Then the directional derivative ${\mathbf{f}}^{\prime }(\mathbf{c};\mathbf{u})$ exists for every $\mathbf{u}\in {\mathbb{R}}^n$ and ${\mathbf{T}}_{\mathbf{c}}(\mathbf{u})={\mathbf{f}}^{\prime }(\mathbf{c};\mathbf{u})$.

Proof

$$\begin{array}{rl}{\mathbf{f}}^{\prime }(\mathbf{c};\mathbf{u})& =\underset{h\to 0}{\lim }\frac{f(\mathbf{c}+h\mathbf{u})-\mathbf{f}(\mathbf{c})}{h}\\ & =\underset{h\to 0}{\lim }\frac{{\mathbf{T}}_{\mathbf{c}}(h\mathbf{u})+\Vert h\mathbf{u}\Vert {\mathbf{E}}_{\mathbf{c}}(h\mathbf{u})}{h}\\ & ={\mathbf{T}}_{\mathbf{c}}(\mathbf{u}).\end{array}$$

Theorem 4(Theorem).

If $\mathbf{f}$ is differentiable at $\mathbf{c}$, then $\mathbf{f}$ is continuous at $\mathbf{c}$.

Proof
Since $\mathbf{c}\in S^{\circ }$, we need to show ${\lim }_{\mathbf{x}\to \mathbf{c}}\mathbf{f}(\mathbf{x})=\mathbf{f}(\mathbf{c})$. This is clear, since

$$\begin{array}{rl}\underset{\mathbf{v}\to 0}{\lim }\mathbf{f}(\mathbf{c}+\mathbf{v})& =\underset{\mathbf{v}\to 0}{\lim }(\mathbf{f}(\mathbf{c})+{\mathbf{T}}_{\mathbf{c}}(\mathbf{v})+\Vert \mathbf{v}\Vert {\mathbf{E}}_{\mathbf{c}}(\mathbf{v}))\\ & =\mathbf{f}(\mathbf{c}).\end{array}$$

Total derivatives in terms of partial derivatives

Since the total derivative is a linear transformation, it can be represented as left multiplication by a matrix. Let $\mathbf{f}:S\to {\mathbb{R}}^m$, $S\subseteq {\mathbb{R}}^n$, be differentiable at $\mathbf{c}\in S^{\circ }$. As we have previously noted, ${\mathbf{T}}_{\mathbf{c}}({\mathbf{e}}_k)={\mathbf{f}}^{\prime }(\mathbf{c};{\mathbf{e}}_k)=D_k\mathbf{f}(\mathbf{c})$. Thus, the matrix of ${\mathbf{T}}_{\mathbf{c}}$ is given by

$$\begin{array}{rl}[{\mathbf{T}}_{\mathbf{c}}]& =\left[\begin{array}{cccc}{D}_1\mathbf{f}(\mathbf{c})& D_2\mathbf{f}(\mathbf{c})& \dots & D_n\mathbf{f}(\mathbf{c})\end{array}\right]\\ & \\ & =\left[\begin{array}{cccc}{D}_1{f}_1(\mathbf{c})& D_2{f}_1(\mathbf{c})& \dots & D_n{f}_1(\mathbf{c})\\ D_1{f}_2(\mathbf{c})& D_2{f}_2(\mathbf{c})& \dots & D_n{f}_2(\mathbf{c})\\ ⋮& ⋮& \ddots & ⋮\\ D_1{f}_m(\mathbf{c})& D_2{f}_m(\mathbf{c})& \dots & D_n{f}_m(\mathbf{c})\end{array}\right],\end{array}$$

and ${\mathbf{T}}_{\mathbf{c}}(\mathbf{v})=[{\mathbf{T}}_{\mathbf{c}}]\mathbf{v}$. $[{\mathbf{T}}_{\mathbf{c}}]$ is denoted by $\mathbf{Df}(\mathbf{c})$, and is called the Jacobian matrix.

The $k$th row of the Jacobian matrix is a vector in ${\mathbb{R}}^n$ called the gradient vector of $f_k$, denoted by $\nabla f_k(\mathbf{c})$. When $m=1$, $\mathbf{D}f(\mathbf{c})=\nabla f(\mathbf{c}{)}^{⊺}$, and ${\mathbf{f}}^{\prime }(\mathbf{c};\mathbf{u})=\nabla f(\mathbf{c})\cdot \mathbf{u}$. More generally, we have

$$\begin{array}{r}{\mathbf{f}}^{\prime }(\mathbf{c})(\mathbf{v})=\sum _{k=1}^m(\nabla f_k(\mathbf{c})\cdot \mathbf{v}){\mathbf{e}}_k.\end{array}$$

Bounding the total derivative

Important

The above equation yields

$$\begin{array}{r}\Vert {\mathbf{f}}^{\prime }(\mathbf{c})(\mathbf{v})\Vert =\Vert \sum _{k=1}^m(\nabla f_k(\mathbf{c})\cdot \mathbf{v}){\mathbf{e}}_k\Vert \le \sum _{k=1}^m|\nabla f_k(\mathbf{c})\cdot \mathbf{v}|\le \Vert \mathbf{v}\Vert \sum _{k=1}^m\Vert \nabla f_k(\mathbf{c})\Vert .\end{array}$$

Therefore,

$$\Vert {\mathbf{f}}^{\prime }(\mathbf{c})(\mathbf{v})\Vert \le M\Vert \mathbf{v}\Vert ,$$

where $M={\sum }_{k=1}^m\Vert \nabla f_k(\mathbf{c})\Vert$.

When does the total derivative exist?

The Jacobian matrix $\mathbf{Df}(\mathbf{c})$ is defined at each point $\mathbf{c}\in {\mathbb{R}}^n$ where all the partial derivatives $D_k{f}_i(\mathbf{c})$ exist. However, recall that the existence of all partial derivatives does not guarantee that the total derivative exists (given that a function is differentiable at a point, its derivative must be given by the Jacobian matrix, of course). So, when is a function differentiable? A simple criterion for differentiability can be stated as follows:

Definition 5(Definition).

A function is continuously differentiable on $U\subseteq {\mathbb{R}}^n$ if all of its partial derivatives exist and are continuous on $U$. Such a function is called a $C^1$ function.

Theorem 6(Theorem (Criterion for differentiability)).

If $U$ is an open subset of ${\mathbb{R}}^n$, and $\mathbf{f}:U\to {\mathbb{R}}^m$ is a $C^1$ mapping, then $\mathbf{f}$ is differentiable on $U$, and its derivative is given by its Jacobian matrix.

This is a special case of a more general criterion which relaxes the hypothesis slightly.

---

Euler’s theorem

[!Theorem]
Let $f$ be defined on an open set $S$ in ${\mathbb{R}}^n$. Assume $f$ is homogeneous of degree $p$ over $S$. If $f$ is differentiable at $\mathbf{x}$,

$$\mathbf{x}\cdot \nabla f(\mathbf{x})=pf(\mathbf{x}).$$

Proof
For fixed $\mathbf{x}$, define $g(\lambda )\equiv f(\lambda \mathbf{x})$. $g^{\prime }(\lambda )=f^{\prime }(\lambda \mathbf{x})\mathbf{x}$, so $g^{\prime }(1)=f^{\prime }(\mathbf{x})\mathbf{x}=\nabla f(\mathbf{x})\cdot \mathbf{x}$. Also, $g(\lambda )={\lambda }^{p}f(\mathbf{x})$, so $g^{\prime }(\lambda )=p{\lambda }^{p-1}f(\mathbf{x})$, and $g^{\prime }(1)=pf(\mathbf{x})$.