Multivariable Calculus

Higher order derivatives

\[f'(x), ~~ y', ~~ \frac{d}{dx}f(x), ~~ \frac{dy}{dx}\]

\[f''(x), ~~ y'', ~~ \frac{d^2}{dx^2}f(x), ~~ \frac{d^2y}{dx^2}\]

Higher order derivatives

\[ \begin{aligned} f(x) &=x^3\\ f^{\prime}(x) &=3x^2\\ f^{\prime\prime}(x) &=6x \\ f^{\prime\prime\prime}(x) &=6\\ f^{\prime\prime\prime\prime}(x) &=0\\ \end{aligned} \]

Partial derivatives

If you can do ordinary derivatives, you can do partial derivatives: just hold all the other input variables constant except for the one you’re differentiang with respect to.

Partial derivatives

Let \(f\) be a function of the variables \((x_1,\ldots,x_n)\). The partial derivative of \(f\) with respect to \(x_i\) is

\[\frac{\partial f}{\partial x_i} (x_1,\ldots,x_n) = \lim\limits_{h\to 0} \frac{f(x_1,\ldots,x_i+h,\ldots,x_n)-f(x_1,\ldots,x_i,\ldots,x_n)}{h}\]

Partial derivatives

• \[f(x,y)=x^2+y^2\]

• \[ \begin{aligned} \frac{\partial f}{\partial x}(x,y) &= 2x \\ \frac{\partial f}{\partial y}(x,y) &= 3y\\ \frac{\partial^2 f}{\partial x^2}(x,y) &= 2\\ \frac{\partial^2 f}{\partial x \partial y}(x,y) &= 2 \end{aligned} \]

Partial derivatives

• \[f(x,y)=x^3 y^4 +e^x -\log y\]

• \[ \begin{aligned} \frac{\partial f}{\partial x}(x,y) &= 3x^2y^4 + e^x\\ \frac{\partial f}{\partial y}(x,y) &=4x^3y^3 - \frac{1}{y}\\ \frac{\partial^2 f}{\partial x^2}(x,y) &= 6xy^4 + e^x\\ \frac{\partial^2 f}{\partial x \partial y}(x,y) &= 12x^3y^2 + \frac{1}{y^2} \end{aligned} \]

Inflection point

For a given function, \(y = f(x)\), a point \((x^∗, y^∗)\) is called an inflection point if the second derivative immediately on one side of the point is signed oppositely to the second derivative immediately on the other side.

Inflection point

Concavity

• Concave up (convex)
• Concave down (concave)
• Where a function is twice differentiable and concave over some area
  • Concave down where \(f''(x) < 0\)
  • Concave up where \(f''(x) > 0\)

Concavity

Types of extreme values

• Minimum or maximum
• Local or global (absolute)

Endpoints

Global maximum

Global minimum

Local minima and maxima

Inflection point

Framework for optimization

1. Find \(f'(x)\)
2. Set \(f'(x)=0\) and solve for \(x\). Call all \(x_0\) such that \(f'(x_0)=0\) critical values
3. Find \(f''(x)\). Evaluate at each \(x_0\)
   • If \(f''(x) > 0\), concave up, and therefore a local minimum
   • If \(f''(x) < 0\), concave down, and therefore a local maximum
   • If it’s the global maximum/minimum, it will produce the largest/smallest value for \(f(x)\)
   • On a closed range along the domain, check the endpoints as well

\(f(x) = -x^2\), \(x \in [-3, 3]\)

\(f(x) = -x^2\), \(x \in [-3, 3]\)

\[ \begin{eqnarray} f'(x) & = & - 2 x \\ 0 & = & - 2 x^{*} \\ x^{*} & = & 0 \end{eqnarray} \]

\[ \begin{eqnarray} f^{'}(x) & = & - 2x \\ f^{''}(x) & = & - 2 \end{eqnarray} \]

• \(f^{''}(x)< 0\), local maximum

\(f(x) = x^3\), \(x \in [-3, 3]\)

\(f(x) = x^3\), \(x \in [-3, 3]\)

\[ \begin{eqnarray} f'(x) & = & 3 x^2 \\ 0 & = & 3 (x^{*})^2 \\ x^{*} & = & 0 \end{eqnarray} \]

\[ \begin{eqnarray} f^{''}(x) & = & 6x \\ f^{''}(0) & = & 0 \end{eqnarray} \]

Maximum likelihood estimation

• Likelihood function
• Related to probability
• Data is known, parameters are unknown
• Maximize the function to find the parameter values located at the global maximum

Maximum likelihood estimation

\[ \begin{eqnarray} f(\mu) & = & \prod_{i=1}^{N} \exp( \frac{-(Y_{i} - \mu)^2}{ 2}) \\ & = & \exp(- \frac{(Y_{1} - \mu)^2}{ 2}) \times \ldots \times \exp(- \frac{(Y_{N} - \mu)^2}{ 2}) \\ & = & \exp( - \frac{\sum_{i=1}^{N} (Y_{i} - \mu)^2} {2}) \end{eqnarray} \]

Maximum likelihood estimation

\[ \begin{eqnarray} \log f(\mu) & = & \log \left( \exp( - \frac{\sum_{i=1}^{N} (Y_{i} - \mu)^2} {2}) \right) \\ & = & - \frac{\sum_{i=1}^{N} (Y_{i} - \mu)^2} {2} \\ & = & -\frac{1}{2} \left(\sum_{i=1}^{N} Y_{i}^2 - 2\mu \sum_{i=1}^{N} Y_{i} + N\times\mu^2 \right) \\ \frac{ \partial \log f(\mu) }{ \partial \mu } & = & -\frac{1}{2} \left( - 2\sum_{i=1}^{N} Y_{i} + 2 N \mu \right) \end{eqnarray} \]

Maximum likelihood estimation

\[ \begin{eqnarray} 0 & = & -\frac{1}{2} \left( - 2 \sum_{i=1}^{N} Y_{i} + 2 N \mu^{*} \right) \\ 0 & = & \sum_{i=1}^{N} Y_{i} - N \mu^{*} \\ N \mu^{*} & = & \sum_{i=1}^{N}Y_{i} \\ \mu^{*} & = & \frac{\sum_{i=1}^{N}Y_{i}}{N} \\ \mu^{*} & =& \bar{Y} \end{eqnarray} \]

Apply the second derivative test

\[ \begin{eqnarray} f^{'}(\mu ) & = & -\frac{1}{2} \left( - 2\sum_{i=1}^{N} Y_{i} + 2 N \mu \right) \\ f^{'}(\mu ) & = & \sum_{i=1}^{N} Y_{i} - N \mu \\ f^{''}(\mu ) & = & -N \end{eqnarray} \]

• \(-N<0\), so the function is concave down at this point and therefore it is a maximum

Computational optimization approaches

• Analytic approaches (like we just saw) are often
  • Difficult
  • Impractical
  • Unavailable
• Computational approaches
  • Algorithm that converges to a solution
  • Hopefully the right one