Chapter 4.8, Problem 2P

Using the two-variable Taylor series [say ( 2.7)] prove the following "second derivative tests" for maximum or minimum points of functions of two variables. If $f_x=f_y=0$ at $(a,b),$ then $(a,b)$ is a minimum point if at $(a,b),f_{xx}>0,f_{yy}>0,$ and $f_{xx}{f}_{yy}>f_{xy}^2;$

$(a,b)$ is a maximum point if at $(a,b),f_{xx}<0,f_{yy}<0,$ and $f_{xx}{f}_{yy}>f_{xy}^2;$ $(a,b)$ is neither a maximum nor a minimum point if $f_{xx}{f}_{yy}<f_{xy}^2.$ (Note that this includes $f_{xx}{f}_{yy}<0$, that is, $f_{xx}$ and $f_{yy}$ of opposite sign.)

Hint: Let $f_{xx}=A,f_{xy}=B,f_{yy}=C;$ then the second derivative terms in the Taylor series are $A{h}^2+2Bhk+C{k}^2;$ this can be written $A{(h+Bk/A)}^2+\left(C-B^2/A\right)k^2$ Find out when this expression is positive for all small $h,k$ [that is, all $(x,y)$ near $(a,b)];$ also find out when it is negative for all small h, k, and when it has both positive and negative values for small h, k .