Let \(P=(x,y,z)\) be an arbitrary point on the surface. The distance between \(P\) and the given point \((7,11,0)\) is given by the function
\(d(x,y,z)=\sqrt{(x-7)^2+(y-11)^2+z^2}\)
Note that \(f(x)\) and \(f(x)^2\) attain their extrema, if they have any, at the same values of \(x\). This allows us to consider the modified distance function,
\(d^*(x,y,z)=(x-7)^2+(y-11)^2+z^2\)
So now you're minimizing \(d^*(x,y,z)\) subject to the constraint \(z^2=xy\). This is a perfect candidate for applying the method of Lagrange multipliers.
The Lagrangian in this case would be
\(\mathcal L(x,y,z,\lambda)=d^*(x,y,z)+\lambda(z^2-xy)\)
which has partial derivatives
\(\begin{cases}\dfrac{\mathrm d\mathcal L}{\mathrm dx}=2(x-7)-\lambda y\\\\\dfrac{\mathrm d\mathcal L}{\mathrm dy}=2(y-11)-\lambda x\\\\\dfrac{\mathrm d\mathcal L}{\mathrm dz}=2z+2\lambda z\\\\\dfrac{\mathrm d\mathcal L}{\mathrm d\lambda}=z^2-xy\end{cases}\)
Setting all four equation equal to 0, you find from the third equation that either \(z=0\) or \(\lambda=-1\). In the first case, you arrive at a possible critical point of \((0,0,0)\). In the second, plugging \(\lambda=-1\) into the first two equations gives
\(\begin{cases}2(x-7)+y=0\\2(y-11)+x=0\end{cases}\implies\begin{cases}2x+y=14\\x+2y=22\end{cases}\implies x=2,y=10\)
and plugging these into the last equation gives
\(z^2=20\implies z=\pm\sqrt{20}=\pm2\sqrt5\)
So you have three potential points to check: \((0,0,0)\), \((2,10,2\sqrt5)\), and \((2,10,-2\sqrt5)\). Evaluating either distance function (I use \(d^*\)), you find that
\(d^*(0,0,0)=170\)
\(d^*(2,10,2\sqrt5)=46\)
\(d^*(2,10,-2\sqrt5)=46\)
So the two points on the surface \(z^2=xy\) closest to the point \((7,11,0)\) are \((2,10,\pm2\sqrt5)\).
