I would begin as you did, by using similarity:
[MATH]\frac{h}{R}=\frac{h-r}{r}[/MATH]
Solve for \(r\):
[MATH]r=\frac{Rh}{h+R}[/MATH]
Now, the volume of the cone is:
[MATH]V=\frac{1}{3}\pi R^2h\implies h=\frac{3V}{\pi R^2}[/MATH]
Hence:
[MATH]r=\frac{R\cdot\dfrac{3V}{\pi R^2}}{\dfrac{3V}{\pi R^2}+R}=\frac{3RV}{\pi R^3+3V}[/MATH]
Maximizing the volume of the sphere is equivalent to maximizing the radius of the sphere, and so we find:
[MATH]r'=\frac{3V(3V-2\pi R^3)}{(\pi R^3+3V)^2}=0[/MATH]
From this, we obtain the critical value:
[MATH]R=\sqrt[3]{\frac{3V}{2\pi}}[/MATH]
Now, the first derivative test easily allows us to state:
[MATH]r_{\max}=r\left(\sqrt[3]{\frac{3V}{2\pi}}\right)=\sqrt[3]{\frac{4V}{9\pi}}[/MATH]
And so the maximum volume of the sphere is:
[MATH]V_{S_{\max}}=\frac{4}{3}\pi r_{\max}^3=\frac{16V}{27}[/MATH]
With [MATH]V=10\text{ m}^3[/MATH], we have:
[MATH]V_{S_{\max}}=\frac{160}{27}\text{ m}^3[/MATH]