Rewrite totally in terms of cos.
\(\displaystyle 8cos^{2}(x)-3\frac{sin^{2}(x)}{cos^{2}(x)}-5=0\)
\(\displaystyle 8cos^{2}(x)-3(\frac{1-cos^{2}(x)}{cos^{2}(x))})-5=0\)
Multiply by \(\displaystyle cos^{2}(x)\):
\(\displaystyle 8cos^{4}(x)-3+3cos^{2}(x)-5cos^{2}(x)=0\)
\(\displaystyle 8cos^{4}(x)-2cos^{2}(x)-3=0\)
Now, let \(\displaystyle u=cos(x)\) and
\(\displaystyle 8u^{4}-2u^{2}-3=0\)
This has two real solutions: \(\displaystyle u=\pm\frac{\sqrt{3}}{2}\)
Now, see how to get \(\displaystyle x=\frac{\pi}{6}\)?.