If I'm understanding you right, you've noticed that:
\(\displaystyle \sin \left(-\frac{\pi}{3} \right) = -\frac{\sqrt{3}}{2}\) and \(\displaystyle \sin \left(\frac{5\pi}{3} \right) = -\frac{\sqrt{3}}{2}\)
But are struggling to reconcile this information with what your calculator tells you:
\(\displaystyle \sin^{-1} \left( -\frac{\sqrt{3}}{2} \right) = -\frac{\pi}{3}\)
Assuming this is correct, the trick here lies in the fact that we define \(\displaystyle \sin^{-1}(x) = \arcsin(x)\) as the inverse function of sine. This means that for every \(x\) in the domain, there must be exactly one output. To ensure this, we restrict the domain of inverse sine to \(\displaystyle [-1, 1]\) which also necessarily restricts the range to \( [-\frac{\pi}{2}, \frac{\pi}{2}] \)
Hence, the inverse sine function maps \( -\frac{\sqrt{3}}{2} \) uniquely to \( -\frac{\pi}{3} \), rather than any of the infinitely many other \(y\) such that \( \sin(y) = -\frac{\sqrt{3}}{2} \)