Your given function is \(\displaystyle y= f(x)= x- 5x^2\) and you have been told that, since the "inverse" function just "reverses" that, that you need to solve \(\displaystyle x= y- 5y^2\) for y. Yes, that is a quadratic equation so the "quadratic formula" would be one way to do it. Have you tried that? You say "I thought the quadratic formula is there to show me when the equation equals zero". Do you not see that \(\displaystyle x= y- 5y^2\) is the same as \(\displaystyle 5y^2- y+ x= 0\)? What do you get with the quadratic formula?
(Do not forget the condition on the original function "x> 1". Do you see what that is important?)