Okay, since you specifically ask about the Laplace transform:
You arrive at the two equation in X(s) and Y(s):
X(s)(s- 2)= Y(s)+ 1 and Y(s)(s- 2)= 5X(s)+ 1.
You then try to write X(s) in terms of Y(s) and Y(s) in terms of X(s). That doesn't lead any where! The whole point of the Laplace transform is that we have reduced the system of differential equations to a system of algebraic equations which we can solve using algebra.
From X(s)(s- 2)= Y(s)+ 1, we get Y(s)= X(s)(s- 2)- 1. Putting that into the other equation (X(s)(s- 2)- 1)(s- 2)= X(s)(s- 2)^2- (s- 2)= 5X(s)+ 1. X(s)((s- 2)^2- 5)= (s- 2)+ 1= s- 1. \(\displaystyle X(s)= \frac{s- 1}{(s- 2)^2- 5}= \frac{s- 1}{s^2- 4x- 1}\).
And then \(\displaystyle Y(s)= X(s)(s- 2)- 1= \frac{(s- 1)(s- 2)}{s^2- 4x- 1}- 1\).
Can you work out the inverse Laplace transform, to get x(t) and y(t), from that?