Well, the area of that rectangle is just xy so the problem is to maximize F(x,y)= xy subject to the condition that \(\displaystyle G(x,y)= x^2+ 2x+ y^2= 3\).
Now I can think of two different ways to do that but I don't know which is appropriate for you since you have not said anything about what this course is or shown any attempt of your own.
One method, if this is a "Calculus III" course, is to write \(\displaystyle H(x,y)= xy- \alpha(x^2+ 2x+ y^2- 3)\), take the derivatives with respect to x, y, and \(\displaystyle \alpha\) and set them equal to 0 (of course, the derivative with respect to \(\displaystyle \alpha\) is just the constraint).
That will give you three equations for x, y, and \(\displaystyle \alpha\).
Another, more basic but more tedious, is to complete the square: \(\displaystyle x^2+ 2x+ 1+ y^2= (x+ 1)^2+ y^2= 4\) so that
\(\displaystyle y= \pm\sqrt{4- (x+1)^2}\) and then
\(\displaystyle xy= \pm x\sqrt{4- (x+1)^2}\).
