It would be helpful if you told us what you got for part "a"!
There are, of course, three "similar triangles" in the picture, the entire right triangle with base length 8 cm and height 6 cm, the triangle on top of the rectangle with base length x and height 6- y, and the triangle to the right with base length 8- x and height y.
Since those are similar we must have
68=34=6−yx=y8−x. From
34=6−yx, 4(6- y)= 3x so 24- 4y= 3x, 3x+ 4y= 24. From
34=y8−x, 4y= 3(8- x), 4y= 24- 3x, 3x+ 4y= 24. Those are the same equation which give "y in terms of x" as
y=6−43x.
The area of the rectangle is
xy=x(6−43x)=6x−43x2=−43(x2−8x). The graph of that is a parabola opening downward. Its maximum value is at the vertex which you can find by "completing the square".