It would be helpful if you told us what you got for part "a"!
There are, of course, three "similar triangles" in the picture, the entire right triangle with base length 8 cm and height 6 cm, the triangle on top of the rectangle with base length x and height 6- y, and the triangle to the right with base length 8- x and height y.
Since those are similar we must have \(\displaystyle \frac{8}{6}= \frac{4}{3}= \frac{x}{6- y}= \frac{8- x}{y}\). From \(\displaystyle \frac{4}{3}= \frac{x}{6- y}\), 4(6- y)= 3x so 24- 4y= 3x, 3x+ 4y= 24. From \(\displaystyle \frac{4}{3}= \frac{8- x}{y}\), 4y= 3(8- x), 4y= 24- 3x, 3x+ 4y= 24. Those are the same equation which give "y in terms of x" as \(\displaystyle y= 6- \frac{3}{4}x\).
The area of the rectangle is \(\displaystyle xy= x(6- \frac{3}{4}x)= 6x- \frac{3}{4}x^2= -\frac{3}{4}(x^2- 8x)\). The graph of that is a parabola opening downward. Its maximum value is at the vertex which you can find by "completing the square".