Typically, if you have an odd power of sine or cosine you can factor one out, to use with the "dx", then use \(\displaystyle sin^2(x)+ cos^2(x) = 1\), so that either \(\displaystyle sin^2(x)= 1- cos^2(x)\) or \(\displaystyle cos^2(x)= 1- sin^2(x)\) to reduce the remaining even power. Here, you have \(\displaystyle \int_0^{\pi/2} sin^3(\theta) cos^2(\theta)d\theta\) which we can write as \(\displaystyle \int_0^{\pi/2} sin^2(\theta) cos^2(\theta) (sin(\theta) d\theta)= \int_0^{\pi/2} (1- cos^2(\theta))cos^2(\theta)(sin(\theta)d\theta)= \int_0^{\pi/2} (cos^2(\theta)- cos^4(\theta))(sin(\theta)d\theta)\).
Now make the substitution \(\displaystyle u= cos(\theta)\).