The use of 0 for theta is perfectly fine as long as you say what you are doing. In the equation maybe this would make it clearer (I assume the x and X are the same)
U(0) = a2 + x2 - 2 a x cos0
Thus, since a and x are not functions of theta, they are treated as constants and their derivatives are zero. The derivative of cos(0) is -sin(0).
As far as the other question, I'm not sure what you mean unless it is where you end up with the part that looks like
\(\displaystyle \sqrt{a^2\, +\, x^2\, -\, 2\, a\, x\, cos(\pi)}\, -\, \sqrt{a^2\, +\, x^2\, -\, 2\, a\, x\, cos(0)}\)
= \(\displaystyle \sqrt{a^2\, +\, x^2\, +\, 2\, a\, x}\, -\, \sqrt{a^2\, +\, x^2\, -\, 2\, a\, x}\)
and that is just the evaluation of \(\displaystyle cos(\pi)\, =\, -1\, and\, cos(0)\, =\, 1\)
EDIT: BTW: you need to be careful with that \(\displaystyle \sqrt{(x-a)^2}\, =\, x\, -\, a\) and similarly with the other square root. Technically \(\displaystyle \sqrt{(x-a)^2}\, =\, |x\, -\, a|\) so the assumption in the problem is that x \(\displaystyle \ge\) |a|.