Actually I've worked it out now. Although I didn't use part a.
For part a I got a^2 +2ab sqrt(2) + 2b^2
For part b, when I expand I get 17 - 12 sqrt(2). Since I know that (1 - sqrt(2))^4 must be 0 or positive, it follows that sqrt(2) must be less than 17/12