OK. You were close. Here is how I would do it in steps to reduce the chance of stupid error.
[MATH]f(x) = x^2 - x.[/MATH]
[MATH]\therefore f(x + h) = (x + h)^2 - (x + h) = x^2 + 2hx + h^2 - (x + h) = (x^2 - x) + 2hx + h^2 - h.[/MATH]
[MATH]\therefore f(x + h) - f(x) = 2hx + h^2 - h \implies[/MATH]
[MATH]\dfrac{f(x + h) - f(x)}{h} = \dfrac{2hx + h^2 - h}{h} = 2x - 1 + h \implies[/MATH]
[MATH]\lim_{h \rightarrow 0}\dfrac{f(x + h) - f(x)}{h} = ( \lim_{h \rightarrow 0}2x - 1 + h) = (\lim_{h \rightarrow 0} 2x - 1) + ( \lim_{h \rightarrow 0} h) = 2x - 1 + 0 = 2x - 1 \implies f'(x) = 2x - 1.[/MATH]
Calculate and simplify algebraically f(x + h) as a separate step. In the next step, subtract f(x) from the result. In the third step, divide by h. And finally take the limit. Now you can do this by simply implementing the definition as the limit of the Newton quotient right at the start, but I have found working up to that result in steps can avoid many errors.