I think part of the reason things might be going awry is that you're missing a bunch of very important grouping symbols. For instance, the original exercise, exactly as you typed it, is:
\(\displaystyle \left( \dfrac{a}{b} - \dfrac{b}{a} \right)^2 \cdot \left( \dfrac{ab}{a}-b \right)^2 = \left( \dfrac{a}{b} - \dfrac{b}{a} \right)^2 \cdot 0^2 = 0\)
Because this doesn't make any sense, I can guess that you actually meant this:
\(\displaystyle \left( \dfrac{a}{b} - \dfrac{b}{a} \right)^2 \cdot \left( \dfrac{ab}{a-b} \right)^2\)
The latter half of this expression would be typeset by ([ab]/[a-b])2. Make sure you understand the importance of these grouping symbols. The strategy you suggest seems good to me, although it is again plagued by some missing grouping symbols and what I suspect is a typo.
As part of your workings, you say: a=(a/b - b/c)2. This is a bit of a poor form, as it might get confusing to have multiple variables denoted by the same letter. You could use x, y, or any other letter, or even capital A, but using lowercase a again should be avoided, since it's already in use in the problem. Aside from that, I suspect the c should have been a, as the letter c appears nowhere in the problem proceeding this, nor following it. Additionally, it should be A = (a/b - b/a) and B = (ab)/(a-b) without the squares, as you later use the identity A2 * B2 = (AB)2. If you have the original terms being squared, you'd end up with fourth powers and you don't want that in this case.
Next... later on, you combine the two parts of the original expression into a single multiplication. This is where you're missing some grouping symbols again. It should be: ([a/b - b/a] * [ab]/[a-b])2
After that point, it doesn't look like anything you did is correct. Instead, I'd begin by distributing the ab term across the subtraction, and simplfying:
\(\displaystyle \left( \left[ \dfrac{a}{b} - \dfrac{b}{a} \right] \cdot \dfrac{ab}{a-b} \right)^2\)
\(\displaystyle = \left( \dfrac{\dfrac{a \cdot ab}{b} - \dfrac{b \cdot ab}{a}}{a-b} \right)^2\)
\(\displaystyle = \left( \dfrac{a^2 - b^2}{a-b} \right)^2\)
Can you finish up from here? Be sure to note that:
\(\displaystyle a^2 - b^2 \ne (a-b)^2\) unless \(\displaystyle b=a\) or \(\displaystyle b=0\)
