for example, given a triangle ABC, above AB is outside the triangle, Above BC is outside the triangle etc
That doesn't really make sense. What if AB is the bottom of the triangle??? It appears that you are not really asking about "above" or "below" at all. That could be part of your problem, or you may just not have clearly stated what you are looking for.
In terms of a triangle, we have to define "inside"; one convenient way is to say that P is inside ABC when it is on the same side of AB as C, on the same side of BC as A, and on the same side of CA as B. This will be the key to understanding the algorithm.
This was one of the examples I seen and I'm not sure what the first equation at the top represents and works out. I understand that it is calculating the sign, but how is it doing that.
For context, I see that this is in
https://stackoverflow.com/questions/2049582/how-to-determine-if-a-point-is-in-a-2d-triangle, and is in response to a question about determining whether a point lies in a triangle. I also see that issues have been raised about what happens when a point outside the triangle lies on one of the (extended) edges. But I'll just comment on the main idea.
The function "sign" returns twice the signed area of the triangle P1P2P3 (that is, positive in one orientation, negative in the other); for one source, see the last formula under Modern Triangles
here. That can be taken as determining whether P3 is on the left or the right of P1P2.
Given that, and my definition above, can you see what is being done?