Help with basic projections?

I am half way through my first graduate econometrics course, but I'm having some significant gaps in my stats background to get everything.


In particular, I am still struggling a bit on how projections work. Generally in applications I have no problem using them (I've become quite familiar with OLS, and am moving on to GLS and FGLS), but here I am trying to re-write some of my notes, and I am having a very hard time putting into words exactly what a projection is, and how it works.


I've been looking around the internet, and I can't get the distinction between any arbitrary projection, an idempotent projection, and an orthogonal projection, specifically in the context of basic linear regressions.


Since the professor spent about 5 minutes on the first day going over it, I guess I should have learned this at a more basic course level, but I've never worked with it before.

What I have so far is that a projection is idempotent if p^2 = p, and it is a matrix which projects a vector onto another matrix (or something like that).

I also have this proof showing that p is symmetric and idempotent:

p' = (x(x'x)^(-1) x')' = (x')'((x'x)^-1)'x'...
x(x'x)^-1 x' = p

I get the matrix algebra easily enough, and if my understanding of idempotency from above is accurate (which I believe it is, given proofs I have done relating to the OLS) then I understand that as well. But I don't know how I am proving symmetry, and its implications.

aegisknight said:

I am half way through my first graduate econometrics course, but I'm having some significant gaps in my stats background to get everything.


In particular, I am still struggling a bit on how projections work. Generally in applications I have no problem using them (I've become quite familiar with OLS, and am moving on to GLS and FGLS), but here I am trying to re-write some of my notes, and I am having a very hard time putting into words exactly what a projection is, and how it works.


I've been looking around the internet, and I can't get the distinction between any arbitrary projection, an idempotent projection, and an orthogonal projection, specifically in the context of basic linear regressions.


Since the professor spent about 5 minutes on the first day going over it, I guess I should have learned this at a more basic course level, but I've never worked with it before.

What I have so far is that a projection is idempotent if p^2 = p, and it is a matrix which projects a vector onto another matrix (or something like that).

Click to expand...

A matrix maps a vector to a vector. A projection matrix projects a vector onto a subspace, called the "invariant subspace" for p because, if v is in that subspace, pv= v.

I also have this proof showing that p is symmetric and idempotent:

p' = (x(x'x)^(-1) x')' = (x')'((x'x)^-1)'x'...
x(x'x)^-1 x' = p

I get the matrix algebra easily enough, and if my understanding of idempotency from above is accurate (which I believe it is, given proofs I have done relating to the OLS) then I understand that as well. But I don't know how I am proving symmetry, and its implications.

Click to expand...

What do you mean by "symmetry" of a linear transformation? That its matrix representation is symmetric (in any basis)? That is true when a matrix has a "complete" set of "eigenvectors"- n independent eigenvectors for an n by n matrix. That's why the fact that a projection projects onto a subspace is important!

Let U be the subspace p projects onto. If v happens to be in that subspace, as I said, pv= v so that v is an eigenvector with eigenvalue 1. If v is in the "orthogonal complement" of that subspace, then pv=0 so that v is an eigenvector with eigenvalue 0. That is, p has the two eigenvalues, 0 and 1. Select a basis for that subspace, and a basis for the orthogonal subspace. The matrix representing p, in that basis, is a diagonal matrix and so symmetric. It follows that the matrix representing p in any basis is symmetric.

So in the context of linear regressions, why is idempotency important? Further, what does it mean to be orthogonal, and why is that important?

For now, I am trying to define everything in terms of the OLS estimator working with the most restrictive assumptions (homogeneity of errors, errors normally distributed, expected value of errors is zero, linear independence of explanatory variables).

I know that in the model Y = Xβ + μ, (x'x)^(-1) x' is an idempotent orthogonal projection, and can project a vector of Y onto a vector of βhat to give me my OLS estimation. I can prove its idempotency, and I can use it with the t and F tests (for the most part), but I don't know the underlying mechanisms for what I'm doing (I'm just going through the motions).

Is idempotency important because it allows me to do the simplifications necessary to define βhat? What does "orthogonal" mean, and why is it important?