Markov Chain Problem

[rsingh628]

Hello all, I am studying Markov chains in my math class, and I have some extra study problems that my professor provided but no solutions given. I would appreciate it if anyone can check over my work and kindly provide guidance if I'm understanding the problem correctly. I am having some trouble with the last 2 parts trying to solve the systems of equations for the long-term pmf, because all I get is 0. Any help with checking my work is greatly appreciated!

Problem:


Approach:

(a) The transient states are 1 and 2, since a path exists where if you leave, there is no way to get back to that state.
(b) The recurrent states are 0 and 3, once leaving these states, there is always a way back
(c) I believe states 1 and 2 are periodic with period 2, always requires an even number of paths to get back to original state
(d) The absorbing states are 0 and 3, since you cannot leave that state, i.e. the $P_{ii} = 1$, where i = 0 or 3
(e) The probability is 0, since you cannot get back to state 1 in just 1 step.
(f) The probability is 0.5 from the graph, with just 1 step to get from state 1 to state 0.
(g)
I believe the the transition matrix $P$ is below.

$$P=\begin{bmatrix} 1& 0 & 0 & 0\\ 0.5 & 0 & 0.5 & 0\\ 0 & 2/3 & 0 & 1/3\\ 0 & 0 & 0 & 1\\ \end{bmatrix}$$
The question is wanting the probability $$P(X_k = j | x_0 = i)$$ of going from state i to j in k steps
$$P(x_6 = 3 | x_0 = 1) = (P^6)_{1,3}$$ or the (1,3) element of $P^6$. Using a matrix power calculator, I find $$P(x_6 = 3 | x_0 = 1) = 13/54$$. Is there a simpler way to compute by hand, or just exploit powers of 2?


(h) I'm somewhat struggling to find the long-term pmf because when I solve $\pi = \pi P$, where $\pi = [\pi_0 \pi_1 \pi_2 \pi_3 ]$, I get 4 equations where I can trivially find that $\pi_1 = \pi_2= 0$ . If I use a Markov calculator, I also find $\pi_1 = 0$
(i) Similarly using a calculator, I find the long-term pmf as $\pi_0 = 0.75$

 

[blamocur]

I know less about Markov chains then I'd like to, but I played with the transition matrix on my computer and got the same answers as yours. Even if both of us have high probabilities of error the probabilities that we both made the same errors should be lower

Regarding (g): you can get from 1 to 3 in 2, 4 or 6 steps, and the probabilities for each of the 3 scenarios are relatively simple and lead to the same answer as yours.

 

[Steven G]

1 too get the same answer for (g)

 

[rsingh628]

I know less about Markov chains then I'd like to, but I played with the transition matrix on my computer and got the same answers as yours. Even if both of us have high probabilities of error the probabilities that we both made the same errors should be lower

Regarding (g): you can get from 1 to 3 in 2, 4 or 6 steps, and the probabilities for each of the 3 scenarios are relatively simple and lead to the same answer as yours.

Right, I verified it right after I posted and got the same answer. Thanks for checking. Do you see obvious errors in the other answers?

 

[blamocur]

Right, I verified it right after I posted and got the same answer. Thanks for checking. Do you see obvious errors in the other answers?

Don't see any errors. I had to read on the definitions of transient, recurrent and other types of states, but your answers match mine.

 

[rsingh628]

Don't see any errors. I had to read on the definitions of transient, recurrent and other types of states, but your answers match mine.

Sounds good. Appreciate your help!