(a) Find the steady-state probabilities for each of the Markov chains in Figure 4.2. Assume that all clockwise probabilities in the first graph are the same, say p, and assume that P4,5 = P4,1 in the second graph.

(b) Find the matrices [P2] for the same chains. Draw the graphs for the Markov chains represented by [P2], i.e., the graph of two step transitions for the original chains. Find the steady-state probabilities for these two-step chains. Explain why your steady-state probabilities are not unique.

(c) Find limn→∞[P2n] for each of the chains.

Text Book: Stochastic Processes: Theory for Applications By Robert G. Gallager.