Consider an irreducible Markov chain that is positive recurrent.  Recall the technique used to find the expected first-passage time from j to i, i.e., Tji, in Section 4.5. The state i was turned into a trapping state by turning all the transitions out of i into a single transition Pii = 1. Here, in order to preserve the positive recurrence, we instead move all transitions out of state i into the single transition Pij = 1.

(a) Use Figure 6.2 to illustrate that the above strategy can turn an irreducible chain into a reducible chain. Also explain why states i and j are still positive recurrent and still in the same class.

(b) Let {π ∗ ; k ≥ 0} be the steady-state probabilities for the positive-recurrent class in the modified Markov chain. Show that the expected first passage time T∗ from j to i in the modified Markov chain is (1/π ∗) - 1.

(c) Show that the expected first passage time from j to i is the same in the modified and unmodified chains.

(d) Show by example that after the modification above, two states i and j that were not positive recurrent before the modification can become positive recurrent and the above technique can again be used to find the expected first-passage time.

Text Book: Stochastic Processes: Theory for Applications By Robert G. Gallager.