Reinforcement Learning Exercise 4.4

Exercise 4.4 The policy iteration algorithm on page 80 has a subtle bug in that it may never terminate if the policy continually switches between two or more policies that are equally good. This is OK for pedagogy, but not for actual use. Modify the pseudocode so that convergence is guaranteed.
$$\begin{array}{rl}& \text{1 Initialization}\\ & V(s)\in \mathbb{R}\text{ and }\pi (s)\in \mathcal{A}(s)\text{ arbitrarily }s\in \mathcal{S}\\ & \text{For each }s\in \mathcal{S}\text{ create an empty list : }old\_list\_of\_a(s)\\ & \text{For each }s\in \mathcal{S}\text{ create an iterator : }iterator\_old\_list\_of\_a(s)\\ & \text{2 Policy Evaluation}\\ & \text{Loop:}\\ & \Delta \leftarrow 0\\ & \text{Loop for each }s\in \mathcal{S}:\\ & v\leftarrow V(s)\\ & V(s)\leftarrow \sum _{s^{\prime },r}p(s^{\prime },r\mid s,\pi (s))[r+\gamma V(s^{\prime })]\\ & \Delta \leftarrow \max (\Delta ,|v-V(s)|)\\ & \text{until }\Delta <\theta \text{ (a small positive number determining the accuracy of estimation) }\\ & \text{3 Policy Improvement}\\ & policy\text{-}stable\leftarrow true\\ & \text{For each }s\in \mathcal{S}:\\ & V_{max}(s)\leftarrow \underset{a}{\max }\sum _{s^{\prime },r}p(s^{\prime },r\mid s,a)[r+\gamma V(s^{\prime })]\\ & \text{Create an empty list: }new\_list\_of\_a(s)\\ & \text{For each }a\in |\mathcal{A}(s)|:\\ & \text{If }\sum _{s^{\prime },r}p(s^{\prime },r\mid s,a)[r+\gamma V(s^{\prime })]\text{ is equal to }{V}_{max}(s):\\ & \text{Append }a\text{ to }new\_list\_of\_a(s)\\ & \text{If }new\_list\_of\_a(s)\text{ is not equal to }old\_list\_of\_a(s)\text{ :}\\ & policy\text{-}stable\leftarrow false\\ & old\_list\_of\_a(s)\leftarrow new\_list\_of\_a(s)\\ & iterator\_old\_list\_of\_a(s)\leftarrow \text{the beginning of }old\_list\_of\_a(s)\\ & \text{else :}\\ & \text{If }old\_list\_of\_a(s)\text{ is empty :}\\ & old\_list\_of\_a(s)\leftarrow new\_list\_of\_a(s)\\ & iterator\_old\_list\_of\_a(s)\leftarrow \text{the beginning of }old\_list\_of\_a(s)\\ & policy\text{-}stable\leftarrow false\\ & \text{else :}\\ & \text{If }iterator\_old\_list\_of\_a(s)\text{ is not equal to the end of }old\_list\_of\_a(s)\text{ :}\\ & \text{Move }iterator\_old\_list\_of\_a(s)\text{ to next.}\\ & policy\text{-}stable\leftarrow false\\ & \text{If }iterator\_old\_list\_of\_a(s)\text{ is not equal to the end of }old\_list\_of\_a(s)\text{ :}\\ & \pi (s)\leftarrow \text{Select }a\text{ in }old\_list\_of\_a(s)\text{ by }iterator\_old\_list\_of\_a(s)\\ & \text{else : }\\ & \pi (s)\leftarrow \text{Select }a\text{ in }old\_list\_of\_a(s)\text{ randomly}\\ & \text{If }policy\text{-}stable=true\text{ then stop and return }V\approx v_{\ast }\text{ and }\pi \approx {\pi }_{\ast }\text{ else go to 2.}\end{array}$$