23. Tabular MDPs

PDF Version

In this lecture we will analize an online learning algorithm for the finite-horizon episodic MDP setting. Let $M=(\mathcal{S},\mathcal{A},P^{\ast },r,\mu ,H)$ be an MDP with finite state and action spaces $\mathcal{S}$ and $\mathcal{A}$, unknown transition matrix $P^{\ast }$, known reward function $r_a(s)\in [0,1]$, an initial state distribution $\mu$, and length of each episode $H\ge 1$. The star-superscript in $P^{\ast }$ is used to distinquish the true environment from other (e.g. estimated) environments that occur in the algorithm and the analysis. The assumption that the reward function $r$ is known is for simplicity. In fact, most of the hardness (in terms of sample complexity and designing the algorithm) comes from unknown transition probabilities.

We will focus on the finite-horizon setting where the learner interacts with the MDP over $k=1,\dots ,K$ episodes of length $H\ge 1$. Most, but not all ideas translate to the infinite-horizon discounted or average reward settings.

Recall that the regret is defined as follows:

$$R_K=\sum _{k=1}^K{v}_0^{\ast }(S_0^{(k)})-V_k$$

where $V_k=\sum _{h=0}^{H-1}{r}_{A_h^{(k)}}(S_h^{(k)})$.

UCRL: Upper Confidence Reinforcement Learning

The UCRL algorithm implements the optimism princple. For this we need to define a set of plausible models. First, we define the maximum likelihood estimates using data from rounds $1,\dots ,k-1$:

$$P_a^{(k)}(s,s^{\prime })=\frac{N_k(s,a,s^{\prime })}{1\vee N_k(s,a)}$$

The definition makes use of the notation $a\vee b=max(a,b)$, and empirical counts:

$$\begin{array}{rl}{N}_k(s,a)& =\sum _{k^{\prime }<k}\sum _{h<H}\mathbb{I}(S_h^{(k)}=s,A_h^{(k)}=a)\\ N_k(s,a,s^{\prime })& =\sum _{k^{\prime }<k}\sum _{h<H}\mathbb{I}(S_h^{(k)}=s,A_h^{(k)}=a,S_{h+1}^{(k)}=s^{\prime })\end{array}$$

Define the confidence set

$$C_{k,\delta }=\{P_a(s):\mathrm{\forall }s,a\Vert P_a^{(k)}(s)-P_a(s){\Vert }_1\le {\beta }_{\delta }(N_k(s,a))\}$$

where ${\beta }_{\delta }:\mathbb{N}\to (0,\mathrm{\infty })$ is a function that we will choose shortly. Our goal of choosing ${\beta }_{\delta }$ is to ensure that

1. $P^{\ast }\in C_{k,\delta }$ for all $k=1,\dots ,K$ with probability at least $1-\delta$
2. $C_{k,\delta }$ is “not too large”

The second point will appear formually in the proof, however note that from a statistical perspective, we want the confidence set to be as efficient as possible.

With the confidence set, we can now introduce the UCRL algorithm:

---

UCRL (Upper confidence reinforcement learning):

In episodes $k=1,\dots ,K$,

1. Compute confidence set $C_{k,\delta }$
2. Use policy ${\tilde{\pi }}_k=\arg \underset{\pi }{max}\underset{P\in C_{k,\delta }}{max}{v}_P^{\pi }$
3. Observe episode data $S_0^{(k)},A_0^{(k)},S_1^{(k)},\dots ,S_{H-1}^{(k)},S_{H-1}^{(k)},S_H^{(k)}$

---

Note that we omitted the rewards from the observation data. Since we made the assumption that the reward vector $r_a(s)$ is known, we can always recompute the rewards from the state and action sequence.

For now we we also glance over the point of how to compute the optimistic policy ${\pi }_k$ efficently, but we will get back to this point later.

Step 1: Defining the confidence set

---

Lemma (L1-confidence set): Let ${\beta }_{\delta }(u)=2\sqrt{\frac{S\log (2)+\log (u(u+1)SA/\delta )}{2u}}$ and define the confidence sets

$$C_{k,\delta }=\{P_a(s):\mathrm{\forall }s,a\Vert P_a^{(k)}(s)-P_a(s){\Vert }_1\le {\beta }_{\delta }(N_k(s,a))\}$$

Then, with probability at least $1-\delta$,

$$\mathrm{\forall }k\ge 1,P^{\ast }\in C_{k,\delta }$$

---

Proof: Let $s,a$ be fixed and denote by $X_v\in \mathcal{S}$ the next state observed upon visiting $(s,a)$ the $v^{\text{th}}$ time. Assume that $(s,a)$ was visited in total $u$ times. Then $P_{u,a}(s,s^{\prime })=\frac{1}{u}\sum _{v=1}^u\mathbb{I}(X_v=s^{\prime })$.

The Markov property implies that $(X_v{)}_{v=1}^u$ is i.i.d. Note that for any vector $p\in {\mathbb{R}}^S$ we can write the 1-norm as $\Vert p{\Vert }_1=\underset{\Vert x{\Vert }_{\mathrm{\infty }}\le 1}{sup}⟨p,x⟩$. Therefore

$$\Vert P_{u,a}(s)-P_a^{\ast }(s){\Vert }_1=\underset{x\in \{\pm 1{\}}^S}{max}⟨P_{u,a}(s)-P_a^{\ast }(s),x⟩$$

Fix some $x\in \{\pm 1{\}}^S$.

$$\begin{array}{rl}⟨P_{u,a}(s)-P_a^{\ast }(s),x⟩& =\frac{1}{u}\sum _{v=1}^u\sum _{s^{\prime }}{x}_{s^{\prime }}(\mathbb{I}(X_v=s^{\prime })-P_a^{\ast }(s,s^{\prime }))\\ & =\frac{1}{u}\sum _{v=1}^u{\mathrm{\Delta }}_v\end{array}$$

where in the last line we defined ${\mathrm{\Delta }}_v=\sum _{s^{\prime }\in \mathcal{S}}{x}_{s^{\prime }}(\mathbb{I}(X_v=s^{\prime })-P_a^{\ast }(s,s^{\prime }))$. Note that $\mathbb{E}[{\mathrm{\Delta }}_v]=0$, $|{\mathrm{\Delta }}_v|\le 1$ and $({\mathrm{\Delta }}_v{)}_{v=1}^u$ is an i.i.d. random variable. Therefore Hoeffding’s inequality implies that with probability at least $1-\delta$,

$$\frac{1}{u}\sum _{v=1}^u{\mathrm{\Delta }}_v\le 2\sqrt{\frac{\log (1/\delta )}{2u}}$$

Next note that $|\{\pm 1{\}}^S|=2^S$, therefore taking the union bound over all $x\in \{\pm 1{\}}^S$, we get that with probability at least $1-\delta$,

$$\Vert P_{u,a}(s)-P_a^{\ast }(s){\Vert }_1\le 2\sqrt{\frac{S\log (2)+\log (1/\delta )}{2u}}$$

In a last step, we take a union bound over $s\in \mathcal{S}$, $a\in \mathcal{A}$ and $u\ge 1$. For taking the union bound over the infinite set of natural numbers, we can use the following simple trick. Note that

$$\sum _{u=1}^{\mathrm{\infty }}\frac{\delta }{u(u+1)}=\delta$$

This follows from the simple obseration that $\frac{1}{u(u+1)}=\frac{1}{u}-\frac{1}{u+1}$ and using a telescoping sum argument. Therefore, with probability at least $1-\delta$, for all $u\ge 1$, $s\in \mathcal{S}$ and $a\in \mathcal{A}$

$$\Vert P_{u,a}(s)-P_a^{\ast }(s){\Vert }_1\le 2\sqrt{\frac{S\log (2)+\log (u(u+1)SA/\delta )}{2u}}$$

Lastly, the claim follows by noting that $P_a^{(k)}(s)=P_{N_k(s,a),a}(s)$. $◼$

Step 2: Bounding the regret

---

Theorem (UCRL Regret): The regret of UCRL defined with confidence sets $C_{k,\delta }$ satisfies with probability at least $1-3\delta$:

$$R_K\le 4c_{\delta }H\sqrt{SAHK}+2c_{\delta }{H}^{2}SA+3H\sqrt{\frac{HK}{2}\log (1/\delta )}$$

where $c_{\delta }=\sqrt{2S\log (2)+\log (HK(HK+1)SA/\delta )}$. In particular, for large enough $K$, surpressing constants and logarithmic factors, we get

$$R_K\le \stackrel{\mathcal{~}}{\mathcal{O}}\left(H^{3/2}S\sqrt{AK\log (1/\delta )}\right)$$

---

Proof: Denote by ${\pi }_k$ the UCRL policy defined as

$${\pi }_k=\arg \underset{\pi }{max}\underset{P\in C_{k,\delta }}{max}{v}_{0,P}^{\pi }(S_0^{(k)})$$

Further, let ${\tilde{P}}^{(k)}=\arg \underset{P\in C_{k,\delta }}{max}{v}_{0,P}^{\ast }(S_0^{(k)})$ be the optimistic model.

In what follows we assume that we are on the event $\mathcal{E}={\cap }_{k\ge 1}{C}_{k,\delta }$. By the previous lemma, $\mathbb{P}(\mathcal{E})\ge 1-\delta$.

Fix $k\ge 1$ and decompose the (instantenous) regret in round $k$ as follows:

$$\begin{array}{rl}{v}_0^{\ast }(S_0^{(k)})-V_k=& \underset{\text{(I)}}{\underbrace{v_{0,P^{\ast }}^{\ast }(S_0^{(k)})-v_{0,{\tilde{P}}_k}^{\ast }(S_0^{(k)})}}\\ & +\underset{\text{(II)}}{\underbrace{v_{0,{\tilde{P}}_k}^{{\pi }_k}(S_0^{(k)})-v_{0,P^{\ast }}^{{\pi }_k}(S_0^{(k)})}}\\ & +\underset{\text{(III)}}{\underbrace{v_{0,P^{\ast }}^{{\pi }_k}(S_0^{(k)})-V_k}}\end{array}$$

Note that we used that $v_{0,{\tilde{P}}_k}^{\ast }(S_0^{(k)})=v_{0,{\tilde{P}}_k}^{{\pi }_k}(S_0^{(k)})$ which holds because by definition ${\pi }_k$ is an optimal policy for ${\tilde{P}}_k$.

The first term is easily bounded. This is the crucial step that makes use of the optimism principle. By $P^{\ast }\in C_{k,\delta }$ and the choice of ${\tilde{P}}_k$ it follows that $\text{(I)}\le 0$. In particular, we already eliminated the dependence on the (unknown) optimal policy from the regret bound!

The last term is also relatively easy to control. Denote ${\xi }_k=\text{(III)}$. Note that by the definition of the value function we have $\mathbb{E}[{\xi }_k|S_0^{(k)}]=0$ and $|{\xi }_k|\le H$. Hence ${\xi }_k$ behaves like noise! If ${\xi }_k$ was an i.i.d variable we could directly apply Hoeffding’s inequality to bound $\sum _{k=1}^K{\xi }_k$.

The sequence ${\xi }_k$ has a property that allows us to obtain a similar bound. Let

$${\mathcal{F}}_k=\{S_0^{(l)},A_0^{(l)},S_1^{(l)},\dots ,S_{H-1}^{(l)},S_{H-1}^{(l)},S_H^{(l)}{\}}_{l=1}^{k-1}$$

be the data available to the learner at the beginning of the episode $k$. Then by definition of the value function, $\mathbb{E}[{\xi }_k|{\mathcal{F}}_k,S_0^{(k)}]=0$.

A sequence of random variables $({\xi }_k{)}_{k\ge 1}$ with this property is called a martingale difference sequence. Lucky for us, most properties that hold for (zero-mean) i.i.d. sequences can also be shown for martingale difference sequences. The analogue result to Hoeffding’s inequality is called the Azuma-Hoeffding’s inequalty. Applied to the sequence ${\xi }_k$, Azuma-Hoeffdings inequality implies that

$$\sum _{k=1}^K{\xi }_k\le H\sqrt{\frac{K}{2}\log (1/\delta )}$$

It remains to bound term (II) in the regret decomposition:

$$\text{(II)}=v_{0,P^{\ast }}^{{\pi }_k}(S_0^{(k)})-v_{0,{\tilde{P}}^{(k)}}^{{\pi }_k}(S_0^{(k)})$$

Using the Bellman equation, we can recursively compute the value function for any policy $\pi$:

$$\begin{array}{rl}{v}_{h,P}^{\pi }& =r^{\pi }+M_{\pi }P{v}_{h+1,P}^{\pi },0\le h\le H-1\\ v_{H,P}^{\pi }& =0\end{array}$$

We introduce the following shorthand for the value difference of policy ${\pi }_k$ under models $P^{\ast }$ and ${\tilde{P}}^{(k)}$:

$${\delta }_h^{(k)}=v_{h,{\tilde{P}}^{(k)}}^{{\pi }_k}(S_h^{(k)})-v_{h,P^{\ast }}^{{\pi }_k}(S_h^{(k)})$$

Let ${\mathcal{F}}_{h,k}$ contain all observation data up to episode $k$ and step $h$ including $S_h^k$. Using the Bellman equation, we can write

$$\begin{array}{rl}{\delta }_h^{(k)}& =M_{{\pi }_k}{\tilde{P}}^{(k)}{v}_{h+1,{\tilde{P}}^{(k)}}^{{\pi }_k}(S_h^{(k)})-M_{{\pi }_k}{P}^{\ast }{v}_{h+1,P^{\ast }}^{{\pi }_k}(S_h^{(k)})\pm M_{{\pi }_k}{P}^{\ast }{V}_{h+1,{\tilde{P}}^{(k)}}(S_h^{(k)})\\ & =(M_{{\pi }^k}({\tilde{P}}^{(k)}-P^{\ast })v_{h+1,{\tilde{P}}^{(k)}}^{{\pi }_k})(S_h^{(k)})+(M_{{\pi }_k}{P}^{\ast }(v_{h+1,{\tilde{P}}^{(k)}}^{{\pi }_k}-v_{h+1,P^{\ast }}^{{\pi }_k})(S_h^{(k)})\\ & \le \Vert P_{A_h^{(k)}}^{\ast }(S_h^{(k)})-{\tilde{P}}_{A_h^{(k)}}^{(k)}(S_h^{(k)}){\Vert }_{1}H+{\delta }_{h+1}^{(k)}+\underset{=:{\eta }_{h+1}^{(k)}}{\underbrace{(\mathbb{E}[{\delta }_{h+1}^{(k)}|{\mathcal{F}}_{h,k}]-{\delta }_{h+1}^{(k)})}}\\ & \le 2H{\beta }_{\delta }(N_k(S_h^{(k)},A_h^{(k)}))+{\delta }_{h+1}^{(k)}+{\eta }_{h+1}^{(k)}\end{array}$$

The first inequality uses that for any two vectors $w,v$, we have $⟨w,v⟩\le \Vert w{\Vert }_1\Vert v{\Vert }_{\mathrm{\infty }}$ and $\Vert v_{h+1,{\tilde{P}}^{(k)}}^{{\pi }_k}{\Vert }_{\mathrm{\infty }}\le H$. Further we use that ${\pi }_k$ is a deterministic policy, therefore $M_{{\pi }_k}P(S_h^{(k)})=P_{A_h^{(k)}}(S_h^{(k)})$. The second follows from the definition of the confidence set in the previous lemma:

$$\begin{array}{rl}& \Vert P_{A_h^{(k)}}^{\ast }(S_h^{(k)})-{\tilde{P}}_{A_h^{(k)}}^{(k)}(S_h^{(k)}){\Vert }_1\\ & \le \Vert P_{A_h^{(k)}}^{\ast }(S_h^{(k)})-P_{A_h^{(k)}}^{(k)}(S_h^{(k)}){\Vert }_1+\Vert P_{A_h^{(k)}}^{(k)}(S_h^{(k)})-{\tilde{P}}_{A_h^{(k)}}^{(k)}(S_h^{(k)}){\Vert }_1\\ & \le 2{\beta }_{\delta }(N_k(S_h^{(k)},A_h^{(k)}))\end{array}$$

Telescoping and using that ${\delta }_H^{(k)}=0$ yields

$${\delta }_0^{(k)}\le {\eta }_1^{(k)}+\cdots +{\eta }_{H-1}^{(k)}+2H\underset{\text{(IV)}}{\underbrace{\sum _{h=0}^{H-1}{\beta }_{\delta }(N_k(S_h^{(k)},A_h^{(k)}))}}$$

Note that $({\eta }_h^{(k)}{)}_{h=1}^{H-1}$ is another martingale difference sequence (with ${\eta }_h^{(k)}|\le H$) that can be bounded by Azuma-Hoeffding:

$$\sum _{k=1}^K\sum _{h=1}^{H-1}{\eta }_h^{(k)}\le 2H\sqrt{\frac{HK}{2}\log (1/\delta )}$$

It remains to bound term $\text{(IV)}$. For this we make use of the following algebraic lemma:

---

Lemma:

For any sequence $m_1,\dots ,m_k$ that satisfies $m_1+\cdots +m_k\ge 0$:

$$\sum _{k=1}^K\frac{m_k}{\sqrt{1\vee (m_1+\cdots +m_k)}}\le 2\sqrt{m_1+\cdots +m_k}$$

---

Proof of Lemma: Let $f(x)=1/\sqrt{x}$. $f(x)$ is a concave function on $(0,\mathrm{\infty })$. Therefore $f(A+x)\le f(A)+x{f}^{\prime }(A)$ for all $A,A+x,>0$. This translates to:

$$\sqrt{A+x}\le \sqrt{A}+\frac{x}{2\sqrt{A}}$$

The claim follows from telescoping. $◼$

Continuing the proof of the theorem where we need to bound $\text{(IV)}$. Denote $c_{\delta }=\sqrt{2S\log (2)+\log (HK(HK+1)SA/\delta )}$. Further let $M_k(s,a)=\sum _{h=1}^{H-1}\mathbb{I}(S_h^{(k)}=s,A_h^{(k)}=a)$ and note that $N_k(s,a)=M_1+\cdots +M_{k-1}$. Then

$$\begin{array}{rl}\sum _{k=1}^K\sum _{h=0}^{H-1}{\beta }_{\delta }(N_k(S_h^{(k)},A_h^{(k)}))& \le c_{\delta }\sum _{s,a}\sum _{k=1}^K\sum _{h=0}^{H-1}\frac{\mathbb{I}(S_h^{(k)}=s,A_h^{(k)}=a)}{\sqrt{1\vee N_k(s,a)}}\\ & =c_{\delta }\sum _{s,a}\sum _{k=1}^K\frac{M_k}{\sqrt{1\vee (M_1+\cdots +M_{k-1})}}\end{array}$$

Next, using the algebraic lemma above and the fact that $M_k(s,a)\le H$, we find

$$\begin{array}{rl}\sum _{k=1}^K\sum _{h=0}^{H-1}{\beta }_{\delta }(N_k(S_h^{(k)},A_h^{(k)}))& \le c_{\delta }\sum _{s,a}\sum _{k=1}^K\frac{M_k(s,a)}{\sqrt{1\vee (M_1(s,a)+\cdots +M_{k-1}(s,a))}}\\ & \le c_{\delta }\sum _{s,a}\sum _{k=1}^K\frac{M_k(s,a)}{\sqrt{1\vee (M_1(s,a)+\cdots +M_k(s,a)-H)}}\\ & \le c_{\delta }\sum _{s,a}\sum _{k=1}^K\frac{M_k(s,a)\mathbb{I}(M_1(s,a)+\cdots +M_k(s,a)>H)}{\sqrt{M_1(s,a)+\cdots +M_k(s,a)-H}}+c_{\delta }HSA\\ & \le 2c_{\delta }\sum _{s,a}\sqrt{N_k(s,a)}+c_{\delta }HSA\\ & \le 2c_{\delta }SA\sqrt{\sum _{s,a}{N}_k(s,a)/SA}+c_{\delta }HSA\\ & =2c_{\delta }\sqrt{SAHK}+c_{\delta }HSA\end{array}$$

The last inequality uses Jensen’s inequality.

Collecting all terms and taking the union bound over two applications of Azuma-Hoeffdings and the event $\mathcal{E}$ completes the proof. $◼$

Unknown reward functions

In our analysis of UCRL we assumed that the reward function is known. While this is quite a common assumption in the literature, it is mainly for simplicity. We also don’t expect the bounds to change by much: Estimating the rewards is not harder than estimating the transition kernels.

To modify the analysis and account for unkown rewards, we first consider the case with deterministic reward function $r_a(s)\in [0,R_{max}]$, where $R_{max}$ is some known upper bound on the reward per step.

Embracing the idea of optimism, we define reward estimates

$${\hat{r}}_a^{(k)}(s)=\{\begin{array}{ll}{r}_{A_h^{(k^{\prime })}}(S_h^{(k^{\prime })})& \text{(s,a) was visited in a round }{k}^{\prime }<k\text{ and step }h\\ R_{max}& \text{else.}\end{array}$$

Clearly this defines an optimistic estimate, ${\hat{r}}_a^{(k)}(s)\ge r_a(s)$. Moreover, we have ${\hat{r}}_{A_h^{(k)}}^{(k)}(S_h^{(k)})\ne r_{A_h^{(k)}}(S_h^{(k)})$ at most $SA$ times. Therefore the regret in the previous analysis is increased by at most $R_{max}SA$.

When the reward is stochastic, we can use a maximum likelihood estimate of the reward and construct confidence bounds around the estimate. This way we can define an optimistic reward. Still not much changes, as the reward estimates concentrate at the same rate as the estimates of $P$.

UCBVI: Upper Confidence Bound Value Iteration

Computing the UCRL policy can be quite challenging. However, we can relax the construction so that we can use backward induction. We define a time-inhomogenous relaxation of the confidence set:

$$C_{k,\delta }^H=\underset{H\text{ times}}{\underbrace{C_{k,\delta }\times \cdots \times C_{k,\delta }}}$$

Let ${\tilde{P}}_{1:H,k}:=({\tilde{P}}_{1,k},\dots ,{\tilde{P}}_{H,k})=\arg \underset{P\in C_{k,\delta }^H}{max}{v}_P^{\ast }(s_0^{(k)})$ be the optimistic (time-inhomogenous) transition matrices and ${\pi }_k=\arg \underset{\pi }{max}{v}_{{\tilde{P}}_{1:H,k}}^{\pi }$ the optimal policy for the optimistic model ${\tilde{P}}_{1:H,k}$. Then $v_{{\tilde{P}}_{1:H,k}}^{{\pi }^k}=v_{{\tilde{P}}_{1:H,k}}^{\ast }=v^{(k)}$ is defined by the following backwards induction:

$$\begin{array}{rl}{v}_H^{(k)}(s)& =0\mathrm{\forall }s\in [S]\\ Q_h^{(k)}(s,a)& =r(s,a)+\underset{P\in C_{k,\delta }}{max}{P}_a(s)v_{h+1}^{(k)}\\ v_h^{(k)}(s)& =\underset{a}{max}{Q}_h^{(k)}(s,a)\end{array}$$

Note that the maximum in the second line is a linear optimization with convex constraints that can be solved efficiently. Further, the proof of the UCRL regret still applies, because we used the same (step-wise) relaxation in the analysis.

We can further relax the backward induction to avoid the optimization over $C_{k,\delta }$ completely:

$$\begin{array}{rl}\underset{P\in C_{k,\delta }}{max}{P}_a(s)v_{h+1}^{(k)}& \le P_a^{(k)}(s)v_{h+1}^{(k)}+\underset{P\in C_{k,\delta }}{max}(P_a(s)-P_a^{(k)}(s))v_{h+1}^{(k)}\\ & \le P_a^{(k)}(s)v_{h+1}^{(k)}+\underset{P\in C_{k,\delta }}{max}\Vert P_a(s)-P_a^{(k)}(s)){\Vert }_1\Vert v_{h+1}^{(k)}{\Vert }_{\mathrm{\infty }}\\ & \le P_a^{(k)}(s)v_{h+1}^{(k)}+{\beta }_{\delta }(N_k(s,a))H\end{array}$$

This leads us to the the UCBVI (upper confidence bound value iteration) algorithm. In episode $k$, UCBVI uses value iteration for the estimated transition kernel $P_a^{(k)}(s)$ and optimistic reward function $r_a(s)+H{\beta }_{\delta }(N_k(s,a))$ to compute the policy.

---

UCBVI (Upper confidence bound value iteration):

In episodes $k=1,\dots ,K$,

1. Compute optimistic value function:

$$\begin{array}{rl}{v}_H^{(k)}(s)& =0\mathrm{\forall }s\in [S]\\ b_k(s,a)& =H{\beta }_{\delta }(N_k(s,a))\\ Q_h^{(k)}(s,a)& =min(r(s,a)+b_k(s,a)+P_a^{(k)}(s)v_{h+1}^{(k)},H)\\ v_h^{(k)}(s)& =\underset{a}{max}{Q}_h^{(k)}(s,a)\end{array}$$

1. Follow greedy policy $A_h^{(k)}=\arg \underset{A}{max}{Q}_h^{(k)}(S_h^{(k)},A)$
2. Observe episode data $S_0^{(k)},A_0^{(k)},S_1^{(k)},\dots ,S_{H-1}^{(k)},S_{H-1}^{(k)},S_H^{(k)}$

---

Note that we truncate the $Q_h^{(k)}$-function to be at most $H$, this avoids a blow up by a factor of $H$ in the regret bound. Carefully checking that the previous analysis still applies shows that UCBVI has regret at most $R_K\le \mathcal{O}(H^{2}S\sqrt{AK})$.

By more carefully designing the reward bonuses for UCBVI, it is possible to achieve $R_K\le \stackrel{\mathcal{~}}{\mathcal{O}}(H^{3/2}\sqrt{SAK})$ which matches the lower bound up to logarithmic factors in the time in-homogeneous setting.

Notes

References

The original UCRL paper. Notice that they consider the infinite horizon average reward setting, which is different from the episodic setting we present.

Auer, P., & Ortner, R. (2006). Logarithmic online regret bounds for undiscounted reinforcement learning. Advances in neural information processing systems, 19. [link]

The UCBVI paper. Notice that they consider the homogeneous setting, which is different from the in-homogeneous setting we present.

Azar, M. G., Osband, I., & Munos, R. (2017, July). Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning (pp. 263-272). PMLR. [link]

The paper that presents the lower bound. Notice the they consider the infinite horizon average reward setting. Thus, there results contains a diameter term $D$ instead of a horizon term of $H$.

Auer, P., Jaksch, T., & Ortner, R. (2008). Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21. [link]

---