14. Politex

PDF Version

The following lemma can be extracted from the calculations found at the
end of the last lecture:

---

Lemma (Mixture policy suboptimality): Fix an MDP $M$. For any sequence ${\pi }_0,\dots ,{\pi }_{k-1}$ of policies, any sequence ${\hat{q}}_0,\dots ,{\hat{q}}_{k-1}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ of functions, and any policy ${\pi }^{\ast }$, the mixture policy ${\overline{\pi }}_k=1/k({\pi }_0+\cdots +{\pi }_{k-1})$ satisfies

$$\begin{array}{}\text{(1)}& v^{{\pi }^{\ast }}-v^{{\overline{\pi }}_k}& \le \frac{1}{k}(I-\gamma P_{{\pi }^{\ast }}{)}^{-1}\underset{T_1}{\underbrace{\sum _{j=0}^{k-1}{M}_{{\pi }^{\ast }}{\hat{q}}_j-M_{{\pi }_j}{\hat{q}}_j}}+\frac{2\underset{0\le j\le k-1}{max}\Vert q^{{\pi }_j}-{\hat{q}}_j{\Vert }_{\mathrm{\infty }}}{1-\gamma }.\end{array}$$

---

In particular, the only restriction is on policy ${\pi }^{\ast }$ so far and that is that it has to be a memoryless policy. To control the suboptimality of the mixture policy, one just needs to control the action-value approximation errors $\Vert q^{{\pi }_j}-{\hat{q}}_j{\Vert }_{\mathrm{\infty }}$ and the term $T_1$ and for this we are free to choose the policies ${\pi }_0,\dots ,{\pi }_{k-1}$ in any way we want them to be chosen. To help with this choice, let us now inspect $T_1(s)$ for a fixed state $s$:

$$\begin{array}{}\text{(2)}& T_1(s)=\sum _{j=0}^{k-1}⟨{\pi }^{\ast }(s,\cdot ),{\hat{q}}_j(s,\cdot )⟩-⟨{\pi }_j(s,\cdot ),{\hat{q}}_j(s,\cdot )⟩,\end{array}$$

where, abusing notation, we use $\pi (s,a)$ for $\pi (a|s)$. Now, recall that ${\hat{q}}_j$ will be computed based on ${\pi }_j$ while ${\pi }^{\ast }$ is unknown. One must thus wonder whether it is possible to control this term?

Online linear optimization

As it happens, the problem of controlling terms of this type is the central problem studied in a subfield of learning theory, online learning. In particular, in online linear optimization, the following problem is studied:

An adversary and a learner are playing a zero-sum minimax game in $k$ discrete rounds, taking actions in an alternating manner. In round $j$ ($0\le j\le k-1$), first, the learner needs to choose a vector $x_j\in \mathcal{X}\subset {\mathbb{R}}^d$. Then, the adversary chooses a vector, $y_j\in \mathcal{Y}\subset {\mathbb{R}}^d$. Before its choice, the adversary learns about all previous choices of the learner, and the learner also learns about all previous choices of the adversary. They also remember their own choices. For simplicity, let us constraint the adversary and the learner to be deterministic. The payoff to the adversary at the end of the $k$ rounds is

$$\begin{array}{}\text{(3)}& R_k=\underset{x\in \mathcal{X}}{max}\sum _{j=0}^{k-1}⟨x,y_j⟩-⟨x_j,y_j⟩.\end{array}$$

In particular, the adversary’s goal is maximize this, while the learner’s goal is to minimize this (the game is zero-sum). Both the adversary and the learner are given $k$ and the sets $\mathcal{X},\mathcal{Y}$. Letting $L$ to denote the learner’s strategy (a sequence of maps of histories to $\mathcal{X}$) and $A$ to denote the adversary’s strategy (a sequence of maps of histories to $\mathcal{Y}$), the above quantity depends on $L$ and $A$: $R_k=R_K(A,L)$.

Taking the perspective of the learner, the quantity defined in $\text{(3)}$ is called the learner’s regret. Denote the minimax value of the game by $R_k^{\ast }$: $R_k^{\ast }=\underset{L}{inf}\underset{A}{sup}{R}_k(A,L)$.

Thus, this only depends on $k$, $\mathcal{X}$ and $\mathcal{Y}$. The dependence is suppressed when it is clear from the context. The central question then is how $R_k^{\ast }$ depends on $k$ and also on $\mathcal{X}$ and $\mathcal{Y}$. In online linear optimization both sets $\mathcal{X}$ and $\mathcal{Y}$ are convex.

Connecting these games to our problem, we can see that $T_1(s)$ in $\text{(2)}$ matches the regret definition in $\text{(3)}$ if we let $d=\mathrm{A}$, $\mathcal{X}={\mathcal{M}}_1(\mathrm{A})=\{p\in [0,1{]}^{\mathrm{A}}:\sum _a{p}_a=1\}$ be the $\mathrm{A}-1$ simplex of ${\mathbb{R}}^{\mathrm{A}}$ and $\mathcal{Y}=[0,1/(1-\gamma ){]}^{\mathrm{A}}$. Furthermore, ${\pi }_j(s,\cdot )$ needs to be chosen first, which is followed by the choice of ${\hat{q}}_j(s,\cdot )$. While ${\hat{q}}_j(s,\cdot )$ will not be chosen in an adversarial fashion, a bound $B$ on the regret against arbitrary choices will also serve as a bound for the specific choice we will need to make for ${\hat{q}}_j(s,\cdot )$.

Mirror descent

Mirror descent (MD) is an algorithm that originates in optimization theory. In the context of online linear optimization, MD is a strategy for the learner which is known to guarantee near minimax regret for the learner under a wide range of circumstances.

To align with the large body of literature on online linear optimization, it will be beneficial to switch signs. Thus, in what follows we assume that the learner will aim at minimizing $⟨x,y⟩$ by its choice $x\in \mathcal{X}$ and the adversary will aim at maximizing the same expression over its choice $y\in \mathcal{Y}$. This means that we also redefine the regret to

$$\begin{array}{rl}{R}_k& =\underset{x\in \mathcal{X}}{max}\sum _{j=0}^{k-1}⟨x_j,y_j⟩-⟨x,y_j⟩\\ \text{(4)}& & =\sum _{j=0}^{k-1}⟨x_j,y_j⟩-\underset{x\in \mathcal{X}}{min}\sum _{j=0}^{k-1}⟨x,y_j⟩.\end{array}$$

Everything else remains the same: The game is zero-sum, minimax, the regret is the payoff for the adversary and the negative regret is the payoff of the learner. This version is called a loss-game. The reason to prefer the loss game is because most of optimization theory is written for minimizing convex functions rather than for maximizing concave functions. However, clearly, this is an arbitrary choice. The second form of the regret shows that the player’s goal is to compete with the best single decision from $\mathcal{X}$ but chosen given the hindsight of knowing all the choices of the adversary. That is, the learner’s goal is to keep its cumulative loss $\sum _{j=0}^{k-1}⟨x_j,y_j⟩$ close to, or even below the best cumulative loss in hindsight, $\underset{x\in \mathcal{X}}{min}\sum _{j=0}^{k-1}⟨x,y_j⟩$. (With this, $T_1(s)$ matches $R_k$ when we change $\mathcal{Y}=[-1/(1-\gamma ),0{]}^{\mathrm{A}}$.)

MD is recursively defined and in its simplest form it has two design parameters. The first is an extended real-valued convex function $F:{\mathbb{R}}^d\to \overline{\mathbb{R}}$, called the “regularizer”, while the second is a stepsize, or learning rate parameter $\eta >0$. (The extended reals is just $\mathbb{R}$ together with $+\mathrm{\infty },-\mathrm{\infty }$ and an appropriate extension of basic arithmetic. By allowing convex functions to take the value $+\mathrm{\infty }$ allows to merge “constraints” with objectives in a seamless fashion. The value $-\mathrm{\infty }$ is added because sometimes we have to work with negated extended real-valued convex functions.)

The specification of MD is as follows: In round $0$, $x_0\in \mathcal{X}$ is picked to minimize $F$:

$$x_0=\arg \underset{x\in \mathcal{X}}{min}F(x).$$

In what follows, we assume that all the minimizers that we need in the definition of MD do exist. In the specific case that we need, $\mathcal{X}$ is the $d-1$ simplex, which is a closed convex set, and since convex functions are also continuous, the minimizers that we will need are guaranteed to exist.

Then, in round $j>0$, MD chooses $x_j$ as follows:

$$\begin{array}{}\text{(5)}& \begin{array}{rl}{x}_j& =\arg \underset{x\in \mathcal{X}}{min}\eta ⟨x,y_{j-1}⟩+D_F(x,x_{j-1})\end{array}\end{array}$$

Here,

$$D_F(x,x^{\prime })=F(x)-(F(x^{\prime })+⟨\mathrm{\nabla }F(x^{\prime }),x-x^{\prime }⟩)$$

is the remainder term in the first-order Taylor-series expansion of the value of $F$ at $x$ when the expansion is carried out at $x^{\prime }$ and, for simplicity, we assume that $F$ is differentiable on the interior of its domain $\text{dom}(F)=\{x\in \mathbb{R}:F(x)<+\mathrm{\infty }\}$. Since for any convex function and any linear approximation of it stays below the graph of the convex function, we immediately get that $D_F$ is nonnegative valued. For an illustration see the figure on the right, which shows a convex function, the first-order Taylor approximation of the function at some point.

One should think of $F$ is a “nonlinear distance inducing function”; above $D_F(x,x^{\prime })$ can be thought of penalty imposed on deviating from $x^{\prime }$. However, $D_F$ is more often than not is not a distance, i.e., often it is not even symmetric. Because of this, we can’t really call $D_F$ a distance. Hence, it is called a divergence. In particular, $D_F(x,x^{\prime })$ is called the Bregman divergence of $x$ from $x^{\prime }$.

In the definition of the MD update rule, we tacitly assumed that $D_F(x,x_{j-1})$ is well-defined. This requires that $F$ should be differentiable at $x_{j-1}$, which one needs to check when applying MD. In our specific case, this will hold, again.

The idea of the MD update rule is to (1) allow the learner to react to the last loss $y_{j-1}$ vector chosen by the adversary, while also (2) limiting how much $x_j$ can depart from $x_{j-1}$, thus, effectively stabilizing the algorithm, the tradeoff governed by the choice of $\eta >0$. (Separating $\eta$ from $F$ only makes sense because there are some standard choices for $F$, but $\eta$ is really just a scale parameter for $F$). In particular, the larger the value of $\eta$ is, the less “data-sensitive” MD will be (here, $y_0,\dots ,y_{k-1}$ constitute the data), and vice versa, the smaller $\eta$ is, the more data-sensitive MD will be.

Where is the mirror?

Under some technical conditions on $F$, the update rule $\text{(5)}$ has a two step-implementation:

$$\begin{array}{}\text{(6)}& {\tilde{x}}_j& =(\mathrm{\nabla }F{)}^{-1}(\mathrm{\nabla }F(x_{j-1})-\eta y_{j-1}),\text{(7)}& x_j& =\arg \underset{x\in \mathcal{X}}{min}{D}_F(x,{\tilde{x}}_j).\end{array}$$

The first equation above explains the name: To obtain ${\tilde{x}}_j$, one first transforms $x_{j-1}$ using $\mathrm{\nabla }F:\text{dom}(\mathrm{\nabla }F)\to {\mathbb{R}}^d$ to the “mirror” (dual) space where “gradients”/”slopes live”, where one then adds to the result $-\eta y_{j-1}$, which can be seen as a “gradient step” (interpreting $y_{j-1}$ as the gradient of some loss). Finally, the result is then mapped back to the original (primal) space using the inverse of $\mathrm{\nabla }F$.

The second step of the update takes the resulting point ${\tilde{x}}_j$ and “projects” it to $\mathcal{X}$ in a way that respects the “geometry induced by $F$” on the space ${\mathbb{R}}^d$.

The use of complex terminology, like “primal” and “dual” spaces, which happen to be the same old Euclidean space, ${\mathbb{R}}^d$, probably sounds like an overkill. Indeed, in the simple case we consider when these spaces are identical it is. The distinction would become important when working with infinite dimensional spaces, which we leave to others for now.

Besides helping with understanding the terminology, the two-step update shown can also be useful for computation. In fact, this will be the case in the special case that we need.

Mirror descent on the simplex

We have seen that in the special case we need,

$$\begin{array}{rl}\mathcal{X}& ={\mathcal{P}}_{d-1}:=\{p\in [0,1{]}^d:\sum _a{p}_a=1\},\\ \mathcal{Y}& =[-1/(1-\gamma ),0{]}^d,\text{and}\\ d& =\mathrm{A}.\end{array}$$

To use MD we need to specify the regularizer $F$ and the learning rate. For the former, we choose

$$F(x)=\sum _i{x}_i\log (x_i)-x_i,$$

which is known as the unnormalized negentropy function. Note that $F$ takes on finite values when $x\in [0,\mathrm{\infty }{]}^d$ (since $\underset{x\to 0+}{lim}x\log (x)=0$, we set $x_i\log (x_i)=0$ whenever $x_i=0$). Outside of this quadrant, we define the value of $F$ to be $+\mathrm{\infty }$. The plot of $x\log (x)-x$ for $x\ge 0$ is shown on the right.

It is not hard to verify that $F$ is convex: First, $\text{dom}(F)=[0,\mathrm{\infty }{]}^d$ is convex. Taking the first derivative, we find that for any $x\in (0,\mathrm{\infty }{)}^d$,

$$\mathrm{\nabla }F(x)=\log (x),$$

where $\log$ is applied componentwise. Taking the derivative again, we find that for $x\in (0,\mathrm{\infty }{)}^d$,

$${\mathrm{\nabla }}^{2}F(x)=\text{diag}(1/x),$$

i.e., the matrix whose $(i,i)$th diagonal entry is $1/x_i$. Clearly, this is a positive definite matrix, which suffices to verify that $F$ is a convex function.

The Bregman divergence induced by $F$ is

$$\begin{array}{rl}{D}_F(x,x^{\prime })& =⟨\mathbf{1},x\log (x)-x-x^{\prime }\log (x^{\prime })+x^{\prime }⟩-⟨\log (x^{\prime }),x-x^{\prime }⟩\\ & =⟨\mathbf{1},x\log (x/x^{\prime })-x+x^{\prime }⟩,\end{array}$$

where again we use an “intuitive” notation when operations are first applied componentwise (i.e., $x\log (x)$ denotes a vector whose $i$th component is $x_i\log (x_i)$). Note that the domain of $D_F$ is $[0,\mathrm{\infty }{)}^d\times (0,\mathrm{\infty }{)}^d$. If both $x$ and $x^{\prime }$ lie in the $d-1$-simplex, $D_F$ becomes the well-known relative entropy, or Kullback-Leibler (KL) divergence.

It is not hard to verify that $x_j$ can be obtained as shown in $\text{(6)}$-$\text{(7)}$ and in particular this two-step update takes the form

$$\begin{array}{rl}{\tilde{x}}_{j,i}& =x_{j-1,i}\exp (-\eta y_{j-1,i}),x_{j,i}=\frac{{\tilde{x}}_{j,i}}{\sum _{i^{\prime }}{\tilde{x}}_{j,i^{\prime }}},i\in [d].\end{array}$$

Unrolling the recursion, we can also that this is the same as

$$\begin{array}{}\text{(8)}& {\tilde{x}}_{j,i}=\exp (-\eta (y_{0,i}+\cdots +y_{j-1,i})),x_{j,i}=\frac{{\tilde{x}}_{j,i}}{\sum _{i^{\prime }}{\tilde{x}}_{j,i^{\prime }}},i\in [d].\end{array}$$

Based on this, it is obvious that MD can be efficiently implemented with this choice of $F$. As far as the regret is concerned, the following theorem holds:

---

Theorem (MD with negentropy on the simplex): Let $\mathcal{X}={\mathcal{P}}_{d-1}$ amd $\mathcal{Y}=[0,1{]}^d$. Then, no matter the adversary, a learner using MD with

$$\eta =\sqrt{\frac{2\log (d)}{k}}$$

is guaranteed that its regret $R_k$ in $k$ rounds is at most

$$R_k\le \sqrt{2k\log (d)}.$$

---

When the adversary plays in $\mathcal{Y}=[a,b{]}^d$ with $a<b$, we can use MD on the transformed sequence ${\tilde{y}}_j=(y_j-a\mathbf{1})/(b-a)\in [0,1{]}^d$. Then, for any $x\in \mathcal{X}$,

$$\begin{array}{rl}{R}_k(x)& :=\sum _{j=0}^{k-1}⟨x_j-x,y_j⟩\\ & =\sum _{j=0}^{k-1}⟨x_j-x,(b-a){\tilde{y}}_j+a\mathbf{1}⟩\\ & =(b-a)\sum _{j=0}^{k-1}⟨x_j-x,{\tilde{y}}_j⟩\\ & \le (b-a)\sqrt{2k\log (d)},\end{array}$$

where the third equality used that $⟨x_j,\mathbf{1}⟩=⟨x,\mathbf{1}⟩=1$. Taking the maximum over $x\in \mathcal{X}$ gives that

$$\begin{array}{}\text{(9)}& R_k\le (b-a)\sqrt{2k\log (d)}.\end{array}$$

By the update rule in $\text{(8)}$,

$$\begin{array}{r}{\tilde{x}}_{j,i}=\exp (-\eta ({\tilde{y}}_{0,i}+\cdots +{\tilde{y}}_{j-1,i}))=\exp (-\eta /(b-a)(y_{0,i}+\cdots +y_{j-1,i}-jb)),i\in [d].\end{array}$$

Note that the “shift” by $-jb$ cancels out in the normalization step. Hence, MD in this case takes the form

$$\begin{array}{}\text{(10)}& \begin{array}{rl}{\tilde{x}}_{j,i}& =\exp (-\eta /(b-a)(y_{0,i}+\cdots +y_{j-1,i})),x_{j,i}=\frac{{\tilde{x}}_{j,i}}{\sum _{i^{\prime }}{\tilde{x}}_{j,i^{\prime }}},i\in [d],\end{array}\end{array}$$

which is the same as before, except that the learning rate is scaled by $1/(b-a)$. In particular, in this case one can set

$$\begin{array}{}\text{(11)}& \eta =\frac{1}{b-a}\sqrt{\frac{2\log (d)}{k}}.\end{array}$$

and use update rule $\text{(8)}$.

MD applied to MDP planning

As agreed, $T_1(s)$ from $\text{(2)}$ takes the form of a $k$-round regret against ${\pi }^{\ast }(s,\cdot )$ in online linear optimization on the simplex with losses in $[-1/(1-\gamma ),0{]}^{\mathrm{A}}$. This suggest to use MD in a state-by-state manner to control $T_1(s)$. Using $\text{(8)}$ and $\text{(11)}$ gives

$$E_j(s,a)=\exp (\eta ({\hat{q}}_0(s,a)+\cdots +{\hat{q}}_{j-1}(s,a))),{\pi }_j(a|s)=\frac{E_j(s,a)}{\sum _{a^{\prime }}{E}_j(s,a^{\prime })},a\in \mathcal{A}$$

to be used with

$$\eta =(1-\gamma )\sqrt{\frac{2\log (\mathrm{A})}{k}}.$$

Note that this is the update used by Politex. Then, $\text{(9)}$ gives that simultaneously for all $s\in \mathcal{S}$,

$$\begin{array}{}\text{(12)}& |T_1(s)|\le \frac{1}{1-\gamma }\sqrt{2k\log (\mathrm{A})}.\end{array}$$

Putting things together, we get the following result:

---

Theorem (Politex suboptimality gap bound): Pick a featurized MDP $(M,\varphi )$ with a full rank feature-map $\phi :\mathcal{S}\times \mathcal{A}\to {\mathbb{R}}^d$ and let $K,m,H\ge 1$. Assume that B2${}_{\epsilon }$ holds for $(M,\varphi )$ and the rewards in $M$ are in the $[0,1]$ interval. For $0\le \zeta <1$, define

$$\kappa (\zeta )=\epsilon (1+\sqrt{d})+\sqrt{d}(\frac{{\gamma }^H}{1-\gamma }+\frac{1}{1-\gamma }\sqrt{\frac{\log (d(d+1)K/\zeta )}{2m}}),$$

Then, in $K$ iterations, Politex produces a mixed policy ${\overline{\pi }}_K$ such that with probability $1-\zeta$, the suboptimality gap $\delta$ of ${\overline{\pi }}_K$ satisfies

$$\begin{array}{r}\delta \le \frac{1}{(1-\gamma {)}^2}\sqrt{\frac{2\log (\mathrm{A})}{K}}+\frac{2\kappa (\zeta )}{1-\gamma }.\end{array}$$

In particular, for any ${\epsilon }^{\prime }>0$, choosing $K,H,m$ so that

$$\begin{array}{rl}K& \ge \frac{32\log (A)}{(1-\gamma {)}^4({\epsilon }^{\prime }{)}^2},\\ H& \ge H_{\gamma ,(1-\gamma ){\epsilon }^{\prime }/(8\sqrt{d})}\text{and}\\ m& \ge \frac{32d}{(1-\gamma {)}^4({\epsilon }^{\prime }{)}^2}\log ((d+1{)}^{2}K/\zeta ),\end{array}$$

policy ${\pi }_K$ is $\delta$-optimal with

$$\begin{array}{r}\delta \le \frac{2(1+\sqrt{d})}{1-\gamma }\epsilon +{\epsilon }^{\prime },\end{array}$$

while the total computation cost is $\text{poly}(\frac{1}{1-\gamma },d,\mathrm{A},\frac{1}{({\epsilon }^{\prime }{)}^2},\log (1/\zeta ))$.

---

Note that as compared to the result of LSPI with G-optimal design, the amplification of the approximation error $\epsilon$ is reduced by a factor of $1/(1-\gamma )$, as it was promised. The price is that now the number of iterations $K$, is a polynomial of $\frac{1}{(1-\gamma ){\epsilon }^{\prime }}$, whereas before it was logarithmic. This suggest that perhaps a higher learning rate can help initially to speed up convergence to get the best of both words.

Proof: As in the proof of the suboptimality gap for LSPI, we get that for any $0\le \zeta \le 1$, with probability at least $1-\zeta$, for any $0\le k\le K-1$,

$$\begin{array}{rl}\Vert q^{{\pi }_k}-{\hat{q}}_k{\Vert }_{\mathrm{\infty }}=\Vert q^{{\pi }_k}-\mathrm{\Pi }\mathrm{\Phi }{\hat{\theta }}_k{\Vert }_{\mathrm{\infty }}\le \Vert q^{{\pi }_k}-\mathrm{\Phi }{\hat{\theta }}_k{\Vert }_{\mathrm{\infty }}& \le \kappa (\zeta ),\end{array}$$

where the first inequality uses that $q_{{\pi }_k}$ takes values in $[0,1]$. On the event when the above inequalities hold, by $\text{(1)}$ and $\text{(12)}$,

$$\begin{array}{r}\delta \le \frac{1}{(1-\gamma {)}^2}\sqrt{\frac{2\log (\mathrm{A})}{K}}+\frac{2\kappa (\zeta )}{1-\gamma }.\end{array}$$

The details of this calculation are left to the reader. $◼$

Notes

Optimality of the Final Policy

Notice that we said the policy returned by Politex after $k$ iterations should be a mixture policy ${\overline{\pi }}_k=\frac{1}{k}({\pi }_0+\cdots +{\pi }_{k-1})$. A more natural policy to return is the final policy ${\pi }_{k-1}$. The question then is: can one ensure similar optimality gaurantees for the final policy ${\pi }_{k-1}$ as we have seen for ${\overline{\pi }}_k$? The answer turns out to be yes, if we use the unnormalized negentropy regularizer for mirror descent (as we have already been using in this lecture note). To see this, we aim to bound $\Vert v^{{\pi }^{\ast }}-v^{{\pi }_{k-1}}{\Vert }_{\mathrm{\infty }}$. We begin by writing.

$$\begin{array}{rl}{v}^{{\pi }^{\ast }}-v^{{\pi }_{k-1}}& =v^{{\pi }^{\ast }}-v^{{\overline{\pi }}_k}+v^{{\overline{\pi }}_k}-v^{{\pi }_{k-1}}\\ & =\frac{1}{k}(I-\gamma P_{{\pi }^{\ast }}{)}^{-1}\sum _{j=0}^{k-1}{M}_{{\pi }^{\ast }}{q}^{{\pi }_j}-M_{{\pi }_j}{q}^{{\pi }_j}\\ & +\frac{1}{k}(I-\gamma P_{{\pi }_{k-1}}{)}^{-1}\sum _{j=0}^{k-1}{M}_{{\pi }_j}{q}^{{\pi }_j}-M_{{\pi }_{k-1}}{q}^{{\pi }_j}\\ & =\frac{1}{k}(I-\gamma P_{{\pi }^{\ast }}{)}^{-1}\underset{T_1}{\underbrace{\sum _{j=0}^{k-1}{M}_{{\pi }^{\ast }}{\hat{q}}_j-M_{{\pi }_j}{\hat{q}}_j}}\\ & +\underset{T_2}{\underbrace{\frac{1}{k}(I-\gamma P_{{\pi }^{\ast }}{)}^{-1}\sum _{j=0}^{k-1}(M_{{\pi }^{\ast }}-M_{{\pi }_j})(q^{{\pi }_j}-{\hat{q}}_j)}}\\ & +\frac{1}{k}(I-\gamma P_{{\pi }_{k-1}}{)}^{-1}\underset{T_3}{\underbrace{\sum _{j=0}^{k-1}{M}_{{\pi }_j}{\hat{q}}_j-M_{{\pi }_{k-1}}{\hat{q}}_j}}\\ & +\underset{T_4}{\underbrace{\frac{1}{k}(I-\gamma P_{{\pi }_{k-1}}{)}^{-1}\sum _{j=0}^{k-1}(M_{{\pi }_j}-M_{{\pi }_{k-1}})(q^{{\pi }_j}-{\hat{q}}_j)}}.\end{array}$$

Notice how $T_1$ and $T_2$ are defined as before, and we already have bounds for both of them. It is also easy to see that that $T_4$ takes a very similar form to $T_2$ and can also be bounded in the same way as $T_2$. If we can show that $T_3(s)\le 0$ for all $s\in \mathcal{S}$ then we would get the result that

$$\begin{array}{r}\Vert v^{{\pi }^{\ast }}-v^{{\pi }_{k-1}}{\Vert }_{\mathrm{\infty }}\le \delta \le \frac{1}{(1-\gamma {)}^2}\sqrt{\frac{2\log (\mathrm{A})}{K}}+\frac{4\kappa (\zeta )}{1-\gamma }.\end{array}$$

Which is identical to the result of the main theorem in the lecture note above, except with the constant $2$ scaling replaced with a constant $4$ scaling infront of the approximation error (since $T_4$ used the same bound as $T_2$).

We are left to show that indeed $T_3(s)\le 0$. To do this we first write out $T_3(s)$ in vector notation to help us aline with the math syntax to come. Fix a state $s\in \mathcal{S}$, then

$$T_3(s)=\sum _{j=0}^{k-1}⟨{\pi }_j(\cdot |s),{\hat{q}}_j(s,\cdot )⟩-⟨{\pi }_{k-1}(\cdot |s),{\hat{q}}_j(s,\cdot )⟩$$

Since we will hold $s$ fixed for all the following steps we slightly abuse notation in favor of avoiding clutter and write the above equation as follows where it is assumed that all functions were first evaluated at $s$.

$$T_3=\sum _{j=0}^{k-1}⟨{\pi }_j,{\hat{q}}_j⟩-⟨{\pi }_{k-1},{\hat{q}}_j⟩$$

Next recall that the policy selected by MD at iteration $k$ is defined as

$$\begin{array}{rl}{\pi }_k& =\arg \underset{\pi \in {\mathcal{M}}_1(A)}{min}\eta ⟨\pi ,-{\hat{q}}_{k-1}⟩+D_F(\pi ,{\pi }_{k-1})\\ & =\arg \underset{\pi \in {\mathcal{M}}_1(A)}{max}\eta ⟨\pi ,{\hat{q}}_{k-1}⟩-D_F(\pi ,{\pi }_{k-1})\end{array}$$

where we have negated ${\hat{q}}_{k-1}$ to formulate our problem as a minimization problem as was needed for the MD analysis. If we set $F$ to the unnormalized negentropy regularizer (as was done in the notes above)

$$F(x)=\sum _i{x}_i\log (x_i)-x_i,$$

we have that

$$\begin{array}{rl}{\pi }_k& =\arg \underset{\pi \in {\mathcal{M}}_1(A)}{max}\eta ⟨\pi ,{\hat{q}}_{k-1}⟩-KL(\pi ||{\pi }_{k-1})\end{array}$$

which turns out to be equivilant to

$$\begin{array}{}\text{(13)}& {\pi }_k& =\arg \underset{\pi \in {\mathcal{M}}_1(A)}{max}\eta ⟨\pi ,\sum _{j=0}^{k-1}{\hat{q}}_j⟩-F(\pi ).\end{array}$$

The above equation is the policy selection made by the Follow The Regularized Leader (FTRL) algorithm. For further details of the equavilance between MD and FTRL when $F$ is the unnormalized negentropy one can refer to chapter 28 of the Bandit Book. Importantly, the above equation will be useful for our proof.

We will now show that $T_3\le 0$ by showing that

$$\sum _{j=0}^{k-1}⟨{\pi }_{k-1},{\hat{q}}_j⟩\ge \sum _{j=0}^{k-1}⟨{\pi }_j,{\hat{q}}_j⟩$$

To do this notice that

$$\begin{array}{rl}\eta \sum _{j=0}^{k-1}⟨{\pi }_{k-1},{\hat{q}}_j⟩& =\eta ⟨{\pi }_{k-1},{\hat{q}}_{k-1}⟩+\eta ⟨{\pi }_{k-1},\sum _{j=0}^{k-2}{\hat{q}}_j⟩-F({\pi }_{k-1})+F({\pi }_{k-1})\\ & \ge \eta ⟨{\pi }_{k-1},{\hat{q}}_{k-1}⟩+\eta ⟨{\pi }_{k-2},\sum _{j=0}^{k-2}{\hat{q}}_j⟩-F({\pi }_{k-2})+F({\pi }_{k-1})\\ & \ge \eta \sum _{j=0}^{k-1}⟨{\pi }_j,{\hat{q}}_j⟩-F({\pi }_0)+F({\pi }_{k-1})\\ & \ge \eta \sum _{j=0}^{k-1}⟨{\pi }_j,{\hat{q}}_j⟩\end{array}$$

where the first inequality holds since by $\text{(13)}$ we know that

$${\pi }_{k-1}=\arg \underset{\pi \in {\mathcal{M}}_1(A)}{max}\eta ⟨\pi ,\sum _{j=0}^{k-2}{\hat{q}}_j⟩-F(\pi ).$$

The second inequality holds by repeatadly apply the first two steps.

The third inequality holds since ${\pi }_0$ was initialized as ${\pi }_0=\arg \underset{\pi \in {\mathcal{M}}_1(A)}{min}F(\pi )$ so we have that $F({\pi }_{k-1})-F({\pi }_0)\ge 0$. Which concludes the argument.

Online convex optimization, online learning

Online linear optimization is a special case of online convex/concave optimization, where the learner chooses elements of some nonempty convex set $\mathcal{X}\subset {\mathbb{R}}^d$ and the adversary needs to choose an element of a nonempty set $\mathcal{Y}$ of concave functions over $\mathcal{X}$: $\mathcal{Y}\subset \{f:\mathcal{X}\to \mathbb{R}:f\text{ is concave}\}$. Then, the definition of regret is changed to

$$\begin{array}{}\text{(14)}& R_k=\underset{x\in \mathcal{X}}{max}\sum _{j=0}^{k-1}{y}_j(x)-y_j(x_j),\end{array}$$

where as before $x_j\in \mathcal{X}$ is the choice of the learner for round $j$ and $y_j\in \mathcal{Y}$ is the choice of the adversary for the same round. Identifying any vector $u$ of ${\mathbb{R}}^d$ with the linear map $x\mapsto ⟨x,u⟩$, we see that online linear optimization is a special case of this problem.

Of course, by negating all functions in $\mathcal{Y}$ (i.e., letting $\tilde{\mathcal{Y}}=\{-y:y\in \mathcal{Y}\}$) and redefining the regret to

$$\begin{array}{}\text{(15)}& R_k=\underset{x\in \mathcal{X}}{max}\sum _{j=0}^{k-1}{\tilde{y}}_j(x_j)-{\tilde{y}}_j(x)\end{array}$$

we get a definition that is used in the literature, which prefers the convex case to the concave. Here, the interpretation is that ${\tilde{y}}_j\in \tilde{\mathcal{Y}}$ is a “loss function” chosen by the adversary in round $j$.

The standard function notation ($y_j$ is applied to $x$) injects unwarranted asymmetry in the notation. After all, from the perspective of the learner, they need to choose a value in $\mathcal{X}$ that works for the various functions in $\mathcal{Y}$. Thus, we can consider any element of $\mathcal{X}$ as a function that maps elements of $\mathcal{Y}$ to reals through $y\mapsto y(x)$. Whether $\mathcal{Y}$ has functions in them or $\mathcal{X}$ has functions in them does not matter that much; it is the interconnection between $\mathcal{X}$ and $\mathcal{Y}$ that matters more. For this reason, one can study online learning when $y(x)$ above is replaced by $b(x,y)$, where $b:\mathcal{X}\times \mathcal{Y}\to \mathbb{R}$ is a specific map that assigns payoffs to every pair of points in $\mathcal{X}$ and $\mathcal{Y}$. When the map is fixed, one can spare an extra symbol by just using $[x,y]$ in place of $b(x,y)$, which makes things almost a full circle given that we started with the linear case when $[x,y]=⟨x,y⟩$.

Truncation or no truncation?

We introduced truncation to simplify the analysis. The proof can be made to go through even without it, with a mild increase of the suboptimality gap (or runtime). The advantage of removing the projection is that without projection, ${\hat{q}}_0+\cdots +{\hat{q}}_{j-1}=\mathrm{\Phi }({\hat{\theta }}_0+\cdots +{\hat{\theta }}_{j-1})$, which leads to a practically significant reduction of the runtime.

References

The optimality of the final policy presented in the Notes was shown by Tadashi Kozuno when he taugh this lecture in Winter 2022.

---