Reinforcement Learning Basics

Policies

A policy is a rule that determines what action to take, typically denoted as $\mu$. When the action is selected stochastically, the policy is represented specifically as $\pi (\cdot |s_t)$ at timestep $t$


When the policy is based on stochastic process, the action is sampled categorically if the action space is discrete, and sampled in a Guassian manner if the action space is continuous.

Value Functions

value functions are functions that return values of a specified state or state-action pair.

1. On-Policy Value Function $V^{\pi }(s)$: starting in state $s$ and always act according to policy $\pi$
   $$V^{\pi }(x)={\mathbb{E}}_{\tau \sim \pi }[R(\tau )|s_o=s]$$

2. On-Policy Action-Value Function: starting in state $s$ and takes an action $a$
   $$Q^{\pi }(s,a)={\mathbb{E}}_{\tau \sim \pi }[R(\tau )|s_0=s,a_0=a]$$

Advantage Functions

Advantage function $A^{\pi }(s,a)$: how much an action is better than others on average


$A^{\pi }(s,a)$ describes how much better it is to take a specific action $a$ in state s over randomly selected action according to $\pi (\cdot |s)$. Hence it can be represented as


$$A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }(s)$$

Discussion

---

References

[1] https://spinningup.openai.com/en/latest/spinningup/rl_intro.html

Footnotes