Introduction to Reinforcement Learning

Introduction to Reinforcement Learning

1. Loop

Agent in state $s$ takes action $a$, gets reward $r$, moves to $s'$. Goal: learn policy $π(s) → a$ maximizing cumulative discounted reward:

$γ$ near 1 = far-sighted; near 0 = myopic.

2. Value Functions

• $V(s)$: expected $G$ from $s$ following $π$.
• $Q(s,a)$: expected $G$ from $s$ taking $a$ then $π$.

Optimal policy acts greedily on $Q*$.

3. Core Algorithms

Method	Needs model?	Idea
Dynamic programming	Yes (transitions known)	Bellman backup
Monte Carlo	No (episodes)	average returns
TD / Q-learning	No (steps)

|
| Policy gradient | No | ascend expected $G$ directly (REINFORCE, actor-critic) |

4. Exploration

$ε$-greedy: random action with prob $ε$ (explore), else best (exploit). Decay $ε$ over time.

5. Exam Points

• Define state, action, reward, $G$, $π$ — 5 marks, very frequent.
• Write Q-learning update — 5 marks, frequent.
• Explore vs exploit ($ε$-greedy) — 3 marks, frequent.