Reinforcement Learning (Concept) — Beginner-Friendly Notes

1. What RL Means (third paradigm)

Agent learns by doing in an environment, getting rewards/penalties (not correct answers), and learning a policy ($state \rightarrow action$) that maximises cumulative future reward.

Supervised: teacher gives $y$. Unsupervised: no teacher. RL: critic gives $reward$, agent figures out $y$ itself.

2. Core Loop (agent-environment again, but with reward)

State → Agent picks Action → Environment transitions → Reward + New State → Agent updates policy

• Reward $r$: scalar feedback (e.g., +1 win, -1 fall, +10 reach goal).
• Policy $\pi$: mapping $state \rightarrow action$ (what to do).
• Value $V(s)$ / $Q(s,a)$: expected future cumulative (discounted) reward from $s$ (or $s,a$) — predicts "how good is this situation/move in long run".

Goal: find $\pi^*$ that maximises expected return $G = r_0 + \gamma r_1 + \gamma^2 r_2 + ...$ where $\gamma \in [0,1)$ is discount (future less than now).

3. Key Trade-off — Exploration vs Exploitation

• Exploitation: use best known action.
• Exploration: try new action to discover better reward.
• Unlike supervised (learn from fixed dataset), RL must generate its own data by exploring — needs problem generator (learning agent, Module 1).

4. Classic Example — Game Playing (AlphaGo)

• State = board, action = move, reward = win/loss at end. Agent plays millions of games, initially random, gradually learns policy/value that wins, without ever seeing "correct move" labels.

Other domains: robotics (walk without falling), self-driving (reward = safety + speed), recommendation (reward = clicks).

5. Two Families of RL Algorithms (concept only)

• Value-based: learn $Q(s,a)$, then act greedily on $Q$ (e.g., Q-learning).
• Policy-based: directly learn $\pi$ (e.g., policy gradient).
  Exams only need concept, not derivations.

6. Challenges for Exam

• Sparse rewards: only at end (need credit assignment — which move caused win?).
• Sample hungry: needs many interactions.
• Stability: exploration can be unsafe in real world → simulate first.

7. Paradigm Contrast (memorise table)

	Signal	Data	Goal
Supervised	label $y$	$(x,y)$	minimise prediction error
Unsupervised	none	$(x)$	find structure
Reinforcement	reward $r$	experience $(s,a,r,s')$	maximise cumulative reward

8. One-Line Summary

Reinforcement learning learns an optimal policy by trial-and-error interaction, maximising discounted cumulative reward with exploration-exploitation, formalised via $Q(s,a)$ and return $G$, and is the paradigm behind game AI and robotics.

9. Quick Self-Check

1. Why does RL need to explore?
2. Name one RL application outside games.