Main Components of Reinforcement Learning (RL):

Agent learns policy interacting with environment via trial-reward.

1. Agent: Learner/decision maker (robot, game player).
2. Environment: External system agent interacts with (maze, market).
3. State $s \in S$: Snapshot of environment (position, sensor readings).
4. Action $a \in A(s)$: Moves agent can take (up/down, buy/sell).
5. Reward $r$: Scalar feedback $r(s,a,s')$ after action (positive for goal, negative for penalty).
6. Policy $\pi(a|s)$: Agent's strategy mapping states → actions (deterministic or stochastic), goal to learn optimal $\pi^*$.
7. Value Function: $V^\pi(s)=E[\sum \gamma^t r_t | s_0=s]$, $Q^\pi(s,a)=E[\sum\gamma^t r_t | s,a]$ — expected cumulative discounted return. Optimal $V^*, Q^*$ satisfy Bellman equations.
8. Model (optional): Transition dynamics $p(s'|s,a)$ and reward model; model-based vs model-free.
9. Discount $\gamma\in[0,1]$: Balances immediate vs future reward.

Loop: $s_0 \xrightarrow{\pi} a_0 \xrightarrow{env} r_0,s_1 \xrightarrow{\pi} a_1 ...$ Goal: maximize $G=\sum \gamma^t r_t$.

Exploration-Exploitation Trade-off:

• Exploitation: Use best-known action (max $Q(s,a)$) to maximize immediate reward.
• Exploration: Try random/new actions to discover potentially better long-term rewards.

Too much exploitation → stuck in suboptimal policy (local optimum). Too much exploration → never converges, low reward.

Example: Multi-Armed Bandit / Maze.

• Robot in maze knows one path gives reward 10 (exploitation), but unexplored path may give 100. $\epsilon$-greedy: with prob $\epsilon$ explore random action, else exploit best. Anneal $\epsilon$ over time. UCB and Thompson Sampling also balance via uncertainty.

In AlphaGo, exploitation = play best known move from $Q$, exploration = try novel moves during self-play to improve.

Solution: decay $\epsilon$ from 1 → 0.1, optimistic initialization, intrinsic motivation.