Support Vector Machines + Kernels

Support Vector Machines + Kernels

1. Max-Margin Idea

Among separating hyperplanes, pick the one with largest gap to nearest points. Gap $= 2 / ||w||$. Support vectors are the points on the margin — they alone define $w$.

$C$ trades margin width vs mistakes. Hinge: $max(0, 1 - y s)$.

2. Why Only Dots Matter

Optimal $w$ is a mix of training points, so predictions use dot products only:

$α_{i} > 0$ only for support vectors (sparse).

3. Kernel Trick

Replace dot $(a^{T}b)$ with kernel $K(a,b)$ = dot in richer space, without building that space:

Kernel	$K(a,b)$	Use
Linear	$a^{T}b$	text, high-$d$
Polynomial	$(a^{T}b + 1)^{p}$	curves
RBF	$exp(-γ ||a-b||²)$	smooth blobs (default)

RBF with small $γ$ → wide influence (smooth); large $γ$ → spiky (may overfit).

4. Practice

Standardize features, tune $C$ and $γ$ by CV. Linear first for $d \gg n$; RBF for nonlinear low-$d$.

5. Exam Points

• Margin objective + support vectors — 7 marks, very frequent.
• Kernel trick + RBF form — 5 marks, very frequent.
• $C$ and $γ$ roles — 3 marks, frequent.