Support Vector Machines (SVM) — Working & Applications

Working

An SVM is a supervised classification technique that finds the optimal separating hyperplane which best separates the data into two classes and maximises the margin — the distance between the hyperplane and the nearest training points, called support vectors.

        Hyperplane: w·x + b = 0
   class −1        │        class +1
    o   o   o      │        +  +  +  +
      o o     o    │        +      +
        o         (margin)   +      +
   ────────────────────────────
         ↑  margin  ↑
      support vectors

For linearly separable data, the decision boundary is:

$$w \cdot x + b = 0, \quad f(x) = \operatorname{sign}(w \cdot x + b)$$

Maximising the margin $2/\|w\|$ is solved as a constrained convex optimisation problem (using the Lagrangian dual). The separating hyperplane depends only on the support vectors, so the model is sparse.

Key ideas:

1. Maximal margin — choosing the hyperplane with the largest margin gives the best generalisation.
2. Soft margin — real data is often not perfectly separable; slack variables $\xi_i \ge 0$ allow a few misclassified (or margin-violating) points, penalised by a cost parameter $C$:

$$\min \frac{1}{2}\|w\|^2 + C\sum_i \xi_i$$

3. Kernel trick — for non-linear data, a kernel function $K(x_i, x_j)$ (linear, polynomial, RBF/Gaussian, sigmoid) implicitly maps the data into a higher-dimensional feature space where a linear hyperplane can separate the classes:

$$f(x) = \operatorname{sign}\left(\sum_{i} \alpha_i y_i K(x_i, x) + b\right)$$

Applications in Data Mining

• Text / document categorisation — spam detection, topic classification (works extremely well on high-dimensional text features).
• Image recognition — handwritten-digit recognition, face detection.
• Bioinformatics — classification of gene-expression data, protein function prediction.
• Financial data mining — credit scoring, fraud detection, bankruptcy prediction.
• Web mining — web-page classification, personalized recommendations.
• Anomaly/outlier detection — one-class SVM for network intrusion detection and fault monitoring.

Advantages

• Strong theoretical guarantees (maximal-margin → good generalisation), effective in high dimensions, robust to overfitting because only support vectors matter.

Limitations

• Sensitive to kernel choice and parameters; training is slow on huge datasets; the basic method is binary (multi-class handled by one-vs-all / one-vs-one decomposition).