Dimensionality Reduction: Role, Importance and Common Methods

Definition. Dimensionality reduction is the transformation of data from a high-dimensional feature space $\mathbb{R}^d$ into a lower-dimensional space $\mathbb{R}^{d'}$ with $d' \ll d$, while preserving as much of the relevant information as possible.

1. Role and importance

1. Combats the curse of dimensionality. As $d$ grows, points become sparse, distances lose contrast and density estimates become unreliable; reducing $d$ restores meaningful geometry.
2. Removes noise and redundancy. Correlated or irrelevant attributes add variance without signal; dropping them often improves accuracy.
3. Reduces overfitting. Fewer parameters mean a simpler model with better generalisation (bias-variance trade-off).
4. Cuts cost. Storage, distance computation and model training all scale with $d$; $O(nd)$ becomes $O(nd')$.
5. Enables visualisation. Projecting to 2-D or 3-D lets analysts see clusters and outliers directly.
6. Improves interpretability. A handful of components or selected features is easier to explain than thousands of raw variables.

2. Common methods

Method	Type	One-line description
PCA	Linear, unsupervised extraction	Projects onto eigenvectors (principal components) of the covariance matrix that capture the most variance
SVD	Linear, unsupervised extraction	Factorises $X = U\Sigma V^T$; the leading singular vectors give a low-rank approximation of the data
Factor analysis	Linear, probabilistic	Explains observed variables through a few uncorrelated latent factors plus noise
ICA	Linear, unsupervised	Finds statistically independent, non-Gaussian source signals (e.g. separating mixed audio)
LDA	Linear, supervised extraction	Maximises class separability by maximising between-class over within-class scatter
Multidimensional scaling (MDS)	Distance preserving	Places points in low dimensions so pairwise distances are kept as intact as possible
t-SNE / UMAP	Non-linear	Preserve local neighbourhood structure for cluster visualisation in 2-D/3-D
Autoencoders	Non-linear, neural	A bottleneck network learns a compressed code by reconstructing its input
Random projection	Linear, random	Projects with a random matrix; the Johnson-Lindenstrauss lemma guarantees distances are roughly preserved
Filter feature selection	Selection	Rank attributes by correlation, mutual information or chi-square and keep the top ones
Wrapper feature selection	Selection	Search attribute subsets by evaluating a learner (forward, backward, recursive feature elimination)
Embedded feature selection	Selection	Selection happens inside training, e.g. L1 (lasso) regularisation
Wavelet transform	Signal/multi-resolution	Decomposes a signal into scaled and shifted wavelets, keeping only the significant coefficients

3. Two families: extraction vs selection

• Feature extraction creates new, compact attributes (PCA, SVD, LDA, autoencoders): maximum compression, but the new features are combinations and lose direct meaning.
• Feature selection keeps a subset of the original attributes (filters, wrappers, embedded): retains interpretability but may discard correlated information.

4. PCA in four steps

1. Standardise each feature to zero mean and unit variance.
2. Compute the covariance matrix $\Sigma = \frac{1}{n-1}X^TX$.
3. Take eigenvectors of $\Sigma$ and sort by eigenvalue; eigenvalue $\lambda_j$ is the variance explained by component $j$.
4. Keep the top $d'$ components and project $X' = XW$, choosing $d'$ so that $\sum_{j=1}^{d'}\lambda_j / \sum_j \lambda_j$ exceeds a chosen threshold such as 95%.

Conclusion: dimensionality reduction is essential preprocessing — it makes mining faster, cleaner and often more accurate, and the right choice depends on whether the goal is compression, visualisation, interpretability or supervised discrimination.