Regression Analysis: Statistical process for estimating relationships among variables. Predict continuous dependent $y$ from independent $X$. Model $y = f(X;\theta)+\epsilon$. Used for prediction, inference of variable importance, trend estimation.

Types: Simple linear, multiple linear, polynomial, ridge/lasso, logistic (classification), non-linear.

Linear Regression:

• Model: $y = X\beta + \epsilon$, $\epsilon\sim N(0,\sigma^2)$
• Assumes linear relationship, homoscedasticity.
• OLS solution: $\hat\beta = (X^T X)^{-1}X^T y$
• Loss: $RSS = \sum(y_i- \hat y_i)^2$
• Suffers when $X^T X$ singular/multicollinear, overfits with many features, sensitivity to outliers.

Ridge Regression (L2 Regularized):

• Loss: $J(\beta)= RSS + \lambda||\beta||_2^2 = \sum(y_i- x_i^T\beta)^2 + \lambda\sum\beta_j^2$, $\lambda\ge0$
• Solution: $\hat\beta_{ridge}=(X^T X + \lambda I)^{-1}X^T y$
• Adds bias but reduces variance (bias-variance trade-off), handles multicollinearity by shrinking coefficients, $X^T X + \lambda I$ always invertible.
• Does feature shrinkage, not selection (all $\beta_j$ non-zero). Lasso ($L1$) does selection.

Aspect	Linear	Ridge
Penalty	none	$\lambda||\beta||^2$
Coefficients	unbiased, high variance if collinear	biased (shrink toward 0), lower variance
When $p>n$	fails	works
Hyperparameter	none	$\lambda$ via CV
Interpretability	direct	shrunk values

Ridge is preferred when multicollinearity, high dimensional or overfitting risk.