MACHINE LEARNING

Answer: (iii) L2 norm

Explanation:

Ridge Regression (Tikhonov regularization) adds L2 penalty to OLS loss:

$J(w) = \sum_{i=1}^n (y_i - w^T x_i)^2 + \lambda ||w||_2^2$ where $||w||_2^2 = \sum_j w_j^2$, $\lambda >0$

• Shrinks coefficients toward zero, handles multicollinearity, reduces variance / overfitting.
• Closed form: $w_{ridge} = (X^T X + \lambda I)^{-1} X^T y$

Contrast:

• L1 norm (i) → Lasso regression, induces sparsity/feature selection.
• Entropy (ii) / Cross-entropy (iv) → used in logistic regression / decision trees, not ridge.

So ridge = L2.

Answer: (ii) Parameters that maximize data likelihood

Explanation:

Maximum Likelihood Estimation (MLE):

$\hat{\theta}_{MLE} = \arg\max_{\theta} L(\theta) = \arg\max_{\theta} p(D|\theta) = \arg\max_{\theta} \prod_i p(x_i|\theta)$

• Finds parameter values that make observed data most probable under assumed model.
• Example: For Gaussian, MLE gives sample mean $\hat{\mu}= \bar{x}$ and sample variance (biased) $\hat{\sigma}^2 = \frac1n\sum(x_i-\bar{x})^2$.
• Contrast MAP: $\arg\max p(\theta|D) \propto p(D|\theta)p(\theta)$ includes prior.

It does NOT find priors (i), decision boundaries (iii) directly, or cluster centroids (iv) — those are consequences after estimation.

So (ii) is correct.

Given: $n=5$, $x=[1,2,3,4,5]$, $y=[3,4,2,5,7]$

Compute $w_0$ (intercept), $w_1$ (slope) for $y = w_0 + w_1 x$ via OLS.

Formulas: $w_1 = \frac{\sum (x_i-\bar x)(y_i-\bar y)}{\sum (x_i-\bar x)^2}$, $w_0 = \bar y - w_1 \bar x$

• $\bar x = (1+2+3+4+5)/5 = 3$
• $\bar y = (3+4+2+5+7)/5 =21/5=4.2$

$x_i$	$y_i$	$x_i-\bar x$	$y_i-\bar y$	$(x_i-\bar x)(y_i-\bar y)$	$(x_i-\bar x)^2$
1	3	-2	-1.2	2.4	4
2	4	-1	-0.2	0.2	1
3	2	0	-2.2	0	0
4	5	1	0.8	0.8	1
5	7	2	2.8	5.6	4
Sum numer $=9.0$, denom $=10$

$w_1 = 9/10 =0.9$

$w_0 =4.2 -0.9*3 =4.2-2.7=1.5$

Model: $\hat y =1.5 +0.9 x$

Prediction at $x=6$: $\hat y =1.5 +0.9*6 =1.5+5.4=6.9 \approx 7$

Residuals check: $\hat y_{train}=[2.4,3.3,4.2,5.1,6.0]$, SSE small.

If using normal equations directly: $w=(X^T X)^{-1}X^T y$ gives same.

Data: $X =\begin{bmatrix}2&0\\0&1\\0&2\end{bmatrix}$, $n=3$, $d=2$

Step 1: Mean center. $\bar x = [(2+0+0)/3, (0+1+2)/3]=[0.667,1]$

Centered $X_c =\begin{bmatrix}1.333&-1\\-0.667&0\\-0.667&1\end{bmatrix}$

Step 2: Covariance $C = \frac1{n-1} X_c^T X_c$ (unbiased; if $1/n$ divide by 3, mention both).

$X_c^T X_c =\begin{bmatrix}1.333^2+0.667^2+0.667^2 &1.333*(-1)+(-0.667)*0+(-0.667)*1\\ sym &1+0+1\end{bmatrix}$
$= \begin{bmatrix}2.667 & -2\\ -2 &2\end{bmatrix}$

Hence:

• Unbiased ($1/(n-1)=1/2$): $C =\begin{bmatrix}1.333 & -1\\ -1 &1\end{bmatrix}$
• Biased ($1/n$): $C_{biased}=\begin{bmatrix}0.889 & -0.667\\ -0.667 &0.667\end{bmatrix}$ (scale by $2/3$)

Step 3: Eigenvalues solve $|C-\lambda I|=0$:

$(1.333-\lambda)(1-\lambda)-1=0 \Rightarrow \lambda^2 -2.333\lambda+0.333=0$
$\lambda = \frac{2.333\pm\sqrt{5.444-1.333}}{2}= \frac{2.333\pm2.028}{2}$
$\lambda_1 \approx 2.18$, $\lambda_2 \approx 0.152$ (Unbiased)
For biased version: $\lambda_1\approx1.454$, $\lambda_2\approx0.102$ (2/3 times).

Explained variance: $93.5\%$ by PC1, $6.5\%$ by PC2.

Step 4: Eigenvectors (PCs):
For $\lambda_1=2.18$: $(C-\lambda I)v=0\Rightarrow -0.847 v_1 -1 v_2=0\Rightarrow v\propto[1,-0.847]$ normalized $v_1\approx[0.763,-0.647]$
For $\lambda_2=0.152$: $v_2\approx[0.647,0.763]$ orthogonal.

Normalized PCs (unit): $PC1\approx [-0.76,0.65]$ direction of max variance (roughly along $x-y$ diagonal), $PC2$ orthogonal.

Projection $Z=X_c V$ gives reduced 1D representation capturing $>93\%$ variance. First PC is the principal component.