MACHINE LEARNING

Answer: (iv) A good decision applies quantitative approaches — though in ML/decision theory all (i)-(iii) also contribute, expected exam answer is (iv).

Explanation:

In Statistical Decision Theory / ML:

• Good decisions are not just based on logic (i) alone — logic without data is insufficient.
• Considering all data (ii) and all alternatives (iii) are part of rational decision making, but alone not distinctive.
• Quantitative approaches (iv) — using probability, expected loss/risk $R(a) = \int L(a,y) p(y|x) dy$, utility, optimization — differentiate good from bad decisions objectively. This aligns with Bayesian decision theory, where you choose action $a^* = \arg\min_a E[L]$.

Textbook often phrases: "A good decision is based on logic, considers all available data and alternatives, and applies quantitative approaches." If single best is asked, (iv) captures the statistical/ML perspective (quantitative risk minimization) that the other options miss. If multiple correct, (ii)+(iii)+(iv) would be joint.

Answer: (iv) Answering probabilistic query — (i) Solving queries is also acceptable but (iv) is more precise.

Explanation:

Bayes Rule: $P(H|E) = \frac{P(E|H)P(H)}{P(E)}$

Uses:

• Answering probabilistic queries — compute posterior $P(query | evidence)$, e.g., $P(disease|symptoms)$.
• Solving queries in Bayesian networks: Inference via enumeration, variable elimination, MCMC.
• Classification (Naive Bayes): $P(y|x) \propto P(x|y)P(y)$
• Updating beliefs with evidence, spam filtering, medical diagnosis.

It does NOT increase/decrease complexity inherently; it enables probabilistic reasoning.

Hence (iv) Answering probabilistic query is textbook best. (i) Solving queries is vague but related — if single choice forced, pick (iv).

Answer: (i) The polynomial degree

Explanation — Underfitting vs Overfitting trade-off:

• Model complexity controls bias-variance trade-off. Polynomial degree directly controls capacity:

  • Degree=1 (linear) — high bias, may underfit non-linear data
  • Degree too high (e.g., 10 for 20 points) — low bias but high variance, overfits noise, oscillates

• (ii) Whether we learn weights by matrix inversion vs gradient descent — Both solve same optimization $w = (X^T X)^{-1}X^T y$ vs iterative; they reach same $w$ (if convex), so no impact on generalization.

• (iii) Use of constant-term (bias $w_0$) — Important to fit intercept, but small impact compared to degree. Without bias, model forced through origin — may underfit slightly, but not as dramatic as degree change.

Thus degree (i) is the dominant hyperparameter governing over/under-fitting.

Other factors affecting trade-off: regularization $\lambda$, training set size, early stopping.

Linear Regression:

Model: $y = w_0 + w_1 x_1 + ... + w_d x_d + \epsilon = w^T x + b + \epsilon$, with $\epsilon\sim N(0,\sigma^2)$. Goal: learn $w$ minimizing squared loss $J(w)=\sum (y_i - w^T x_i)^2$ or MLE under Gaussian noise.

Closed form: $w^* = (X^T X)^{-1}X^T y$ (OLS). Gradient descent also: $w:=w-\eta
abla J$.

Example: Predict house price $y$ (in lakhs) from size $x$ (sq ft). Data: (1000,50), (1500,70), (2000,90) → learn $y\approx 0.04x +10$ → for 1200 sq ft predict $58$L.

Types: Simple (one predictor), Multiple (many), Polynomial (features $x^2, x^3$ but linear in parameters).

Assumptions to be met before Linear Regression:

1. Linearity: Relationship between $X$ and $Y$ is linear (check scatter plot, residuals vs fitted).
2. Independence of errors: Observations independent (no autocorrelation, Durbin-Watson).
3. Homoscedasticity: Constant variance of residuals $Var(\epsilon|X)=\sigma^2$ (Breusch-Pagan test).
4. Normality of residuals: $\epsilon \sim Normal$ for inference (Q-Q plot, Shapiro-Wilk) — not strictly for prediction.
5. No multicollinearity: Predictors not highly correlated (VIF <5-10).
6. No autocorrelation and no endogeneity ($E[\epsilon|X]=0$).
7. No influential outliers (Cook's distance).
8. Features not random, measured without error, sample size $n \gg d$.

Checks/Remedies: Transform variables (log), remove outliers, ridge if multicollinearity, Weighted LS if heteroscedasticity. Violations bias estimates / invalidate p-values.

Regression Analysis: Statistical process for estimating relationships among variables. Predict continuous dependent $y$ from independent $X$. Model $y = f(X;\theta)+\epsilon$. Used for prediction, inference of variable importance, trend estimation.

Types: Simple linear, multiple linear, polynomial, ridge/lasso, logistic (classification), non-linear.

Linear Regression:

• Model: $y = X\beta + \epsilon$, $\epsilon\sim N(0,\sigma^2)$
• Assumes linear relationship, homoscedasticity.
• OLS solution: $\hat\beta = (X^T X)^{-1}X^T y$
• Loss: $RSS = \sum(y_i- \hat y_i)^2$
• Suffers when $X^T X$ singular/multicollinear, overfits with many features, sensitivity to outliers.

Ridge Regression (L2 Regularized):

• Loss: $J(\beta)= RSS + \lambda||\beta||_2^2 = \sum(y_i- x_i^T\beta)^2 + \lambda\sum\beta_j^2$, $\lambda\ge0$
• Solution: $\hat\beta_{ridge}=(X^T X + \lambda I)^{-1}X^T y$
• Adds bias but reduces variance (bias-variance trade-off), handles multicollinearity by shrinking coefficients, $X^T X + \lambda I$ always invertible.
• Does feature shrinkage, not selection (all $\beta_j$ non-zero). Lasso ($L1$) does selection.

Aspect	Linear	Ridge
Penalty	none	$\lambda||\beta||^2$
Coefficients	unbiased, high variance if collinear	biased (shrink toward 0), lower variance
When $p>n$	fails	works
Hyperparameter	none	$\lambda$ via CV
Interpretability	direct	shrunk values

Ridge is preferred when multicollinearity, high dimensional or overfitting risk.

Given: Test correct positive $P(+|D)=0.98$ (sensitivity), correct negative $P(-|
eg D)=0.97$(specificity) → false positive$P(+|
eg D)=0.03$, disease prevalence$P(D)=0.008$.

We compute posterior using Bayes: $P(D|+) = \frac{P(+|D)P(D)}{P(+)}$ where $P(+)=P(+|D)P(D)+P(+|
eg D)P(
eg D)$.

• $P(D)=0.008$, $P(
  eg D)=0.992$
• $P(+|D)=0.98$, $P(+|
  eg D)=1-0.97=0.03$

$P(+) =0.98*0.008 +0.03*0.992 =0.00784+0.02976=0.0376$

(i) $P(D|+) =0.00784/0.0376 \approx 0.2085 \approx 20.9\%$

**(ii) $P(
eg D|+) =1 -0.2085=0.7915 \approx 79.2%$** (or$0.02976/0.0376=0.7915$)

(iii) Diagnosis: Despite positive test, patient most probably does NOT have cancer/disease (about 79% chance healthy) because disease is rare (base rate fallacy). Positive predictive value only ~21%. Recommend further testing (more specific test, follow-up) — do not immediately diagnose cancer. If used as MAP classifier, predict $
eg D$ (no cancer).

Note: This classic low-prevalence paradox shows importance of prior. If prevalence higher, posterior rises sharply.

(a) Partial Least Squares (PLS):

PLS is dimensionality reduction + regression method for high-dimensional, multicollinear, $p\gg n$ data (e.g., chemometrics, genomics).

Goal: Find latent components that maximize covariance between $X$ (predictors) and $Y$ (responses), not just variance of $X$ like PCA.

Idea: Decompose $X = TP^T + E$, $Y = UQ^T + F$ with $T = XW$, $U = YC$ latent scores, weights $W$ maximize $Cov(T,U) = Corr(T,U) * \sqrt{Var(T)Var(U)}$.

Algorithm (NIPALS):

1. Center/scale $X, Y$.
2. For component $h=1..A$: compute weight $w_h \propto X^T y$ (for univariate $Y$) normalized, $t_h = X w_h$, $p_h = X^T t_h / (t_h^T t_h)$, $q_h = Y^T t_h / (t_h^T t_h)$, deflate $X:= X - t_h p_h^T$, $Y:= Y - t_h q_h^T$.
3. Regression $Y = T B Q^T$ yields $\hat\beta_{PLS}= W(P^T W)^{-1} B Q^T$.

Types: PLS1 ($Y$ univariate), PLS2 ($Y$ multivariate), PLS-DA (classification).

Advantages: Handles multicollinearity, $n<p$, noisy predictors, stable; gives interpretable latent factors (like PCA but supervised).

Vs PCA/OLS: PCA maximizes $Var(T)$, PLS maximizes $Cov(T,U)$ → better predictive components. OLS fails when $X^T X$ singular; PLS works.

Choosing components $A$: Cross-validation (Q² metric).

Example: Predict chemical concentration $Y$ from 1000-wavelength spectra $X$ (highly correlated) with 50 samples → PLS finds 3 latent factors capturing predictive spectra shape.