Question 1

What's the difference between bias and variance, and why can't you minimize both freely?

Accepted Answer

Bias is error from overly simplistic assumptions (model underfits). Variance is error from being too sensitive to training data fluctuations (model overfits). Reducing one often increases the other; the goal is the sweet spot minimizing total generalization error.

Question 2

Explain gradient descent as if teaching someone with no ML background, then show the update rule in code.

Accepted Answer

Gradient descent iteratively updates parameters in the direction that reduces the loss fastest (negative gradient). Too large a learning rate overshoots; too small is painfully slow.

Question 3

What is the curse of dimensionality, and why does it hurt distance-based models like k-NN?

Accepted Answer

As dimensions increase, data becomes sparse and the ratio between nearest and farthest neighbor distances approaches 1, so 'closeness' loses discriminative power — hurting k-NN, clustering, and other distance-based methods. Fixes: dimensionality reduction (PCA, t-SNE), feature selection, or non-distance-based models.

Question 4

Why do we use cross-entropy loss instead of mean squared error for classification?

Accepted Answer

Cross-entropy measures divergence between predicted and true probability distributions, producing steeper gradients for probabilistic outputs and pairing naturally with softmax/sigmoid. MSE on probabilities produces vanishing gradients when predictions are confidently wrong.

Question 5

What is the difference between a generative and a discriminative model?

Accepted Answer

Discriminative models learn P(y|x) directly — the decision boundary (logistic regression, SVM). Generative models learn P(x|y) and P(y), modeling how data is generated (Naive Bayes, GANs, diffusion models), letting them also generate new samples.

Question 6

Explain the intuition behind eigenvectors/eigenvalues in the context of PCA.

Accepted Answer

PCA computes the data's covariance matrix, finds its eigenvectors (directions of max variance) and eigenvalues (variance explained). Projecting onto the top-k eigenvectors gives a lower-dimensional representation preserving maximum variance.

Question 7

What is the difference between Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP) estimation?

Accepted Answer

MLE maximizes P(data|θ) with no prior on θ. MAP maximizes the posterior P(θ|data) ∝ P(data|θ)·P(θ), incorporating a prior. L2 regularization can be derived as MAP with a Gaussian prior.

Question 8

What is the difference between parametric and non-parametric models?

Accepted Answer

Parametric models (linear/logistic regression, fixed-architecture nets) have a fixed parameter count regardless of dataset size. Non-parametric models (k-NN, decision trees, kernel methods) can grow in complexity with more data, making fewer assumptions but often needing more data/compute.

Question 9

Why is the softmax function used for multi-class classification, and what's a numerically stable implementation?

Accepted Answer

Softmax converts logits into probabilities summing to 1, amplifying differences via exponentials, pairing well with cross-entropy. Naive implementation overflows with large logits, so subtract the max logit first — mathematically equivalent but numerically stable.

Question 10

What is the difference between convex and non-convex optimization, and why does it matter for training neural networks?

Accepted Answer

Convex functions have one global minimum, guaranteeing convergence with gradient methods (linear/logistic regression). Neural net loss surfaces are non-convex with many local minima/saddle points; good initialization, momentum, adaptive learning rates, and overparameterization tend to make most local minima 'good enough' in practice.

Question 11

How does a Random Forest reduce overfitting compared to a single Decision Tree?

Accepted Answer

Random Forests combine bagging (bootstrap sampling) with random feature selection per split, training many decorrelated trees. Averaging/voting across trees reduces variance without much increase in bias.

Question 12

Explain how gradient boosting (e.g., XGBoost) differs fundamentally from bagging methods like Random Forest.

Accepted Answer

Bagging trains models in parallel/independently on random subsets, reducing variance. Boosting trains models sequentially, fitting each new tree to residual errors/gradients of the ensemble so far, reducing bias — more prone to overfitting without regularization (learning rate, depth, early stopping).

Question 13

Why do we scale/normalize features before training models like SVM or k-NN, but not for tree-based models?

Accepted Answer

Distance/gradient-based algorithms (k-NN, SVM, k-means, linear/logistic regression, neural nets) are sensitive to feature scale. Tree-based models split on per-feature thresholds, so monotonic scaling doesn't change splits.

Question 14

What's the difference between L1 and L2 regularization, and why does L1 tend to produce sparse models?

Accepted Answer

L2 (Ridge) penalizes squared weights, shrinking all coefficients smoothly toward zero. L1 (Lasso) penalizes absolute weights; its non-differentiable corners at zero often push small coefficients exactly to zero, performing feature selection.

Question 15

How does the kernel trick let SVMs handle non-linearly separable data without explicitly computing high-dimensional features?

Accepted Answer

The kernel trick computes dot products as if points were mapped into higher dimensions, using a kernel function (RBF, polynomial), without ever computing that mapping explicitly — enabling efficient non-linear decision boundaries.

Question 16

What is k-means clustering, and what's a major limitation of the algorithm?

Accepted Answer

K-means iteratively assigns points to the nearest of k centroids, then recomputes centroids as the mean of assigned points, until convergence. Limitations: must choose k in advance, assumes roughly spherical/similarly-sized clusters, sensitive to initialization (mitigated by k-means++) and to outliers/feature scale.

Question 17

What's the difference between bagging and stacking as ensemble techniques?

Accepted Answer

Bagging trains many instances of the *same* algorithm on bootstrapped samples, combining via averaging/voting. Stacking trains several *different* base models, then a meta-model on their outputs to learn the optimal combination, capturing complementary strengths.

Question 18

Explain the exploration-exploitation tradeoff via Bayesian optimization for hyperparameter tuning.

Accepted Answer

Bayesian optimization builds a probabilistic surrogate model (often a Gaussian Process) of the objective vs. hyperparameters, and uses an acquisition function (e.g., Expected Improvement) balancing exploitation of known-good regions against exploration of uncertain ones — far more sample-efficient than grid/random search.

Question 19

What is feature leakage, and give a concrete example of how it silently inflates reported performance.

Accepted Answer

Leakage occurs when information not legitimately available at prediction time sneaks into features, inflating validation performance. Example: a feature timestamped *after* the target event occurred, or normalizing using statistics computed over the full dataset including test data instead of fitting only on train.

Question 20

What is the difference between a validation set and a test set, and why do you need both?

Accepted Answer

The validation set is used repeatedly during development to tune hyperparameters/model choices. The test set is held out and used only once at the end to estimate real-world generalization; using it for tuning would leak information and give an overly optimistic estimate.

Question 21

What is the vanishing gradient problem, and how do architectures like ResNet or LSTMs address it?

Accepted Answer

Gradients computed via the chain rule can shrink exponentially with depth if derivatives are consistently <1 (sigmoid/tanh), so early layers barely learn. ResNet's skip connections let gradients flow directly through shortcuts. LSTMs use gating with an additive cell-state update preserving gradient flow across time steps.

Question 22

Explain self-attention in Transformers in plain terms, then show the scaled dot-product formula.

Accepted Answer

Self-attention computes, for each token, a weighted combination of all tokens' 'value' vectors, with weights from the similarity (dot product) between query and key vectors, scaled and softmaxed — letting the model dynamically focus on relevant parts regardless of distance.

Question 23

What is batch normalization, and why does it help training converge faster?

Accepted Answer

BatchNorm normalizes layer activations (zero mean, unit variance) per mini-batch, then applies a learnable scale/shift, reducing internal covariate shift, allowing higher learning rates, less sensitivity to initialization, and a mild regularizing effect.

Question 24

Dropout seems like it should hurt performance — why does it actually help generalization?

Accepted Answer

Dropout randomly zeroes neurons during training, preventing co-adaptation and acting like training an ensemble of thinned sub-networks sharing weights. At inference, all neurons are used (appropriately scaled), approximating an ensemble average — improving generalization.

Question 25

Why do we need positional encodings in Transformers, when RNNs don't need them?

Accepted Answer

Self-attention is permutation-invariant with no inherent sense of order. Positional encodings (sinusoidal or learned) are added to embeddings to inject order information while preserving parallel computation.

Question 26

Why is ReLU generally preferred over sigmoid/tanh as an activation function in deep networks?

Accepted Answer

Sigmoid/tanh saturate for large inputs, driving gradients toward zero, worsened in deep nets. ReLU has a constant gradient of 1 for positive inputs and is cheap to compute; its downside ('dying ReLU') is addressed by Leaky ReLU or GELU variants.

Question 27

What's the difference between a CNN's convolutional layer and a fully connected layer, and why are CNNs so effective for images?

Accepted Answer

Convolutional layers apply a small, shared filter set across spatial regions, exploiting local connectivity, parameter sharing, and translation invariance — drastically fewer parameters than fully connected layers and matches images' spatial structure.

Question 28

What is the difference between an autoencoder and a variational autoencoder (VAE), and why does the VAE's latent space enable generation?

Accepted Answer

A standard autoencoder learns a deterministic latent code optimized purely for reconstruction with no guarantee of a continuous latent space. A VAE learns a distribution (mean/variance) per input and adds a KL-divergence term pulling it toward a standard normal prior, producing a smooth, sampleable latent space for generation.

Question 29

What is transfer learning, and why does fine-tuning a pretrained model usually beat training from scratch on a small dataset?

Accepted Answer

Transfer learning reuses a model pretrained on a large general dataset, then fine-tunes on a smaller task-specific dataset. Early layers already encode broadly useful features, so far less task-specific data is needed to adapt later, more specialized layers.

Question 30

What is the difference between layer normalization and batch normalization, and why do Transformers use LayerNorm?

Accepted Answer

BatchNorm normalizes across the batch dimension per feature, requiring consistent batch statistics — awkward for variable-length sequences/small batches. LayerNorm normalizes across the feature dimension per sample independently, robust to variable sequence length and batch size — why Transformers use it.

AI & Machine Learning Interview Questions

Foundations & Math Intuition

Classical Machine Learning

Deep Learning

Large Language Models & GenAI

MLOps & Production Systems

Computer Vision

Natural Language Processing

Reinforcement Learning

Model Evaluation & Statistics

ML System Design