ML Fundamentals for Data Scientists / ML Engineers (2026)

Bias-variance: the canonical ML interview question

Every FAANG-tier ML interview probes bias-variance trade-off. The textbook framing: total error = bias-squared + variance + irreducible-error. High-bias models underfit (decision tree with depth 1); high-variance models overfit (decision tree with depth 50). The senior bar is to articulate the trade-off in terms of the specific architecture you are using.

For tree-based gradient boosting (XGBoost, LightGBM, CatBoost):

• Bias controls. Increase max_depth (more complex trees), increase n_estimators (more boosting rounds), reduce min_child_weight / min_data_in_leaf (allow more leaf nodes).
• Variance controls. Reduce max_depth, reduce n_estimators, increase min_child_weight, add column-subsampling (colsample_bytree), increase L2 regularization (lambda).
• Diagnostic. Train and validation loss diverging means high variance, regularize. Train and validation loss both high means high bias, increase capacity.

For neural networks (PyTorch / JAX), three orthogonal regularization techniques:

import torch.nn as nn
# High-variance fix: dropout regularization
self.fc1 = nn.Linear(input_dim, 256)
self.dropout1 = nn.Dropout(0.3)  # zero out 30% of activations
self.fc2 = nn.Linear(256, output_dim)

# In training loop: weight-decay (L2 regularization on parameters)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
# Early stopping on validation loss is a third regularizer.

The senior interview answer: name three orthogonal regularization techniques for the architecture, articulate when each binds (dropout binds when activations are too rich; weight-decay binds when parameters are too large; early stopping binds when training-set fit exceeds generalization).

Gradient boosting: the production-ML workhorse

Gradient boosting is the dominant approach for tabular ML in 2026. XGBoost (Chen & Guestrin, 2016, arxiv.org/abs/1603.02754) is the most-deployed; LightGBM (Microsoft) and CatBoost (Yandex) are alternatives with different optimization choices.

A canonical production pattern at fintech, ad-tech, and recommendation companies:

import xgboost as xgb
from sklearn.model_selection import TimeSeriesSplit

# 1. Time-series cross-validation (NEVER use simple k-fold for temporal data)
tscv = TimeSeriesSplit(n_splits=5, gap=7)  # 7-day gap to avoid leakage
for train_idx, val_idx in tscv.split(X):
    X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
    y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]

    # 2. Train with early stopping on the validation fold
    dtrain = xgb.DMatrix(X_train, label=y_train)
    dval = xgb.DMatrix(X_val, label=y_val)
    params = {
        "objective": "binary:logistic",
        "eval_metric": "logloss",
        "max_depth": 6,                # bias-variance: 6 is a strong default
        "min_child_weight": 5,
        "subsample": 0.8,
        "colsample_bytree": 0.8,
        "learning_rate": 0.05,
        "reg_lambda": 1.0,
    }
    model = xgb.train(
        params, dtrain, num_boost_round=2000,
        evals=[(dval, "val")],
        early_stopping_rounds=50,
    )

# 3. Production deployment: persist with native XGBoost JSON format
model.save_model("model.json")  # JSON for cross-version stability

The senior MLE conversation around this code: why the 7-day gap? (Gap prevents leakage from features computed from future data.) Why time-series split rather than k-fold? (k-fold over temporal data trains on future and validates on past, producing optimistic estimates.) Why save_model JSON rather than a binary serializer? (Binary formats are fragile across XGBoost versions; JSON is the documented stable format.)

Real-world hyperparameter ranges that work well across many tabular problems: max_depth 4–8, learning_rate 0.01–0.1, num_boost_round 500–3000 with early stopping, subsample / colsample 0.7–0.9. Senior MLE candidates can articulate why these defaults work and when they need to change.

Neural-net optimization: SGD, Adam, AdamW

The optimization landscape in 2026: SGD with momentum (Nesterov 1983) is still used for ConvNets at scale; Adam (Kingma & Ba 2015, arxiv.org/abs/1412.6980) is the default for transformers; AdamW (Loshchilov & Hutter 2019, arxiv.org/abs/1711.05101) is the default at AI-labs for foundation-model training.

Why AdamW over Adam: Adam couples weight decay with the per-parameter adaptive learning rate, which causes weight-decay to depend on the parameter magnitude in a way that breaks regularization. AdamW decouples them, making weight-decay behave the way an L2 regularizer should. Practical implication: AdamW gives better generalization than Adam at the same hyperparameters on transformer-scale training.

import torch.optim as optim

# Standard Adam — coupled weight decay, weaker regularization
optimizer = optim.Adam(model.parameters(), lr=1e-3, weight_decay=0.01)

# AdamW — decoupled weight decay, the default at AI-labs in 2026
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)

# With cosine learning rate schedule and warmup (the foundation-model standard)
from torch.optim.lr_scheduler import CosineAnnealingLR, LambdaLR

warmup_steps = 1000
total_steps = 100000
warmup = LambdaLR(optimizer, lr_lambda=lambda s: min(s / warmup_steps, 1.0))
cosine = CosineAnnealingLR(optimizer, T_max=total_steps - warmup_steps)
# In practice, transformers use SequentialLR to chain warmup -> cosine

The senior MLE conversation: why warmup? (At training start, gradient estimates are noisy; large learning rates cause divergence. Warmup ramps the LR up over the first 1–5% of training.) Why cosine schedule rather than constant LR? (Empirically produces better final loss across many transformer training runs; the original Llama and OPT papers used cosine schedules.) Why decoupled weight decay? (See Loshchilov & Hutter for the formal argument; the empirical answer is that it makes regularization more controllable across architectures.)

Cross-validation: time-series, group, and stratified

Cross-validation methodology is one of the most common interview blind-spots for junior MLE candidates. The wrong methodology produces overly-optimistic estimates that do not hold in production.

• Standard k-fold. Use only when data points are truly independent. Most production data has at least one of: temporal correlation, group structure (multiple observations per user), or distribution shift. Naive k-fold across these structures produces leakage.
• TimeSeriesSplit (forward-chaining). Use when data has temporal structure. Each fold trains on past, validates on future. Optionally include a gap between train and validation to avoid features-computed-from-future-data leakage. Real production: at Stripe, Uber, and most fintech, TimeSeriesSplit is the default.
• GroupKFold. Use when data has group structure (multiple rows per user, per session, per device). Splits ensure that all rows for a given group are in the same fold — preventing the model from memorizing individual users.
• StratifiedKFold. Use for classification with class imbalance. Each fold has roughly the same class distribution as the overall data.
• Combinations. Real production often combines: a time-series split with a group constraint (e.g., users registered in the same month must be in the same fold). The scikit-learn `GroupTimeSeriesSplit` does not exist out of the box; teams typically implement custom splitters.

Canonical reference: the scikit-learn cross-validation guide (scikit-learn.org/stable/modules/cross_validation.html) is the canonical introduction. The senior bar: identify the leakage risks in a given dataset and pick a CV strategy that prevents them.