Causal Inference for Data Scientists / ML Engineers (2026)

Why causal inference, not A/B tests

A/B testing is the gold standard when you can randomize. Causal inference is the toolkit when you can not. The most common reasons randomization is not available in production:

• Policy rollouts. A new feature rolls out to specific markets at specific times based on legal / regulatory / business considerations. Randomization was not an option; the rollout sequence determined who saw the feature when. DiD or synthetic-control is the natural fit.
• Self-selection. Users opt into a feature (newsletter signup, premium subscription, beta program). The opted-in cohort is not comparable to the non-opted cohort because they self-selected based on unobserved characteristics. Propensity-score matching or IV is the natural fit.
• External events. A regulatory change, a competitor launch, a macroeconomic shock affects some users but not others. DiD or synthetic-control captures the causal impact.
• Spillover and network effects. A/B tests assume the stable-unit-treatment-value-assumption (SUTVA): treating one user does not affect another. In two-sided marketplaces (Airbnb, Uber, Doordash), SUTVA is violated — treating one side of the market affects the other. Cluster-randomized A/B tests or causal-inference methodologies are needed.

Hernán and Robins's "Causal Inference: What If" (free PDF at hsph.harvard.edu/miguel-hernan/causal-inference-book) is the canonical practitioners' textbook. Susan Athey's NBER work (athey.people.stanford.edu/research) covers tech-industry-applicable causal-inference methodology, including her work with Stefan Wager on causal forests and her work with Guido Imbens on heterogeneous treatment effects.

Difference-in-differences: a worked example

Difference-in-differences (DiD) is the most-used causal-inference methodology at FAANG analytics-DS in 2026. The intuition: compare the change in outcome between a treated group and a control group, before and after treatment. The "difference of differences" cancels out time-invariant confounders.

A worked example. Suppose you are a senior DS at a streaming platform; the company rolled out a new pricing tier in Country A starting January 1, 2026, while Country B (similar in demographics) did not get the rollout. You want to know: did the pricing tier affect retention?

import pandas as pd
import statsmodels.formula.api as smf

# Long-format data: one row per user-month
# Columns: user_id, country (A or B), month, retention (0 or 1)
df["treated"] = (df["country"] == "A").astype(int)
df["post"] = (df["month"] >= "2026-01").astype(int)
df["did"] = df["treated"] * df["post"]

# DiD regression with two-way fixed effects
model = smf.ols(
    "retention ~ did + treated + post + C(month) + C(country)",
    data=df,
).fit(cov_type="cluster", cov_kwds={"groups": df["country"]})

print(model.summary())
# The coefficient on `did` is the DiD estimate of the causal effect.

The senior DS conversation around this code: (1) the parallel-trends assumption — does retention in country A and country B move in parallel before the rollout? Verify by plotting pre-period trends. If the pre-trends diverge, DiD is invalid. (2) The clustered standard errors — clustering at the country level accounts for within-country correlation. (3) Two-way fixed effects — month FE absorbs time shocks; country FE absorbs time-invariant country differences.

Failure modes that show up in production:

• Pre-trend violation. If country A and country B do not share parallel pre-period trends, the DiD estimate conflates the policy effect with the trend differential. Plot the pre-period trends; abandon DiD if they do not overlap visually.
• Staggered rollout bias. If treatment rolled out to multiple groups at different times, naive two-way fixed effects produces biased estimates (Goodman-Bacon 2021 decomposition). Use Callaway and Sant'Anna estimator (bcallaway11.github.io/did) instead.
• Spillover. If country A and country B share users who travel between them, treatment in A can spill over to B. Cluster-randomization at the country level helps; explicit modeling of spillover is needed when it is material.

Synthetic control: when there is only one treated unit

Synthetic control is the methodology when you have one treated unit (one market, one product variant) and many candidate-control units. Abadie, Diamond, and Hainmueller's original paper (web.stanford.edu/~jhain/Paper/JASA2010.pdf) introduced the methodology to study the effect of California's 1988 anti-tobacco legislation; the methodology has since become canonical for policy-rollout analyses in tech.

The intuition: construct a synthetic counterfactual by taking a weighted combination of control units that closely match the treated unit's pre-treatment trajectory. The weighted combination — the "synthetic California" — represents what California would have looked like absent the treatment. The post-treatment difference between treated unit and synthetic counterfactual is the causal effect.

A tech-industry application: a streaming platform launches a major UI redesign first in Argentina (the treated market). 30 other markets serve as candidate controls. Goal: measure the redesign's causal impact on session-completion rate.

from synthdid import SynthDID  # github.com/synth-inference/synthdid
import pandas as pd

# Data: market * month * outcome
# Treated unit: Argentina; treatment month: 2026-01
df_treated = df[df["market"] == "Argentina"]
df_controls = df[df["market"] != "Argentina"]

model = SynthDID(
    data=df,
    treated_unit="Argentina",
    treatment_period="2026-01",
    outcome_col="session_completion_rate",
    unit_col="market",
    time_col="month",
)
model.fit()
print(f"Estimated treatment effect: {model.att:.4f}")
print(f"95% CI: ({model.ci_lower:.4f}, {model.ci_upper:.4f})")

The senior DS conversation: (1) which markets contribute weight to the synthetic counterfactual? (Look at the weights — interpretability is built in.) (2) Does the synthetic counterfactual track the treated unit's pre-treatment trajectory? (Plot pre-treatment fit — if it is poor, the methodology cannot be trusted post-treatment.) (3) Permutation inference — apply the synthetic control to each control unit as a placebo; if the placebo effects are similar magnitude to the treated effect, the result is not statistically distinguishable from chance. The Abadie et al. paper covers all three.

Heterogeneous treatment effects: causal forests and uplift

Heterogeneous treatment effect estimation answers: "for whom does the treatment work?" not just "does the treatment work on average?" This is increasingly important at growth-stage and FAANG companies for personalization.

• Causal forests (Wager and Athey 2018, arxiv.org/abs/1510.04342). A random-forest variant that estimates conditional average treatment effects (CATE) — the treatment effect as a function of covariates. Implementation: EconML (econml.azurewebsites.net), grf in R.
• Uplift modeling. A more applied flavor of CATE estimation, oriented toward "who should we target with this treatment to maximize ROI?" CausalML (github.com/uber/causalml) was originated at Uber for personalization use cases.
• Double machine learning (Chernozhukov et al. 2018). Provides valid inference for treatment effects when nuisance parameters are estimated with machine learning. The DoubleML package (doubleml.org) is the canonical implementation.

Senior DS at Meta and Airbnb explicitly use heterogeneous-treatment-effect methodology for personalization decisions — for which user segments is a feature beneficial vs neutral vs harmful? The senior bar: articulate when CATE estimation provides actionable signal vs when it produces overfitted noise that does not hold in subsequent A/B tests.