Detecting Concept Drift Without Ground Truth
Every ML team learns the same hard lesson eventually: models degrade in production. The world changes. Distributions shift. And by the time you have labelled data to prove it, the damage is done.
The standard drift detection playbook — monitor prediction distributions, wait for label lag to catch up, retrigger training — adds 12–48 hours of latency to the detection loop. For high-stakes production systems, this is too slow.
The Core Insight
If we cannot wait for labels, we must detect drift from inputs alone. The challenge: distinguishing covariate shift (the input distribution changed) from natural variation (inputs vary but the decision boundary hasn't moved).
We frame this as a two-sample test problem. At training time, we learn a low-dimensional representation of the input manifold using a contrastive encoder. At inference time, we run a streaming Maximum Mean Discrepancy (MMD) test between a sliding window of live inputs and the training distribution embedding.
Detection Performance
On our internal sensor fusion benchmark (injected drift at known timestamps):
For reference, waiting for label lag in this system averages 19 hours.