What Is Training-Serving Skew?
Training-serving skew is a difference between a machine learning model's performance during training and its performance during serving, often because the model receives different inputs in production than it was trained on. Common causes are feature logic implemented twice, once for batch training and once for online serving; data that changes between training and serving; and feedback loops between the model and the system around it. Skew rarely raises an error. The model keeps returning predictions that look normal and are measurably worse.
Updated
What is training-serving skew?
Google's Rules of Machine Learning, by Martin Zinkevich, defines training-serving skew as "a difference between performance during training and performance during serving" and lists three causes: "A discrepancy between how you handle data in the training and serving pipelines," "A change in the data between when you train and when you serve," and "A feedback loop between your model and your algorithm." Google's ML glossary uses the same definition.
Several rules target it. Rule #29 says to "save the set of features used at serving time, and then pipe those features to a log to use them at training time," and reports that "YouTube home page switched to logging features at serving time with significant quality improvements and a reduction in code complexity." Rule #32 says to "Re-use code between your training pipeline and your serving pipeline whenever possible." Rule #37 is "Measure Training/Serving Skew."
Skew shows up per example or in aggregate: the same example can get a different feature value at serving than in training, or the overall distribution of values can differ. The TensorFlow Data Validation guide covers the second: distribution skew "occurs when the distribution of feature values for training data is significantly different from serving data." TFDV measures drift separately, between consecutive spans of data.
How training-serving skew works
The usual mechanism is one feature with two implementations. An illustrative fraud model uses txn_count_1h, the card's transactions in the past hour.
- 1.Training: a batch SQL job computes the count from the complete transaction log, for the hour before each labeled transaction, including declined attempts.
- 2.Serving: a stream job maintains the count in a cache, counts only approved transactions, and trails the newest events by its processing lag.
- 3.The result: the same card at the same moment shows 8 in training and 5 in serving. The model learned that 8 is high risk, so in production it under-scores the card. Nothing fails and no alert fires.
Rule #29's fix doubles as the measurement: log the values the model received, recompute them with the training pipeline, and compare.
-- served_features: values the model received, logged per request
-- training_features: the same features recomputed by the batch pipeline
SELECT s.feature_name,
count(*) AS requests,
avg((s.value IS DISTINCT FROM t.value)::int) AS share_mismatched,
avg(abs(s.value - t.value)) AS mean_abs_diff
FROM served_features s
JOIN training_features t
ON t.request_id = s.request_id
AND t.feature_name = s.feature_name
WHERE s.served_at >= now() - interval '1 day'
GROUP BY s.feature_name
ORDER BY share_mismatched DESC;Why training-serving skew matters in real-time systems
Freshness is a skew source of its own. Training features are computed after the fact from complete history, so they reflect every event up to the label's timestamp. Serving features come from a store that trails its sources by pipeline lag. The faster the state changes, the further apart the two values drift. Velocity counts, balances and session state change fastest, so they are the features most exposed to it.
There are two structural answers: train on logged serving features, or define each feature once and run that definition for both. The second is Tacnode's approach: a feature is defined once in SQL, the query that builds training data and the serving read run the same definition against the same tables, and the serving read can be computed on demand against the latest committed data. That removes the duplicate-code cause. Data that changes between training and serving, and feedback loops, are separate causes, so logging served features stays the check. The serving side is covered in real-time ML inference and feature freshness.
FAQ
Related terms
Point-in-time correctness means each training example gets feature values as they stood at the label's timestamp, never later ones, so no future data leaks.
Derived context is state computed from events rather than stored directly — aggregates, velocity counts, features, embeddings — that decisions consume.
The divergence problem: services each maintain their own pipeline over the same events, advancing at different paces, so their views of shared state disagree.
Context lag is the delay between a fact changing at its source and the system an automated decision reads from reflecting it. Also called stale context.
A freshness budget is the maximum staleness a decision can tolerate in its inputs — and it equals the worst input, since one slow signal spends it all.
State velocity is the rate at which the state a decision depends on changes. High velocity plus concurrency is what makes cached context structurally invalid.
A real-time feature store serves ML features that reflect events from seconds ago, fast enough for online prediction, using streaming and request-time features.