MLOps data contracts: point-in-time features, training-serving consistency and promotion
Connect training data, feature transformations, serving inputs and model promotion. Detect temporal leakage and training-serving skew with a worked point-in-time feature example and explicit quality gates.
TL;DR: MLOps must preserve the meaning and availability of data across training and serving, alongside the identity of the model artifact. Test feature contracts and prediction-time information boundaries before trusting an offline score or promoting an automatically retrained model.
The deployable unit includes a data dependency
A prediction service consumes features: values derived from events, records or user input. The meaning of those values is part of the model's operating contract. A field called spend could mean lifetime rupees, the previous thirty days in rupees, or the same interval in paise. A matching column name and numeric type cannot detect those differences.
Document entity keys, units, window boundaries, missing-value behavior, transformation version and freshness requirements. Version the serving transformation with the model where they must change together. Model release provenance covers artifact identity; this page concentrates on the data those artifacts consume.
Google's MLOps architecture guidance separates pipeline automation, data validation and model validation. Retraining can automate candidate creation without granting every candidate permission to serve production traffic.
| Contract | Example violation | Suitable check |
|---|---|---|
| Schema | Required feature disappears | Validate input names, types and permitted nulls |
| Meaning | Currency unit changes by a factor of 100 | Compare transformations on known fixtures |
| Time | Training joins information learned after prediction | Reconstruct features available at that time |
| Quality | Aggregate score improves while a critical segment regresses | Evaluate declared slices and promotion criteria |
Reconstruct what the prediction could have known
Suppose a fraud model predicts at 10:00. A payment event occurred at 09:58 but only reached the feature pipeline at 10:04. A training job run tomorrow can see that event. The production model at 10:00 could not. Using it in training creates a timing advantage that disappears at serving time.
Point-in-time joins retrieve historical feature values relative to the example timestamp. Feast's point-in-time guide also explains that feature TTL is measured relative to each example's timestamp, not the day the training query runs. Where arrival delays matter, record information availability separately from event time.
The following local model makes both checks explicit. Integer times are illustrative minutes on one clock; the keys are teaching data, not Feast API fields:
features = [
{"event": 95, "available": 96, "value": 20},
{"event": 98, "available": 104, "value": 80},
{"event": 101, "available": 101, "value": 90},
]
def feature_at(rows, prediction_time, ttl):
eligible = [r for r in rows
if prediction_time - ttl <= r["event"] <= prediction_time
and r["available"] <= prediction_time]
return max(eligible, key=lambda r: r["event"])["value"] if eligible else None
assert feature_at(features, 100, 10) == 20
assert feature_at(features, 100, 2) is None
assert feature_at(features, 104, 10) == 90
Selecting the latest row from the completed dataset would return 90 for the 10:00 example. Selecting by event time alone would return 80. Neither reconstructs what was available then. Real datasets need entity grouping, deterministic handling of corrections and ties, and a defined fallback when no valid feature exists.
Validate meaning as well as distributions
Schema checks can catch an absent field. Distribution checks can detect unusual values or changing populations. TensorFlow Data Validation documents schema, skew and drift validation. These signals need interpretation: a holiday can legitimately change traffic, while a currency conversion bug changes feature meaning.
Use shared transformation code where practical, then compare offline and online output on the same input fixtures. Shared code still permits mismatched configuration, time zones or external lookups. Include boundary cases such as midnight, missing values and late events. A feature store helps organize retrieval; it does not automatically repair a wrong entity key or an inappropriate training cutoff.
Do not retrain automatically in response to every distribution alarm. If a pipeline converted rupees to paise incorrectly, retraining on that defect can produce another apparently valid candidate with the wrong contract. Fix the data path and assess affected predictions first.
Promote with quality and operating evidence
Retain a holdout appropriate to the prediction task and test relevant segments. Future labels must not leak into features, and repeated evaluation should not gradually turn the holdout into a tuning set. Serving validation also needs latency, resource demand and fallback behavior, because a high-scoring model that cannot obtain timely features may be unusable.
Self-check: offline accuracy rises after a backfill, but online performance does not. The model artifact and serving code are unchanged. What should you examine before increasing compute? Compare feature cutoffs and availability, transformations and population composition. A backfill can improve the historical dataset using information production never had at prediction time.