ML Feature Store Architecture: Ensuring Online-Offline Consistency

Feature stores address one of the most insidious bugs in ML systems: training-serving skew—where features are computed using one logic during training and a different one during online serving, causing silent discrepancies that make offline metrics look great while online performance tanks. This article explains the three sources of this problem, the standard architecture of a feature store, point-in-time correctness, and an honest assessment of whether you need a feature store.

1. Three Sources of Training-Serving Skew

Inconsistent computation logic: The training pipeline computes "number of orders in the last 30 days" using Spark/SQL, while the online service reimplements it with different Java code—differences in the boundary of "30 days", timezone handling, and deduplication details introduce skew.

Time leakage: Training samples use "future" data—for example, using the average computed over all historical data to train a model predicting last week's behavior. This inflates offline metrics, but the pattern cannot be reproduced online.

Freshness differences: Training uses T-1 batch data, while online uses real-time streams—the same feature is inherently out of sync between the two environments.

The core promise of a feature store: Define features once, consume them consistently in both environments.

2. Standard Architecture

text
Feature definitions (code, versioned)
   |  Single transformation logic
   ├── Offline store (Parquet/Data Warehouse) ──→ Training: point-in-time join for historical snapshots
   └── Online store (Redis/DynamoDB) ──→ Serving: millisecond key-based lookup for latest values
        ↑ Materialization (batch backfill + streaming real-time updates)

Offline side stores full history for training and backtesting; online side stores only the latest feature value per entity for low-latency inference.

Materialization is the synchronization mechanism: values computed from the same definition are written to both stores—consistency is guaranteed by a "single definition", not by two teams being careful.

Point-in-time join is the critical operation on the training side: for each training sample, it retrieves the feature value as it was at the sample's timestamp, mechanically preventing time leakage. This is the hardest part to implement correctly by hand.

Mainstream options: open-source Feast (most common starting point), Tecton (managed commercial), cloud vendor offerings (SageMaker/Vertex Feature Store), and built-in solutions in Databricks/Snowflake ecosystems.

3. When Do You Really Need It?

Need it: Multiple models share the same set of features, online inference requires real-time features (fraud detection/recommendation/pricing), team size ≥ a few people with training and serving maintained by different individuals—here the coordination value of "single definition" pays off.

Don't need it: Single model, batch prediction, features can be computed on the fly at request time—an offline feature table plus request-time computation suffices; a platform would be over-engineering. Honest rule of thumb: Only adopt after you've been bitten by training-serving skew once—then you'll know what you're buying.

A footnote for the LLM era: In RAG/Agent applications, "features" are mostly text and embeddings, following the vector store and prompt assembly route. But when LLM applications need structured user features in prompts (e.g., "this user's activity level in the last 30 days"), the online side of a feature store serves as the data retrieval endpoint—the two infrastructures are converging in agent systems. Traditional tabular ML (insurance pricing, risk control) remains the primary domain of feature stores.

4. Practical Tips for Adoption

Version-control feature definitions, and require code review for changes—features are shared APIs; casual modifications can silently break downstream models.

Monitor consistency between the two sides: periodically sample and compare online values against offline recomputed values, and alert on deviations beyond a threshold (profile-based monitoring also applies to feature distribution drift).

Explicitly define freshness SLAs: each feature should have a "staleness threshold"; the degradation behavior when a stale value is retrieved (use default / reject prediction) should be coded.

Start with Feast + existing infrastructure (offline: existing data warehouse, online: existing Redis), validate the workflow before considering a managed platform.

FAQ

Q: What is the relationship between a feature store and a data warehouse? The data warehouse is the offline foundation; the feature store adds three things on top that the warehouse doesn't handle: online serving, point-in-time semantics, and definition reuse.

Q: Do real-time (streaming) features require Flink? It depends on freshness requirements: minute-level freshness can be achieved with micro-batch backfill; second-level (anti-fraud) requires a streaming pipeline, which is an order of magnitude more complex—don't adopt it just for the sake of it.

Q: How does it integrate with the model registry in MLOps? The registry manages model versions, while the feature store manages feature versions—both version IDs should be pinned in the training record to fully reproduce a training run (similar to registry practices).

---

*Last updated: June 2026.*