Data Engineering for AI: Building Pipelines That Feed Production ML

Why Data Engineering Is the AI Bottleneck

Survey after survey finds the same answer: data preparation and management consumes 60-80% of data scientist and ML engineer time. The model is often 20% of the problem. The data pipeline is 80%.

Poorly engineered data causes: training/serving skew (model trained on different distribution than production), data freshness issues (stale features in real-time model), data quality problems (garbage in, garbage out), and feature leakage (using future information to predict past, inflating reported accuracy).

Feature Engineering and Feature Stores

The Feature Engineering Problem

Raw data → features that models can learn from. This process: one-hot encoding, normalization, handling nulls, creating interaction features, temporal features (days since last purchase, rolling averages).

The problem: feature engineering logic is duplicated between training pipeline and serving pipeline. They get out of sync. Training/serving skew is often subtle and devastating.

Feature Stores

Feature stores centralize feature computation, storage, and serving:

Define features once, reuse across models

Consistent offline (training) and online (serving) feature computation

Feature versioning and lineage

Point-in-time correct feature retrieval for historical training (avoids future leakage)

Components: feature definitions (code), offline store (historical features in data warehouse), online store (low-latency features for serving), feature registry (catalog of available features), materialization jobs (compute features and write to stores).

Tools: Feast (open source, most widely adopted), Tecton (managed feature store), Vertex AI Feature Store (GCP), SageMaker Feature Store (AWS), Databricks Feature Store.

Implementation example with Feast:

python
from feast import Entity, Feature, FeatureView, Field
from feast.types import Float32, Int64
customer = Entity(name="customer_id", value_type=ValueType.INT64)
customer_features = FeatureView(
    name="customer_stats",
    entities=["customer_id"],
    ttl=timedelta(days=30),
    schema=[
        Field(name="total_purchases", dtype=Int64),
        Field(name="avg_order_value", dtype=Float32),
        Field(name="days_since_last_purchase", dtype=Int64),
    ],
    source=customer_stats_source,
)

Real-Time Streaming for ML

When to Use Streaming

Batch ML: features computed over historical data, model served on demand. Sufficient for: fraud detection with historical patterns, recommendation with daily updates, classification with stable features.

Streaming ML: features computed from real-time event streams, model reflects current state. Required for: fraud detection needing current session behavior, real-time personalization, anomaly detection on live sensor data.

Kafka + Flink for ML Features

Apache Kafka: event streaming backbone. All business events (purchases, clicks, logins, sensor readings) published as events.

Apache Flink: stateful stream processing. Compute features from event streams: rolling aggregates (purchases in last 30 minutes), session features (click sequence patterns), real-time anomaly detection.

Simplified Flink feature pipeline:

Input: Kafka topic with user events

Processing: session window (30 min), count events by type, compute behavioral features

Output: feature store (Redis) for real-time model serving

Lambda Architecture for ML

Batch layer: recompute features over all historical data nightly. High accuracy, high latency. Speed layer: compute features from real-time events. Lower accuracy (partial data), low latency. Serving layer: merge batch and speed layer features for serving.

Modern alternative: Kappa architecture. One streaming pipeline that handles both historical and real-time. Simpler operationally. Made practical by Kafka replay and large state backends in Flink/Spark Streaming.

Training Data Quality

Data Quality Dimensions

Completeness: are required fields present? Null rate per feature, record count vs. expected. Accuracy: are values correct? Statistical validation (distribution checks, range checks), spot-checking against source. Consistency: same fact represented consistently across systems. Timeliness: is data current? Freshness SLAs per data source. Validity: does data conform to schema? Type checking, enum validation. Uniqueness: no duplicates in training data (data contamination).

Data Validation Frameworks

Great Expectations: Python library for data validation. Define expectations (test assertions), run against data, alert on failures.

python
import great_expectations as gx
context = gx.get_context()
batch = context.sources.pandas_default.read_dataframe(df)
validator = context.get_validator(batch_request=batch)
validator.expect_column_values_to_not_be_null("user_id")
validator.expect_column_values_to_be_between("age", min_value=0, max_value=150)
validator.expect_column_values_to_be_in_set("status", ["active", "inactive", "suspended"])
results = validator.validate()

Deequ (Amazon, Spark-based), dbt tests, Soda Core—all provide data quality validation at different layers of the stack.

Training Data Monitoring

Training data distributions shift over time. Models trained on historical data may become less accurate as the world changes.

Data drift detection: monitor input feature distributions over time. Compare current distribution to training distribution (KL divergence, population stability index, Kolmogorov-Smirnov test).

Tools: Evidently AI, Whylogs, Amazon SageMaker Model Monitor, Arize AI.

Data Labeling Workflows

The Labeling Problem

Supervised learning requires labeled data. High-quality labels are expensive. Crowdsourced labels are noisy. The labeling bottleneck slows AI development.

Labeling Tools

Label Studio (open source): general-purpose labeling tool. Text classification, NER, image segmentation, audio transcription.

Scale AI: enterprise labeling platform with human-in-the-loop QA. Best quality, highest cost. Used by autonomous vehicle companies, defense.

Labelbox, Prodigy, Roboflow: mid-market options with strong tooling for specific data types.

Active Learning

Instead of labeling randomly, active learning selects the examples that will most improve the model:

Train model on labeled subset

Apply model to unlabeled data

Identify examples model is least confident about

Label only those examples

Retrain and repeat

Result: achieve equivalent model performance with 30-70% less labeling. Particularly valuable when labeling is expensive (medical image annotation, expert review).

Data Versioning

Why Version Data

"My model accuracy dropped 5% last week. What changed?" Without data versioning: impossible to answer. With: compare training data from last week to this week, identify distribution shift.

DVC (Data Version Control)

Git for data and models. Track data files in Git-compatible way without storing large files in Git.

bash
Initialize DVC in your repo
dvc init
Track a large dataset
dvc add data/training_set.csv
Push to remote storage (S3, GCS, Azure Blob)
dvc push
Checkout previous version
git checkout v1.2
dvc checkout  # Restores data files matching this git commit

Lakehouse Architecture

Modern data architecture for AI: Delta Lake or Apache Iceberg on cloud storage (S3/GCS/Azure Blob). Provides:

ACID transactions on large datasets

Schema evolution

Time travel (query historical versions of data)

Unified batch and streaming

Tools: Databricks (Delta Lake), Apache Hudi, Apache Iceberg on AWS Glue or Snowflake.

Modern Data Stack for AI

Reference architecture for an AI-first data platform:

Ingestion: Fivetran, Airbyte, or custom ingestion → raw data lake (S3)

Transformation: dbt for business logic transforms → feature engineering in Spark or dbt → feature store

Warehousing: Snowflake, BigQuery, or Databricks SQL → analytics and model training data

Streaming: Kafka → Flink → feature store (online)

Orchestration: Airflow or Prefect for pipeline scheduling and monitoring

Data quality: dbt tests + Great Expectations

Experiment tracking: MLflow or Weights & Biases

Model registry: MLflow or SageMaker Model Registry

Each layer is swappable. The architecture pattern is stable even as specific tools evolve.