Apache Kafka for Real-Time ML Pipelines

Why Streaming ML?

Batch ML is insufficient for: fraud detection (must act in under 100ms), real-time personalization (user preferences change per session), anomaly detection (detect and respond to issues as they happen), dynamic pricing (react to supply/demand changes in real time).

Streaming ML uses Apache Kafka as the nervous system, with stream processors computing features and ML models serving predictions in real time.

Kafka Architecture for ML

Producer → Kafka Topics → Consumer Groups → ML Model → Output Topic.

Kafka guarantees message durability, ordering within partitions, and exactly-once semantics (with Kafka Streams or Flink). Key concepts: topics (named streams of records), partitions (parallelism unit), consumer groups (load-balanced consumption), offsets (position in stream).

Real-Time Feature Computation

Kafka Streams for Feature Engineering

Count events in sliding windows: create a KTable that counts click events per user in a 5-minute hopping window, updating every 1 minute. This feature is available for ML serving within 1 minute of each click event.

Join streams: enrich transaction events with user profile data from a KTable (replicated from database). Result: transactions with user context, available for fraud scoring in under 100ms.

Apache Flink for Complex Event Processing

Flink provides: event time processing (using event timestamps, not processing time), complex event patterns (detect sequences of events), stateful computation (maintain state per user/entity), exactly-once semantics even with failures.

Flink ML: real-time model inference within Flink operators. Feed Kafka events → compute features → run ML model → emit predictions → back to Kafka. Entire pipeline in under 50ms.

Fraud Detection Pipeline

Architecture: transactions → Kafka → Flink computes velocity features (count/amount in 1min, 5min, 60min windows) → joins with user history from Redis → ML model scores transaction (< 10ms) → high-score transactions → alert topic → case management system.

Velocity features: transaction_count_1m, transaction_count_5m, total_amount_1h, unique_merchants_24h, distance_from_last_transaction. These are computed in real-time by Flink and pushed to Redis for sub-millisecond lookup during ML inference.

Real-Time Recommendation Engine

Event stream: user clicks, views, purchases → Kafka. Feature computation: Flink maintains user interest vectors (TF-IDF over categories viewed in past 30 minutes), item popularity scores (view counts in past 1 hour). ML model: collaborative filtering or two-tower neural network. Serving: given user context features + item candidate features → rank top-K recommendations → return within 50ms.

Online Feature Store Integration

Kafka → Flink → Online Feature Store (Redis/DynamoDB). Flink writes computed features to the online store after each event or window update. ML serving reads latest features from the online store at prediction time. Consistency model: features are eventually consistent with a lag of seconds to minutes.

Monitoring Streaming ML

Consumer Lag Monitoring

Alert when consumer lag exceeds threshold—means processing can't keep up with ingestion rate. Scale out consumers or optimize processing logic.

Feature Freshness Monitoring

Track how recent features are: if fraud detection model reads a velocity feature that hasn't been updated in 5 minutes, that's a pipeline failure. Alert on feature staleness.

Prediction Latency Distribution

Track p50, p95, p99 latency for end-to-end pipeline (event ingested → prediction produced). p99 under 200ms is a common SLA for real-time fraud detection.

Data Quality Monitoring

Monitor for: schema changes in incoming events (field added/removed), null rates above threshold, value distribution shifts, volume anomalies (sudden drop or spike).

Scaling Kafka ML Pipelines

Kafka handles millions of events/second through partitioning. Scale ML serving horizontally: 10 model serving pods can process 10x the throughput of 1. Use Kubernetes HPA to auto-scale based on Kafka consumer lag. Kafka Connect for integrating with databases, data warehouses, and cloud storage.

Real-time ML streaming pipelines enable business capabilities impossible with batch processing, particularly in fraud prevention, real-time personalization, and operational monitoring.