KOEN
Read Original
#Backend

Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II

Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II
01

Summary

How Netflix Scaled Ad Event Pipelines for Live Events to 100M+ Viewers

Eliminating database bottlenecks on the ad-serving critical path through stateful stream processing with Apache Flink

This article details how Netflix evolved its Ads Event Publisher pipeline to support massive live streaming events like NFL games and fight nights. By moving away from synchronous database writes and adopting real-time streaming joins in Apache Flink, Netflix achieved elastic scalability, lower serving latency, and high system reliability.

  • 01Removed database writes from the critical path, eliminating a major single point of failure
  • 02Implemented real-time streaming joins using a custom Flink KeyedCoProcessFunction with a 60-minute window
  • 03Avoided costly global data replication by dynamically routing cross-region client events based on embedded tokens
  • 04Built a companion Spark and Maestro batch recovery path to reliably process long-tail late-arriving events
  • 05Executed a zero-downtime migration with deterministic key-based cutover and automated audit comparisons

RECOMMENDATION

Essential reading for backend and data platform engineers looking to migrate high-throughput synchronous database architectures to stateful, real-time stream processing systems.

The Problem

Netflix's legacy ads enrichment pipeline relied on the ad server writing metadata synchronously to a Key-Value store on the serving path. Under live streaming events with tens of millions of concurrent viewers, this synchronous write pattern created a massive scaling bottleneck and posed a single point of failure risks on the ad serving critical path.

The Solution

Netflix replaced the synchronous database lookup with a stateful stream processing architecture using Apache Flink. By streaming both ad serving responses and client events through Kafka, Flink performs a real-time streaming join using a KeyedCoProcessFunction within a 60-minute retention window, backed by a Spark batch job orchestrated by Maestro for long-tail event recovery.

The Result

The team completely eliminated database write dependencies from the ad serving critical path, removing scaling bottlenecks for live events. Through shadow testing and automated audits, they verified over 99.99% output consistency before smoothly decommissioning the legacy registry and service without downtime.

Trade-off

Restricting the in-memory state window to 60 minutes means that late-arriving long-tail events (~0.001%) cannot be enriched in real time and must rely on an offline Spark batch recovery path, introducing slight processing latency for edge cases.

03

Key Concepts

Concept · 01

Streaming Join

A stream processing technique that combines events from two distinct data streams in real time based on a shared key, eliminating the need for intermediate database lookups.

  • Used to match client-side ad playback events with ad server response metadata in real time via Kafka streams.
Concept · 02

KeyedCoProcessFunction

A low-level stateful operator in Apache Flink that allows granular control over state retention and timers across two keyed streams.

  • Used to build a custom streaming join operator holding event state up to 60 minutes for real-time ad enrichment.
Concept · 03

At-Least-Once with Best-Effort Deduplication

A system design pattern ensuring event delivery at least once while performing in-stream deduplication via localized state, leaving exact deduplication to downstream consumers with longer time windows.

  • Used in Flink with deterministic event ID hashing to filter duplicate client events locally without network roundtrips.