KOEN
Read Original
#Backend

State of Routing in Model Serving

State of Routing in Model Serving
01

Summary

Routing 1 Million ML Inferences per Second: The Netflix Way

Evolution from Switchboard to Lightbulb: Designing for Extreme Scale and Rapid Experimentation

This article explores Netflix's technical journey in solving complex routing challenges for large-scale ML model serving. It covers the architectural shift from a centralized proxy strategy to a decentralized Envoy-based approach, focusing on performance optimization and developer velocity.

  • 01Operational insights into managing global ML serving infrastructure at 1M+ RPS
  • 02Designing the 'Objective' layer to decouple business logic from model implementation
  • 03Utilizing JavaScript-based Switchboard Rules for flexible traffic splitting and A/B tests
  • 04Addressing SPOF and latency bottlenecks by evolving towards the Lightbulb architecture
  • 05Leveraging Envoy's sidecar capabilities for efficient and scalable traffic management

RECOMMENDATION

A must-read for infrastructure engineers and MLOps professionals managing large-scale microservices. The 'Objective' abstraction model is particularly valuable for teams needing to implement complex A/B testing without impacting client-side stability.

The Problem

Netflix needed to enable researchers to rapidly experiment with and deploy ML models at a scale of 1M+ RPS without exposing the underlying infrastructure complexity or model sharding details to client microservices. The core challenge was ensuring stable routing across hundreds of model versions and dynamic cluster configurations with zero client-side code changes.

The Solution

The team introduced the 'Objective' abstraction for business use cases and initially built 'Switchboard,' a centralized custom proxy layer for context-aware routing and A/B testing. This evolved into the 'Lightbulb' architecture, which decouples the control and data planes using Envoy proxies to minimize latency and improve system reliability.

The Result

The architecture achieved complete decoupling between client services and model versions, allowing independent experimentation for researchers while maintaining high availability at 1M+ RPS. The transition to Lightbulb eliminated the 10-20ms serialization latency inherent in the previous proxy-based approach.

Trade-off

The initial Switchboard proxy introduced a single point of failure (SPOF) risk and significant latency overhead due to serialization/deserialization. Additionally, shared resources across different tenants led to noisy neighbor issues and inconsistent performance across various service tiers.

03

Key Concepts

Concept · 01

Objective

An abstract identifier representing a business use case (e.g., ranking, fraud detection) provided in client requests. It shields clients from the specifics of model versions and backend cluster locations.

  • Acts as a contract between the calling app and the ML serving platform
  • Enables seamless model selection and A/B cell allocation without client coordination
Concept · 02

Switchboard

A custom-built centralized proxy that serves as the entry point for all ML serving traffic, handling context-aware routing and lifecycle management.

  • Integrates deeply with Netflix's internal experimentation and fact store ecosystems
  • Encountered challenges with tail latency amplification and single-point-of-failure risks
Concept · 03

Lightbulb Architecture

The next-generation routing framework designed to improve isolation and reduce latency. It separates the selection of a model from the actual routing of the request.

  • Removes the routing service from the direct request path using Envoy
  • Decouples model inputs from request metadata to reduce serialization overhead