KOEN
Read Original
#Backend

How Netflix Simplified Batch Compute with Kueue

How Netflix Simplified Batch Compute with Kueue
01

Summary

Why Netflix Replaced Its 6-Year-Old Custom Batch System with 'Kueue'

Secrets to optimizing resources and operational efficiency by migrating millions of jobs without disruption.

This article explores Netflix's journey of transitioning its homegrown Compute Managed Batch (CMB) to the Kubernetes-native Kueue. It details the strategies and technical challenges involved in reducing complexity while gaining advanced scheduling capabilities like preemption.

  • 01Reduced maintenance overhead by opting for Kueue instead of building custom scheduling features.
  • 02Achieved a 'Zero-lift' migration strategy by maintaining API parity for end-users.
  • 03Mitigated risks by migrating the most complex customers first, completing production rollout in 4 weeks.
  • 04Maximized idle resource utilization through Preemption-based Fair Sharing.
  • 05Practical performance tuning of Kueue controllers to handle high-scale batch processing.

RECOMMENDATION

Highly recommended for infrastructure engineers managing large-scale Kubernetes batch workloads or ML training infra who need to balance resource isolation and sharing.

The Problem

Netflix's legacy batch solution, CMB, was a custom system built in 2018 that became increasingly difficult to maintain as modern features like preemption and fair sharing became standard in the Kubernetes ecosystem.

The Solution

The team adopted Kueue, a cloud-native job queueing system, to replace CMB's custom logic. They mapped the existing tenant hierarchy to Kueue's Cohorts and ClusterQueues while maintaining API parity to ensure a transparent migration for users.

The Result

Millions of batch jobs were successfully migrated, resulting in a significant increase in resource utilization through preemption-based fair sharing. The system handles high throughput after tuning controller QPS and concurrency for Netflix scale.

Trade-off

To meet Netflix's throughput requirements, the default Kueue configurations for QPS, Burst, and concurrency had to be significantly increased, requiring extensive load testing to ensure stability under heavy load.

03

Key Concepts

Concept · 01

Kueue

A Kubernetes-native job queueing system that manages job admission and quotas without replacing the underlying kube-scheduler.

  • Integrated with Titus to manage batch job queueing and scheduling logic.
Concept · 02

Preemption-based Fair Sharing

A mechanism that allows higher-priority jobs to preempt lower-priority ones and enables borrowing of unused reserved capacity.

  • Used to shorten turnaround times for business-critical tasks and improve overall resource efficiency.
Concept · 03

Cohort

A logical grouping in Kueue that allows multiple ClusterQueues to share resources within a common pool.

  • Mapped to CMB's internal tenant structure to facilitate hierarchical resource sharing.