KOEN
Read Original
#DevOps

Netflix Stratum Media Processing: Automated Container Right-Sizing

Netflix Stratum Media Processing: Automated Container Right-Sizing
01

Summary

How Netflix Automatically Right-Sizes Hundreds of Thousands of Serverless Containers Daily

An inside look at Stratum Resource Tuner—Netflix's safe, automated container right-sizing engine

Stratum Resource Tuner (SRT) automatically optimizes CPU, memory, disk, and network allocations for Netflix's media processing containers without developer toil. By combining long-term usage analytics in Iceberg with automated verification and a 1.5x auto-retry fail-safe, Netflix reduces hardware waste at scale while maintaining platform reliability.

  • 01Analyzes long-term sub-minute resource metrics stored in Apache Iceberg via Keystone streaming
  • 02Applies tier-specific logic (p99 for CPU/network, p99.9 + safety buffer for memory/disk)
  • 03Enforces a safety clamp limiting single-cycle memory and disk reductions to a maximum of 15%
  • 04Self-heals edge cases via automated task retries on 1.5x larger containers when usage exceeds 85/95% thresholds

RECOMMENDATION

Highly recommended for DevOps, infrastructure, and platform engineers managing large-scale containerized batch or serverless workloads looking to automate resource optimization and cut cloud infrastructure costs.

The Problem

Netflix's serverless media processing platform, Stratum, suffered from widespread container over-provisioning caused by manual YAML configurations. Evolving cloud hardware, changing video codecs, and expanding media formats made manual optimization impractical and high-risk for developers.

The Solution

Netflix built the Stratum Resource Tuner (SRT), an automated system that right-sizes container resources in production without human intervention. SRT collects usage metrics via Keystone/Iceberg, generates conservative recommendations via Conductor workflows, and incorporates a fail-safe mechanism that retries tasks on a 1.5x larger container upon resource spikes.

The Result

SRT successfully optimized high-throughput workloads (representing 70-80% of total platform usage) and significantly reduced wasted compute and memory capacity. For example, the CPU request for VideoEncoder.encodeV1 was halved from 8 to 4 CPUs and memory was downsized from 18,000 MB to 15,300 MB safely.

Trade-off

Triggering the fallback retry mechanism causes transient task execution delays and additional compute overhead during workload spikes. Additionally, limiting single-cycle reductions to 15% slows convergence to optimal sizing, and architecture constraints currently necessitate pre-set usage thresholds (85%/95%) instead of true OOM event detection.

03

Key Concepts

Concept · 01

Stratum Resource Tuner (SRT)

An automated system at Netflix that safe-guards and right-sizes container resource limits across the Stratum serverless media processing platform.

  • Generates daily recommendations from historical usage trends and safely deploys validated configs.
  • Integrates automated verification workflows and post-apply stability monitoring.
Concept · 02

Titus

Netflix's container management platform that executes containerized workloads at massive scale.

  • Executes packaged Stratum OCI container functions as batch jobs triggered by upstream services.
Concept · 03

Retry with More Resources

A fail-safe mechanism that cancels container execution upon approaching resource limits and reschedules the job on a larger container instance.

  • Triggers when non-cached memory hits 85% or disk hits 95%, re-queueing the task on a container with 1.5x larger limits.