KOEN
Read Original
#Backend

A Tale of Two Flink Autoscalers

A Tale of Two Flink Autoscalers
01

Summary

How Netflix Auto-Scales 30,000+ Flink Jobs to Slash $1.1 Million in Annual Compute Costs

A pragmatic transition from a homegrown external metrics scaler to a customized OSS Flink Autoscaler powered by Temporal

This article shares Netflix's engineering journey of moving from their custom-built external autoscaling tool to the community-driven Apache Flink Autoscaler. To adapt the OSS logic to Netflix-scale workloads, they built a robust orchestration service powered by Temporal and resolved critical metric and engine-level bottlenecks. The post provides deep insights into metric reliability, sensible defaults, and future optimizations using disaggregated state.

  • 01Provides a transparent blueprint for the build vs. buy decision-making process in platform engineering.
  • 02Utilizes the True Processing Rate (TPR) algorithm to dynamically adjust vertex parallelism based on real processing capability.
  • 03Isolates scaling blast radiuses by wrapping each job's evaluation and actuation in its own resilient Temporal workflow.
  • 04Overcomes JobManager bottlenecks by adding server-side metric filtering and caching to support jobs up to 3,000 subtasks.
  • 05Trades marginal efficiency for system stability by tuning the target utilization down to a conservative 0.45.

RECOMMENDATION

Highly recommended for platform engineers, data architects, and site reliability engineers looking to scale large, stateful stream processing clusters efficiently while keeping infrastructure costs low.

The Problem

Netflix managed over 30,000 Flink jobs and faced massive inefficiencies in manually scaling resources for complex stateful DAGs. Their homegrown external autoscaler relied on coarse, container-level metrics from Atlas, which failed to perceive inner-job bottlenecks in multi-operator pipelines and could easily lead to performance degradation invisible to CPU metrics.

The Solution

Netflix adopted the community's 'Apache Flink Autoscaler' library to analyze True Processing Rate (TPR) from within the job DAG. They integrated this standalone core with a Spring Boot application orchestrated by Temporal workflows, while contributing and implementing custom optimizations to Flink's runtime, such as metric caching, preserving forward chaining, and async sink backpressure detection.

The Result

Deploying the OSS-based autoscaler allowed the client telemetry and logging team to reduce annualized Flink compute expenditures by 58%, saving approximately $1.1 million annually. The isolated, per-job workflow model eliminated failure propagation, and the scaling platform successfully supported extremely large jobs with up to 3,000 Flink subtasks.

Trade-off

To avoid aggressive downscaling that leads to CPU saturation, lag spikes, and disruptive scaling loops, Netflix compromised on absolute resource utilization by lowering the target utilization parameter to 0.45 (compared to the community default of 0.7). Additionally, the most significant scaling bottleneck remains the restart and state recovery time of Flink itself rather than the scaler's decision speed.

03

Key Concepts

Concept · 01

True Processing Rate (TPR)

The theoretical maximum throughput that an operator subtask can process if it were 100% busy without any backpressure or idle time. It is computed by dividing observed throughput by the active busy fraction.

  • Used to walk the DAG starting from sources, computing precise vertex-level parallelism to prevent bottlenecks without resizing the whole cluster uniformly.
Concept · 02

Temporal Workflow Engine

A developer-focused stateful orchestrator that manages complex, distributed, and long-running workflows with built-in persistence, retry policies, and fault isolation.

  • Netflix implemented a 'one workflow per Flink job' architecture to isolate the scaling evaluation loop, ensuring slow metrics on one job don't block others.
Concept · 03

Disaggregated State (Flink 2.0)

A decoupled state-backend architecture in Flink 2.0 that offloads state storage to remote distributed storage instead of keeping it bound to local task manager disks.

  • Expected to significantly speed up state recovery during scale up/down actions, eliminating the local disk state-restore bottleneck that currently slows down Flink rescales.