KOEN
Read Original
#Backend

Running Apache Spark experiments in my sleep (and on a plane)

Running Apache Spark experiments in my sleep (and on a plane)
01

Summary

Running Spark Experiments in My Sleep: How Netflix Conquered Scale-Only OOMs

A masterclass in parallelizing multi-hour Spark runs and solving subtle JVM memory bottlenecks at Netflix scale.

This article details how a Netflix engineer overcame critical executor OOM errors in a massive sessionization pipeline. By shifting from single-run optimization to branch-isolated concurrent experiments automated by AI agents, the author dramatically shortened iteration cycles and uncovered deep Spark memory pitfalls involving Option boxing and AQE broadcast joins.

  • 01Shifted paradigm from single-run speedups to running 5 concurrent experiments via branch-driven environment isolation
  • 02Leveraged an AI agent to handle deployment, metric collection, variant comparison, and experiment logging
  • 03Reduced peak executor memory from 112% to 58% by unboxing Scala Option wrappers on non-nullable fields
  • 04Identified silent OOM issues caused by accumulated AQE broadcast hash joins and lowered runtime threshold to 10 MB
  • 05Fixed Window operator memory buffering issues by shifting event truncation ahead of the buffering stage

RECOMMENDATION

Highly recommended for Data Engineers, Backend Engineers, and System Architects managing large-scale Apache Spark pipelines and complex JVM memory tuning.

The Problem

During an upgrade of Netflix's critical member-sessionizer pipeline, production-scale failures occurred, including executor OOMs, shuffle spillage, and stage retries, driven by severe data skew. Standard 1% sampling failed to reproduce these long-tail issues, requiring every experiment to run against full production datasets.

The Solution

The author established a parallel testing workflow using branch-driven isolation in Dataflow and Maestro, orchestrated via tmux and automated AI agent skills. The underlying Spark memory bugs were resolved by unboxing Scala Option wrappers, slimming event payloads, lowering AQE runtime broadcast thresholds, and truncating events prior to Window operations.

The Result

Unboxing fields and slimming payloads reduced peak executor memory from 112% to 58% of the slot with zero disk spill. The branch-driven parallel experiment setup allowed 4-5 complete full-scale experiment cycles within a single day during a flight.

Trade-off

While performance tuning successfully achieved pipeline stability and reclaimed executor memory headroom, it did not alter the fundamental aggregation architecture; unbounded growth in upstream data events will eventually re-introduce memory pressure.

03

Key Concepts

Concept · 01

Dataflow & Maestro (Branch Isolation)

Netflix's CLI and workflow orchestrator tools that enforce sanitized branch-driven development and resource isolation for data pipelines.

  • Tagged workflows, output tables, and run namespaces with branch names to allow zero-collision concurrent testing
  • Automatically provisioned and updated table schemas without manual DDL statements
Concept · 02

Option Boxing & Unboxing in Spark

The JVM memory overhead associated with wrapping dataset fields in Scala Option wrappers, which inflates object allocations and GC pressure.

  • Identified heavy JVM deserialization overhead in collect_list memory bags due to Option wrappers
  • Unboxed non-nullable fields to plain primitives, cutting peak executor memory usage from 112% to 58%
Concept · 03

AQE (Adaptive Query Execution) Broadcast Hash Join

A dynamic Spark feature that re-optimizes query plans at runtime, such as converting sort-merge joins into broadcast hash joins based on actual data sizes.

  • Discovered AQE pinning ~3.1 GB of cumulative broadcasted data in a single executor's memory across multiple joins
  • Lowered spark.sql.adaptive.autoBroadcastJoinThreshold to 10 MB to prevent runtime memory pressure