KOEN
Read Original
#DevOps

Finding zombies in our systems: A real-world story of CPU bottlenecks

Finding zombies in our systems: A real-world story of CPU bottlenecks
01

Summary

Are There Zombies in Your System? Pinterest's 3-Month CPU Bottleneck Hunt

How a default AWS AMI setting created thousands of zombie cgroups that paralyzed an ML cluster

This article details the journey of Pinterest engineers as they dive into kernel-level profiling to solve mysterious network failures in their Ray ML cluster. It reveals how what seemed like a simple networking error was actually a complex CPU starvation issue caused by tens of thousands of zombie memory cgroups, offering deep insights for infrastructure engineers.

  • 01Establishing the correlation between AWS ENA driver resets and CPU starvation
  • 02Techniques for monitoring 96-vCPU systems per core using mpstat and temporal perf data
  • 03Using Flamescope for 'time-travel' profiling to capture the exact moment of intermittent failure
  • 04Analyzing how kernel-level memcg iteration overhead impacts overall system stability
  • 05A cautionary tale of unexpected side effects from default services (ecs-agent) in the AWS Deep Learning AMI

RECOMMENDATION

Never trust default cloud vendor AMI settings blindly. If you encounter inexplicable performance degradation, don't rely on average CPU metrics; instead, perform per-core profiling and check for kernel system call latencies.

The Problem

Ray-based ML training jobs at Pinterest experienced intermittent crashes due to network connectivity loss and ENA driver resets, particularly on GPU instances in a specific AWS availability zone. The issue was correlated with high system CPU usage and page faulting, but standard memory optimizations failed to resolve the problem.

The Solution

Using mpstat for per-core monitoring and Flamescope with time-traveling perf profiles, the team identified that the Kubelet was pinning a single CPU core at 100% while executing the mem_cgroup_nr_lru_pages system call. The root cause was discovered to be tens of thousands of 'zombie memory cgroups' created by a crashing ecs-agent, a default systemd unit in the AWS Deep Learning AMI, which was subsequently disabled.

The Result

By eliminating the CPU starvation caused by zombie memcgs, the team restored the stability of ML training jobs and stopped the loss of expensive GPU compute hours. The investigation also highlighted the critical need to audit default configurations in cloud vendor-provided machine images.

Trade-off

The investigation spanned over three months, involving significant resource expenditure on ineffective initial hypotheses like Huge pages and alternative memory allocators. Resolving the issue required manual intervention to disable default system services, adding a minor layer of configuration management overhead.

03

Key Concepts

Concept · 01

ENA Driver

AWS's next-generation network interface driver that triggers a hardware reset if the CPU fails to respond within a specific threshold (e.g., 5 seconds).

  • Resets are triggered during CPU starvation when Tx threads cannot get CPU time
  • Causes momentary network drops that crash sensitive distributed computing jobs
Concept · 02

Zombie Memory Cgroup

Memory control groups that remain in the kernel after a container or process exits because some memory references are still active.

  • Accumulated in tens of thousands due to repeatedly crashing containers
  • Causes severe CPU overhead when management processes like Kubelet iterate through the cgroup list
Concept · 03

Flamescope

A performance visualization tool developed by Netflix that breaks down profile data into sub-second intervals to track intermittent latency spikes.

  • Visualizes 2-minute perf snapshots into 2D heatmaps for granular stack analysis
  • Crucial for identifying the spike in Kubelet CPU usage just before ENA resets