KOEN
Read Original
#AI

Making Failure Diagnosis Reliable with Pensive AI

Making Failure Diagnosis Reliable with Pensive AI
01

Summary

How Netflix Reduced Manual Debugging Across 100k Daily Workload Failures with Pensive AI

Combining deterministic rules with LLM reasoning to achieve 100% failure extraction coverage and automated CI/CD eval gates.

Netflix platform engineers introduced Pensive AI to resolve complex workload failures across Spark, Trino, Druid, and Titus. By using bounded line-number extraction and on-demand LLM debugging, Netflix scaled diagnosis while strictly controlling costs and latency. Integrated continuous evals in CI/CD pipelines ensure model precision and prevent system regressions.

  • 01Optimized latency and eliminated text hallucinations by requesting line-number ranges rather than generating raw log text
  • 02Implemented step-attempt cached debugging with SSE/WebSocket streaming to minimize LLM compute overhead and improve UX
  • 03Built an automated evaluation pipeline using archived S3 log snapshots to gate PR merges based on precision and inclusion metrics
  • 04Observed 321% more manual debug-log exploration actions on failed attempts where Pensive AI was not utilized

RECOMMENDATION

Highly recommended for platform engineers and MLOps architects building production-grade AI observability. It offers a blueprint for combining deterministic rules with LLM reasoning while managing costs and latency.

The Problem

Netflix runs millions of daily workloads where roughly 100,000 fail on peak days, and traditional regex-based rules failed to classify nearly 50% of these failure modes.

The Solution

Netflix built Pensive AI, combining deterministic legacy rule engines with AI error extraction using line-number range outputs and step-attempt cached on-demand LLM failure diagnosis.

The Result

Pensive AI expanded error extraction coverage to 100% and resulted in 321% more manual debug-log exploration actions for failed attempts where Pensive AI was not used.

Trade-off

Full LLM debugging introduces higher latency (tens of seconds to a minute) and compute costs, requiring on-demand user triggers and step-attempt caching instead of automatic execution.

03

Key Concepts

Concept · 01

Line-Number Range Extraction

A prompt design technique where the LLM identifies error spans by returning only start and end line numbers rather than generating entire log strings. This significantly reduces output token latency, cost, and hallucination risks.

  • The model returns two integers regardless of error log length
  • Original log content is directly reconstructed from raw log stores using returned line numbers
Concept · 02

Step-Attempt Workload Caching

A caching mechanism centered on immutable workload execution units (step attempts) to prevent redundant LLM inference calls across multiple users and entry points.

  • Uses conditional writes to lock work in progress during concurrent debug requests
  • Serves cached diagnosis across UI, programmatic APIs, and MCP protocols with a 90-day retention policy
Concept · 03

CI/CD Evaluation Gates

An automated continuous evaluation framework integrated into the deployment pipeline to measure extraction precision and recall before code merges.

  • Evaluates model and prompt changes against archived S3 log snapshots to ensure reproducible tests
  • Gates pull request merges based on strict precision and inclusion thresholds in isolated preview environments