KOEN
Read Original
#Backend

From Silos to Service Topology: Why Netflix Built a Real-Time Service Map

From Silos to Service Topology: Why Netflix Built a Real-Time Service Map
01

Summary

Inside Netflix's Living Map: Taming 1,000+ Microservices in Real-Time

How Netflix uses eBPF and Graph Databases to build a unified observability backbone for faster troubleshooting.

This article explores Netflix's transition from siloed observability signals to a unified service topology. By merging data from network, application, and request layers, they created a powerful navigation tool that allows engineers to understand the backbone of their distributed architecture with sub-second precision.

  • 01Three-layer data strategy combining eBPF, IPC metrics, and tracing for a complete picture
  • 02Sophisticated resolution logic that collapses network hops into direct app-to-app dependencies
  • 03Distributed pipeline using Apache Pekko Streams for fault-tolerant, high-volume processing
  • 04High-performance graph database abstraction for sub-second traversal at Netflix scale
  • 05'Time Travel' feature enabling engineers to reconstruct the topology from any point in the past

RECOMMENDATION

Highly recommended for SREs and platform engineers working with complex MSA who face challenges in identifying dependency-related failures. The approach to non-intrusive monitoring via eBPF and efficient graph modeling provides a great blueprint for modern observability stacks.

The Problem

In Netflix's massive microservices environment, engineers struggled to identify service dependencies, blast radius, and root causes during incidents using only fragmented metrics and logs. Static architecture diagrams were often outdated, requiring manual synthesis of information from multiple tools at critical times like 3 AM outages.

The Solution

Netflix built a real-time 'Service Topology' system that unifies three data sources: eBPF-based network flows, IPC metrics, and distributed tracing. They implemented a three-stage distributed processing pipeline using Apache Pekko Streams to resolve network intermediaries (like load balancers) and persisted the enriched relationships in a high-throughput graph database.

The Result

The system achieves sub-second response times for multi-hop dependency queries even at scale and features a 'Time Travel' capability to analyze historical topology states. This has streamlined incident response by providing instant visibility into upstream/downstream impacts and enabled automated architectural validation.

Trade-off

The request layer relies on sampling to minimize performance overhead, potentially missing rare execution paths, and IPC metrics are limited to services with specific instrumentation. Additionally, storage efficiency is prioritized via time-window aggregation, which may sacrifice fine-grained granularity for historical data snapshots.

03

Key Concepts

Concept · 01

eBPF (Extended Berkeley Packet Filter)

A technology that allows running sandboxed programs in the OS kernel without changing kernel source code or loading modules.

  • Used to capture ground-truth network flow records at the kernel level regardless of application instrumentation.
Concept · 02

Service Topology

A structural map representing the relationships and dependencies between various microservices within a distributed system.

  • Implemented as a dynamic, real-time updated graph rather than a static design document to reflect the actual state of the infrastructure.
Concept · 03

Inter-Process Communication (IPC) Metrics

Detailed metrics emitted by services when communicating via protocols like gRPC, GraphQL, or REST.

  • Provides rich application context, such as specific API endpoints and latency distributions, which raw network flows lack.