KOEN
Read Original
#Backend

Building a fault-tolerant metrics storage system at Airbnb

Building a fault-tolerant metrics storage system at Airbnb
01

Summary

50 Million Samples/Sec: How Airbnb Built a Fault-Tolerant Metrics Storage System

Navigating Multi-tenancy and Cluster Federation for 1.3 Billion Active Time Series

This article explores Airbnb's journey in building a massive observability platform. It details the architectural shifts from a single cluster to a federated multi-cluster model, focusing on isolation techniques and automated operations to handle petabyte-scale data.

  • 01Isolating tenant workloads using shuffle sharding to prevent cascading failures
  • 02Implementing cross-cluster querying and alerting through enhanced Promxy
  • 03A progressive rollout strategy ensuring over 99.9% system reliability
  • 04Automating stateful deployments to eliminate manual configuration drift
  • 05Sharding compaction workloads to maintain query performance for large tenants

RECOMMENDATION

Essential reading for infrastructure engineers and SREs dealing with hyper-scale metrics systems. It provides a blueprint for building resilient, multi-tenant architectures that can survive regional outages and traffic spikes.

The Problem

Airbnb faced performance and reliability challenges when migrating from a hosted provider to an in-house metrics solution processing 50 million samples per second and 1.3 billion active time series. Key issues included noisy neighbor effects, slow query performance for large payloads, and high blast radius from single-cluster outages.

The Solution

The team implemented shuffle sharding for tenant isolation and introduced strict read/write guardrails. They transitioned to a multi-cluster federation architecture using Promxy and automated stateful application deployments via Kubernetes operators to ensure consistency and minimize manual overhead.

The Result

The architecture achieved over 99.9% availability and improved system resilience through tenant-level controls and multi-zone deployments. Automation reduced configuration drift across clusters and transformed clusters into 'cattle' that can be easily scaled or replaced.

Trade-off

Federated queries across multiple clusters were found to be 5–10x more resource-intensive than single-cluster queries. Additionally, managing a multi-cluster environment increased the initial operational complexity and tooling requirements.

03

Key Concepts

Concept · 01

Shuffle Sharding

A technique that isolates different tenants' workloads by assigning them to a subset of nodes, minimizing the impact of a single tenant's spike on the entire system.

  • Used to give each tenant a 'single-tenant experience' within a shared, high-throughput cluster.
Concept · 02

Federated Query

A querying mechanism that retrieves and aggregates data from multiple independent data sources or clusters as if they were one.

  • Leveraged Promxy to provide a unified interface for querying metrics across diverse application and infrastructure clusters.
Concept · 03

Blast Radius Control

A design philosophy aimed at limiting the impact of a failure within a system to a small, contained area.

  • Achieved by distributing workloads across multiple failure domains and clusters to prevent total system outages.