KOEN
Read Original
#Backend

Building a high-volume metrics pipeline with OpenTelemetry and vmagent

Building a high-volume metrics pipeline with OpenTelemetry and vmagent
01

Summary

Conquering 100M+ Samples per Second: Inside Airbnb's Next-Gen Metrics Pipeline

Scaling Observability with OpenTelemetry and vmagent Aggregation Strategies

This article explores Airbnb's technical journey in modernizing their massive metrics infrastructure from StatsD to a modern Prometheus ecosystem. It provides deep insights into achieving performance efficiency through vmagent sharding and solving data accuracy issues with an innovative 'Zero Injection' approach.

  • 01Over 90% reduction in metrics-related CPU overhead by switching to OTLP
  • 02Achieved 100M TPS scale using a sharded, two-layer vmagent aggregation architecture
  • 03Solved Prometheus undercounting issues for sparse metrics using the 'Zero Injection' method
  • 04Strategic application of Delta vs. Cumulative Temporality for memory optimization
  • 05Strengthened open-source maturity by contributing native histogram support to VictoriaMetrics

RECOMMENDATION

Essential reading for SREs and Backend Engineers balancing observability costs with data accuracy in high-scale environments. The 'Zero Injection' technique is a practical fix for anyone using Prometheus with sparse or high-cardinality data.

The Problem

Airbnb needed to migrate from its legacy StatsD/Veneur metrics pipeline to a Prometheus-based storage system due to high CPU overhead and packet loss issues. The primary challenge was maintaining scale and accuracy, especially addressing the undercounting of sparse counters caused by how Prometheus handles resets.

The Solution

The team implemented a dual-write strategy using OpenTelemetry (OTLP) and built a two-layer streaming aggregation architecture (Router/Aggregator) with VictoriaMetrics' vmagent. They also developed a 'Zero Injection' technique at the aggregation tier to ensure accurate rate calculations in Prometheus for newly created counters.

The Result

The migration reduced metrics-related CPU usage from 10% to less than 1% and scaled to handle over 100 million samples per second. It significantly cut infrastructure costs while improving the reliability and fidelity of high-cardinality metrics.

Trade-off

Adopting Delta Temporality for high-volume services reduced memory pressure but introduced the risk of data gaps during failures. Additionally, the Zero Injection technique causes the initial increment to lag by one flush interval before appearing in dashboards.

03

Key Concepts

Concept · 01

vmagent

A lightweight metrics collector from VictoriaMetrics that supports streaming aggregation and horizontal scaling.

  • Used in a Router-Aggregator topology to reduce label cardinality and lower downstream storage costs at Airbnb.
Concept · 02

Zero Injection

A technique that injects a synthetic zero value at the start of a metric stream to help Prometheus correctly calculate rates of change.

  • Implemented in the aggregation tier to ensure sparse counters are initialized, preventing data loss during the first flush.
Concept · 03

Delta Temporality

A reporting mode that sends only the changes since the last export, significantly reducing in-process memory state.

  • Selected for extremely high-volume metric emitters to mitigate heap growth and garbage collection pressure.