KOEN
Read Original
#Backend

Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world

Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world
01

Summary

Scaling Beyond One: How Airbnb Rebuilt Its Data Engine for a Multi-Product Future

Mastering the balance between monolithic consistency and product-specific flexibility in data modeling

This article explores Airbnb's journey in evolving its data architecture to support its expansion from a single-product focus on Homes to a multi-product world including Experiences and Services. It details the strategic decisions made by data engineers to maintain a robust foundation while accommodating new business complexities. Readers will gain insights into building a scalable offline data warehouse that serves as a single source of truth for diverse business lines.

  • 01Introduction of the 'No Hybrid Data Models' principle for better long-term scalability
  • 02Implementation of strict identifier naming conventions to prevent join errors and confusion
  • 03Definition of the offline data warehouse as a crucial translation layer for raw production data
  • 04Successful use of monolithic models for cross-cutting concepts like Payments and Messaging
  • 05Adaptation to complex new concepts like flexible business hours and radius-based location models

RECOMMENDATION

Highly recommended for data engineers and architects at companies facing business diversification. Airbnb's framework offers a practical guide on balancing central consistency with domain-specific autonomy.

The Problem

Airbnb's legacy data infrastructure was optimized for its decade-old 'Homes' business, making it difficult to scale and integrate new pillars like 'Experiences' and 'Services'. A fragmented strategy risked creating data silos, inconsistent analytics, and technical debt that could slow down innovation.

The Solution

The team built a flexible framework combining centralized principles with decentralized modeling guidelines. They established three foundational principles: no hybrid models, consistent identifier naming, and clear namespace organization, while empowering teams to choose between separate or monolithic models based on domain-specific attributes.

The Result

Successfully integrated two new product pillars without compromising the stability of the core Homes business. The new architecture provided a scalable foundation that improved clarity and the speed of delivering accurate insights.

Trade-off

The transition required a massive undertaking to migrate and deprecate legacy assets with hundreds of downstream consumers. It necessitated careful communication, dual pipeline runs for validation, and a slow deprecation cycle to ensure business continuity.

03

Key Concepts

Concept · 01

Offline Data Warehouse

An analytics-oriented data infrastructure separate from online systems, optimized for complex queries and reporting.

  • Acts as a crucial translation layer between raw production data and actionable insights
  • Owned by data and analytics engineers to ensure a standardized source of truth
Concept · 02

Monolithic vs Separate Modeling

The design choice between unified tables for reusability versus distinct tables for tailored product logic.

  • Separate models were used for user-facing features with unique attributes like Service offerings
  • Monolithic models were applied to product-agnostic concepts like Payments and Customer Support
Concept · 03

Namespace Organization

A method of categorizing data tables into logical spaces based on their scope and ownership to manage assets efficiently.

  • Used dedicated product namespaces for core tables and a global namespace for cross-cutting data
  • Allowed team-specific namespaces for managing intermediate tables and individual assets