KOEN
Read Original
#AI

Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use

Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use
01

Summary

How Pinterest Manages Trillions of Events: Redesigning Next-Gen ML Data Infrastructure

Slashing costs and doubling speed by unifying fragmented data pipelines at Pinterest

This article explores Pinterest's comprehensive redesign of its user-sequence platform to optimize recommendation performance. It details the practical application of 'Configuration-as-Code' and Lambda Architecture to balance data freshness, consistency, and efficiency. It provides a blueprint for engineers building large-scale data engineering and ML infrastructure.

  • 01Eliminated training-serving drift using a unified execution engine for all runtimes
  • 02Enabled infrastructure-less signal deployment through Python-based configurations
  • 03Optimized network bandwidth and I/O by migrating from heavy blobs to columnar storage
  • 04Balanced immediate event freshness with historical correctness via Lambda Architecture
  • 05Executed a risk-free migration strategy using sophisticated shadow comparison and validation

RECOMMENDATION

Highly recommended for engineers working on ranking and recommendation systems where real-time performance and data consistency are paramount. For data platform teams struggling with pipeline fragmentation, 'Configuration-as-Code' is a game-changer.

The Problem

User-sequence data is critical for ranking and recommendation systems but is notoriously expensive and fragile to maintain at scale. Challenges included maintaining consistency between real-time serving and offline training, as well as the low productivity caused by needing bespoke pipelines for every new signal onboarding.

The Solution

The platform was redesigned based on a 'One Definition, Many Runtimes' principle using Python-based Configuration-as-Code. Key implementations include a shared execution engine with pluggable executors, a lambda architecture for real-time and batch synchronization, and a columnar, time-partitioned storage layout for I/O efficiency.

The Result

The redesign led to significant reductions in infrastructure costs and dramatically improved developer productivity by speeding up signal onboarding. These technical improvements resulted in higher user engagement metrics on major surfaces while ensuring both data freshness and consistency.

Trade-off

Due to the nature of online systems, achieving a 100% data match with legacy systems was impossible, necessitating an acceptance of approximate consistency. Additionally, parallel operations of shadow pipelines during migration required temporary extra computing resources and operational overhead.

03

Key Concepts

Concept · 01

User Sequence

An ordered list of a user's recent actions combined with enriched signals like embeddings and metadata.

  • Captures temporal context vital for advanced sequence-aware models like Transformers
  • Serves as a reusable data primitive across ranking, search, and anomaly detection
Concept · 02

Configuration-as-Code

The practice of defining data filtering and enrichment logic as source code to be executed consistently across different runtimes.

  • Configs defined in Python are compiled to JSON and shared across streaming and batch jobs
  • Reduces signal-to-production time by minimizing bespoke pipeline engineering
Concept · 03

Lambda Architecture

A data-processing architecture that combines a real-time speed layer with a batch processing layer.

  • Streaming path ensures sub-second freshness while batch path ensures historical completeness
  • Allows the system to reconcile late-arriving events and maintain consistent training datasets