KOEN
Read Original
#Backend

Data Projects: Managing Data Assets at Netflix Scale

Data Projects: Managing Data Assets at Netflix Scale
01

Summary

Managing Data Assets at Netflix Scale: Why They Switched from People to Projects

A deep dive into Netflix's 'Data Projects' for durable identity and automated data governance.

Netflix's Data Platform has evolved to handle the complexity of millions of assets by hoisting management granularity from individual tables to logical 'Projects.' This transition addresses the 'permissions whack-a-mole' caused by human-tied identities, providing a stable and auditable framework for large-scale data engineering.

  • 01Introduction of durable, synthetic identities that survive organizational changes and personnel departures.
  • 02The concept of 'Gravity' ensuring automatic governance of assets produced by project workflows.
  • 03Integration with Maestro orchestrator as a Trusted Workload Manager for secure identity minting.
  • 04Cryptographic exchange between Netflix application identities and AWS IAM roles for hybrid cloud support.
  • 05Proven success in critical pipelines like Streaming Quality of Experience (QoE) and Member Analytics.

RECOMMENDATION

Highly recommended for Platform Engineers and Data Architects managing large-scale infrastructure where organizational churn impacts system reliability.

The Problem

At Netflix scale, managing millions of tables and workflows using individual asset ACLs and human-tied identities led to broken pipelines and security gaps during organizational restructures and personnel changes.

The Solution

Netflix introduced 'Data Projects,' a container that groups related assets under a logical umbrella with a synthetic, durable identity that executes workloads independently of any human lifecycle.

The Result

The model successfully manages tens of thousands of production assets, ensuring pipeline stability through 'Gravity'—a feature that automatically associates new outputs with the project's identity.

Trade-off

Migrating legacy workflows requires careful mapping of existing permissions to the new project model, and the abstraction shifts focus from fine-grained individual control to aggregate management.

03

Key Concepts

Concept · 01

Synthetic Identity

A durable, non-human application identity that asynchronous workloads execute under, independent of employee lifecycles.

  • Prevents workflow failures when authors change teams or leave the company.
  • Serves as an 'identity umbrella' for both internal and cloud resources.
Concept · 02

Gravity

A platform property where new assets created by a project's workload are automatically added to that project.

  • Eliminates manual effort for cataloging and securing new data outputs.
  • Ensures consistent access control for downstream data products.
Concept · 03

Trusted Workload Manager (TWM)

An authorized system, like Maestro, capable of minting fresh identity tokens on behalf of scheduled workloads.

  • Validates caller access to the project before assuming its identity.
  • Secures multi-step executions across data warehouses and AWS environments.