KOEN
Read Original
#AI

CASTLE: training natural language search before the first real query

CASTLE: training natural language search before the first real query
01

Summary

How Airbnb Solved the Search Cold-Start Problem with Synthetic Data Generation

Inside CASTLE: A novel framework leveraging user booking signals and contrastive learning to train natural language search models

The Airbnb engineering team introduced CASTLE at CIKM 2026, a framework designed to train natural language search models before logging a single real query. By combining contrastive listing pairs from de-identified booking sessions with seed-guided LLM prompting, CASTLE bridges the gap between synthetic data generation and authentic user behavior. This article explores how to automatically generate highly accurate relevance labels and handle long-tail search intents without relying on manual annotations.

  • 011.01 KL Divergence Achieved: 9.2x closer to real user search distribution compared to previous state-of-the-art baselines
  • 02Automatic Relevance Labeling: Eliminates circular LLM evaluations by deriving positive/negative labels directly from booking behaviors
  • 03Realistic Query Distribution: Matches real user query length variation (σ = 2.48 vs. real user σ = 2.49)
  • 04Long-Tail Search Optimization: Enhances search relevance for rare feature combinations even after live user traffic accumulates

RECOMMENDATION

Essential reading for Search & AI engineers implementing zero-shot/cold-start search models, as well as backend engineers designing scalable LLM synthetic dataset pipelines.

The Problem

When introducing natural language search to Airbnb to replace traditional filter controls, the team faced a severe cold-start problem due to the lack of historical user queries and relevance labels. Existing synthetic query generation approaches like InPars and Promptagator showed a high distributional mismatch compared to actual user search behavior or suffered from error propagation when using separate LLMs to judge relevance post-hoc.

The Solution

The engineering team developed CASTLE, a framework that jointly generates synthetic queries and relevance labels using contrastive listing pairs (positive vs. negative) derived from de-identified booking sessions. By combining template-based, few-shot (using 500 user-research seed queries), and attribute-grounded prompts with a query-variety mechanism, CASTLE creates realistic queries while guaranteeing accurate relevance labels by design through contrastive pair construction.

The Result

CASTLE achieved a Kullback–Leibler (KL) divergence score of 1.01 against real user search distributions, representing a 9.2x improvement over the best baseline (Promptagator at 9.33), with human domain expert evaluation confirming 91–93% label agreement. Additionally, its query length standard deviation (σ = 2.48) closely matched real user patterns (σ = 2.49), while continuing to effectively oversample long-tail search intents even after production launch.

Trade-off

Newly listed properties without booking history cannot form contrastive listing pairs and require fallback to an LLM-based Virtual Judge (VJ) evaluator, while offline generation incurs compute overhead for prompting LLMs and managing contextual prompt pipelines.

03

Key Concepts

Concept · 01

Cold-Start Problem in Search

A scenario where a new search service lacks historical query logs and query-to-item relevance labels, making supervised model training difficult.

  • Occurred when Airbnb introduced natural language search before gathering organic production traffic
Concept · 02

Contrastive Listing Pairs

A structured data pair formed by comparing an item selected by a user (Positive) against viewed-but-not-selected alternatives (Negative) during a booking session.

  • Fed into the LLM as structured input to generate queries satisfied by the positive listing but not the negative one
  • Guarantees accurate relevance labels by design without requiring post-hoc human or LLM annotation
Concept · 03

KL Divergence (Kullback–Leibler Divergence)

A statistical measure indicating how much one probability distribution differs from a baseline reference distribution, where values closer to zero indicate higher similarity.

  • Used to evaluate how closely synthetic query length and attribute distributions match real guest queries
  • CASTLE reduced KL divergence from 9.33 (best baseline) down to 1.01, demonstrating superior distribution alignment