KOEN
Read Original
#AI

How we knew COVID was over (and what our models had to unlearn)

How we knew COVID was over (and what our models had to unlearn)
01

Summary

Why Airbnb's ML Models Had to Learn to Forget When COVID Ended

How to govern high-stakes forecasting models using three strategic actions: Refit, Respecify, or Hold.

Maintaining forecasting model reliability in a highly volatile market is a major challenge for any machine learning team. This article shares how Airbnb managed severe model drift during the pandemic recovery by introducing a pragmatic decision framework that determines whether to update parameters, overhaul the model structure, or purposely do nothing.

  • 01Deconstructs the ambiguous term 'retrain' into three precise actions: refit, respecify, and hold.
  • 02Analyzes critical forecasting failures like 'chasing noise' from one-off spikes and 'carrying ghosts' of stale pandemic-era assumptions.
  • 03Shares a real-world case study where restructuring the geographic borrowing of models halved forecast errors during recovery.
  • 04Emphasizes the vital discipline of 'purposeful forgetting' to prevent temporary external shocks from permanently biasing predictions.

RECOMMENDATION

If your production forecasting models suffer from recurring biases despite routine updates, stop blindly retraining and use Airbnb's triage framework to evaluate if you need a structural respecification.

The Problem

Airbnb's demand and booking forecasting models suffered from persistent bias and errors during the post-pandemic recovery because anomalous historical data and outdated assumptions poisoned the predictions. Simply relying on routine, automated model retraining often led the models to overreact to temporary noise or continuously carry over obsolete crisis-era behaviors.

The Solution

Airbnb established a formal model governance framework that breaks down retraining into three distinct actions: Refitting, Respecifying, and Holding. To fix post-pandemic forecast drift, they respecified the model's structural assumptions by allowing thinner markets to dynamically borrow strength across geographic adjacency and shared recovery dynamics rather than a static global hierarchy.

The Result

Applying this structured framework prevented unstable model updates during periods of high volatility, and respecifying the forecasting model reduced error rates across the recovery period by over 50% compared to a refit-only approach, bringing accuracy back to pre-pandemic baselines.

Trade-off

Respecifying a model introduces high complexity and risks replacing a long-trusted, well-understood production system with an unproven design, while choosing to hold (do nothing) carries high organizational friction because it can be perceived as inaction by stakeholders.

03

Key Concepts

Concept · 01

Refitting

An update process that trains the existing model architecture and feature set with more recent data, updating only the parameters and weights.

  • Used as the default, low-cost option to handle routine data drift without altering the verified model logic.
Concept · 02

Respecifying

Modifying the core model structure, such as changing priors, adding or removing features, or adjusting how variables relate to each other, to align with structural shifts.

  • Implemented to restructure how markets share trends when the pre-pandemic hierarchical relationship permanently broke down.
Concept · 03

Holding

A deliberate decision to keep a model unchanged despite a temporary forecast miss, avoiding rash updates triggered by unexplained noise.

  • Applied during sudden, localized booking spikes to prevent the model from adapting to anomalous peaks and degrading future forecast accuracy.