Accountability
If the Model Fails, You’ll See It Failing Here
Updated June 24, 2026 · active
This page is the platform’s accountability mechanism: a public, pre-registered validation of the Firewatch model during the 2026 fire season. The rules for scoring are written down below, before the outcomes are known. Predictions are timestamped when they are issued, and they get scored against what actually burned, whether that makes the model look good or not. A model that is only graded by the people who built it, after the fact, hasn’t been graded at all.
Once daily forecasts are live, each day the model issues corridor-wide rankings of every 4km cell. Each prediction is timestamped before outcomes are known. Monthly during the season, and again at season end, we score three things:
| Date | Prediction | Outcome | Score |
|---|---|---|---|
| 2026-06-24 | Firewatch v1.3 daily snapshot | ||
| 2026-06-23 | Firewatch v1.3 daily snapshot | ||
| 2026-06-22 | Firewatch v1.3 daily snapshot | ||
| 2026-06-18 | Firewatch v1.3 daily snapshot | ||
| 2026-06-17 | Firewatch v1.3 daily snapshot | ||
| 2026-06-16 | Firewatch v1.3 daily snapshot | ||
| 2026-06-15 | Firewatch v1.3 daily snapshot | ||
| 2026-06-14 | Firewatch v1.3 daily snapshot | ||
| 2026-06-13 | Firewatch v1.3 daily snapshot |
An accountability page that only starts counting once the model is fixed isn’t accountability. These are the mistakes we have already made, what they affected, and what we did about them. The list will grow.
v0.7 published a leaked metric
The v0.7 AUC of 0.913 was computed on a test set that had been used during model tuning, which inflates the number. We disclosed it on the Methods model-history table (it carries an asterisk there), rebuilt the evaluation with a proper three-way train/validation/test split in v0.8, and reported the lower, honest number.
“91% accuracy” misphrasing in early posts
Early posts described the model's 0.911 AUC-ROC as “91% accuracy.” That is wrong: AUC-ROC is a ranking metric, not a hit rate. Given one cell that burned and one that didn't, the model ranks the burned cell higher 91% of the time, at a 0.1% base rate, most flagged cells will not burn. The phrasing has been corrected and Methods now explains the metric in plain language.
Nine-day data outage during active fire season
Daily updates stopped on June 2 and did not resume until June 11, with fire season underway, the site served stale active-fire and conditions data for nine days without a visible warning. Updates have resumed. The deeper failure was not flagging staleness to readers; surfacing data age prominently is now on the fix list.
Headline 0.911 AUC suspected to be inflated
An internal audit found that the v2.1 evaluation allowed spatial memorization (the model could learn which cells burn) and several forms of feature leakage. The v2.2 re-validation uses spatially blocked held-out testing and fixes the leaks; the honest number is expected to be lower. Until it lands, treat 0.911 as an upper bound, not a result. The corrected figure will be published here and on Methods, whatever it is.
Scoring data lives in a public file (/scorecard.json) so anyone can check our arithmetic. Model details, known limitations, and the full version history are on Methods. Live fire weather and the seasonal risk surface are on the Forecast page. Found an error we haven’t listed? Tell us. it goes on the list.