If the Model Fails, You’ll See It Failing Here

Firewatch v3 weekly outlook: experimental, live issues not yet graded

An experimental weekly outlook (Firewatch v3) is shown on the map from the week of 2026-09-28. In a pre-registered back-test on 2019–2024 (not a blind test: an earlier version had been scored on that period) it beat a fire-history baseline. Scored after the fact on 2025–2026 (preliminary), its top-25% edge held (+7.4 points, 95% CI +3.2 to +11.6) but its top-10% edge was not confirmed (+0.8 points, CI −2.7 to +4.4). Live issues are archived with hashes (pre-registration sha256 5b1e6e32…e3ab). A grading rule for the live issues has not yet been pre-registered, so they are not scored here yet. The first issue was late: generated 2026-10-01 for a week that began 2026-09-28.

Alert trigger rules redefined (break in the counties_triggered series)

From 2026-10-01 the county alert triggers changed: fire alerts leave out prescribed and pile burns and fully contained fires; the weekly-outlook trigger is no longer used in emails; the air-quality trigger needs two EPA-corrected sensors at least 2 km apart with data no more than 3 hours old; east-wind events are not emailed on their own. The counties_triggered and ssi_composite values in the daily snapshot series in scorecard.json therefore have a break on 2026-10-01; rows from that date carry triggers_rule_version and ssi_inputs_version fields. The Season Severity Index air-quality component also switched to EPA-corrected values.

Site-wide factual corrections

The summer 2026 reservoir projection on /water was withdrawn (it projected 3.6% of capacity by September for California’s five largest reservoirs; measured storage on 2026-09-30 was 57.2%). The smoke-impact map layer was withdrawn pending verification. The Policy page and several blog posts were corrected (OSHA heat program, Colorado River status, insurance law year, fuels-treatment costs, PurpleAir sensor count, and the withdrawn 0.911 model score).

v0.7 published a leaked metric

The v0.7 AUC of 0.913 was computed on a test set that had been used during model tuning, which inflates the number. We disclosed it on the Methods model-history table (it carries an asterisk there), rebuilt the evaluation with a proper three-way train/validation/test split in v0.8, and reported the lower, honest number.

“91% accuracy” misphrasing in early posts

Early posts described the model's 0.911 AUC-ROC as “91% accuracy.” That is wrong: AUC-ROC is a ranking metric, not a hit rate. Given one cell that burned and one that didn't, the model ranks the burned cell higher 91% of the time, at a 0.1% base rate, most flagged cells will not burn. The phrasing has been corrected and Methods now explains the metric in plain language.

Nine-day data outage during active fire season

Daily updates stopped on June 2 and did not resume until June 11, with fire season underway, the site served stale active-fire and conditions data for nine days without a visible warning. Updates have resumed. The deeper failure was not flagging staleness to readers; surfacing data age prominently is now on the fix list.

Headline 0.911 AUC suspected to be inflated

An internal audit found that the v2.1 evaluation allowed spatial memorization (the model could learn which cells burn) and several forms of feature leakage. The v2.2 re-validation uses spatially blocked held-out testing and fixes the leaks; the honest number is expected to be lower. Until it lands, treat 0.911 as an upper bound, not a result. The corrected figure will be published here and on Methods, whatever it is. Update, October 2026 (draft): the 0.911 and 10.4x figures have been withdrawn from Methods, About, the blog and the map popups.