Subscribe to receive future updates, new wildfire research, and analysis directly by email.
Support independent research — your contribution helps fund wildfire hazard analysis and transparency work.
Independent Model Validation of Oregon’s Wildfire Hazard Map
A data-driven evaluation that revealed key flaws in the state’s wildfire risk model — findings that helped influence public policy and legislative change.
Overview
After receiving a notice that his property was classified as high wildfire risk, Jeremy Kauwe launched an independent investigation into the state’s wildfire hazard map. With no prior experience in geospatial analysis, he taught himself GIS tools, cleaned and restructured raw model inputs, and aligned them with historical fire data to evaluate the model’s accuracy. His findings revealed serious statistical flaws and systemic bias. Most notably, he translated these technical insights into clear, actionable messaging for lawmakers and the public—leading to widespread support and the eventual repeal of the map’s use statewide.
Approach
To evaluate the model's performance, I focused exclusively on the burn probability output produced by Oregon State University and compared it to historical wildfire records—the same in-sample data the model was trained on. The geospatial data provided was fragmented across multiple shapefile formats, so I had to learn GIS workflows and manually align the layers. I created a uniform grid of spatial points to evenly cover the state and then intersected each point with both the model’s burn probabilities and historical fire events. This process required significant computing time and frequent troubleshooting, as my system often crashed while processing large spatial overlays. Once aligned, I converted the data into a CSV where each row represented a geographic location with its corresponding burn probability and whether or not a fire had historically occurred. Due to severe class imbalance in the fire data, I used precision-recall AUC rather than ROC AUC to measure performance. The final analysis showed that the model lacked classification power, and I published the results and methodology in a public-facing website to promote transparency.
Results
The analysis revealed that the model had no classification power—its precision-recall AUC was equivalent to random chance, even on the in-sample historical fire data it was trained on. I also uncovered a structural bias in how “high-risk” thresholds were set: instead of being derived from a representative sample, they were calculated using only properties located within the Wildland-Urban Interface (WUI), a subset that already skewed high in hazard values. The 90th percentile threshold established from this group corresponded to just the 42nd percentile when applied to the full dataset, resulting in over 58% of properties statewide being overclassified—contrary to the legislative intent of identifying only the top 10%. These findings were shared through a public website, direct outreach, and presentations at town halls. Notably, I presented this evidence to Senator Jeff Golden, who soon after advocated for the repeal of the wildfire hazard map.
Challenges
One of the biggest challenges was working with geospatial data for the first time. I had no prior experience with GIS tools or spatial analysis, so I had to self-teach everything from scratch using open-source software like QGIS. Cleaning, aligning, and transforming spatial datasets—often in incompatible formats—was time-intensive and pushed the limits of my local machine, which crashed frequently during processing. Equally challenging was translating these highly technical findings into something a non-technical audience could understand. To overcome this, I relied on visuals, simplified metaphors, and even physical paper cutouts to explain key statistical concepts—such as showing how the model’s predicted fire and no-fire distributions weren’t separable. These efforts proved essential in gaining public trust and helping decision-makers grasp the stakes.
Project Links
Tags
- Statistical Consulting
- Machine Learning
- Model Validation
- Precision-Recall AUC
- QGIS
- Geospatial Analysis
- Wildfire Risk Modeling
- Wildfire Map Audit
- Oregon Wildfire Map
- Policy Impact