Frankenstein Datasets and the Crisis in Climate Science Integrity

Science, at its best, is a self-correcting process—an ongoing pursuit of truth through rigorous evidence, transparent methodology, and open debate. But what happens when errors are introduced, perpetuated, and defended? The tale of the so-called “Frankenstein dataset” in hurricane damage research, brought to light by Roger Pielke Jr., illustrates how flawed data practices can distort both public perception and policy.

The Frankenstein dataset, born from undocumented modifications to a rigorous, peer-reviewed dataset, highlights a crisis in scientific integrity. Its story is not just a cautionary tale about bad data but a case study in how the scientific process can fail when institutional accountability is lacking. Let’s unpack how this dataset came to be, why it matters, and what it reveals about the state of climate science.

The Origins of the Original Dataset

The original dataset, as developed through decades of research by Pielke and his colleagues, aimed to normalize hurricane damages by adjusting for inflation, population growth, and other economic factors. This normalization process allowed researchers to compare historical hurricane damages on an apples-to-apples basis, isolating trends in economic loss from changes in wealth or development.

The dataset, thoroughly documented in studies like Weinkle et al. (2018) and Pielke et al. (2008), served as a reliable tool for understanding hurricane impacts. It was grounded in NOAA’s “best track” data, covering U.S. landfalling hurricanes, and adhered to consistent methodologies​​.

However, the story took a dark turn when this dataset fell into the hands of ICAT, an insurance company.

How the Frankenstein Dataset Was Born

After Pielke et al. (2008) was published, Pielke’s team partnered with ICAT to create the ICAT Damage Estimator, an online tool designed to visualize hurricane damages using the peer-reviewed dataset. Initially, the collaboration worked as intended: the tool increased access to high-quality research for industry stakeholders​.

But in 2010, ICAT was acquired by another company, and Pielke ceased his involvement. Over the following years, ICAT employees, who lacked expertise in disaster normalization, made undocumented changes to the dataset. These alterations included replacing post-1980 entries with data from NOAA’s Billion-Dollar Disasters (BDD) database, which utilized a completely different methodology​​.

Key Modifications

1. Substitution with NOAA’s BDD Data: ICAT replaced post-1980 entries with BDD data, which included inland flooding damages (from the National Flood Insurance Program, or NFIP) and broader economic impacts like commodity losses and disaster relief payouts. These additional factors inflated damage estimates post-1980, creating an artificial upward trend​​.
2. Additional Events: ICAT17, the modified dataset, introduced 61 additional storm damage events, none of which were sourced or documented. Most of these undocumented events occurred after 1980, further skewing the dataset​.
3. Methodological Discontinuity: NOAA’s BDD methodology, adopted in 2016, was incompatible with the original dataset. For example, NFIP payouts didn’t exist before 1968, making comparisons between pre- and post-1968 damages inherently flawed​​.
4. Unsupervised Alterations: Beyond substituting BDD data, ICAT17 contained additional undocumented changes to the original dataset. These changes introduced upward biases even before normalization adjustments were applied​.

Steve McIntyre commented on Pielke Jr.’s post.

By the time ICAT published this Frankenstein dataset online, it had diverged so far from the original peer-reviewed data that it bore no resemblance to a rigorous research product.

How the Frankenstein Dataset Was Misused

The ICAT17 dataset, later extended and rebranded as “XCAT/ICAT 23 in Willoughby et al 2024,” was adopted by researchers who assumed it was a professionally maintained and credible resource. Notably:

• Grinsted et al. (2019) and Willoughby et al. (2024) used XCAT to claim an upward trend in normalized U.S. hurricane damages, attributing this trend to climate change.
• These studies were published in prominent journals like PNAS and JAMC and subsequently cited in influential reports, including the IPCC’s AR6​​.

However, Pielke’s analysis reveals that these trends vanish when the original dataset (Weinkle et al. 2018) is used instead of XCAT/ICAT23. In other words, the upward trends claimed in these studies are entirely a product of flawed data practices​.

The Impact on Climate Science and Policy

The consequences of these errors are far-reaching:

1. Distorted Public Perception: The flawed studies, amplified by major journals and the IPCC, reinforce the narrative that climate change is driving increased hurricane damages. While politically expedient, this narrative is unsupported by NOAA’s direct measurements, which show no long-term trends in U.S. landfalling hurricanes or their intensity​​.
2. Undermined Scientific Integrity: The willingness of peer-reviewed journals to publish studies based on undocumented, methodologically inconsistent data, AND REFUSAL TO RETRACT WHEN SUCH FLAWS ARE CLEARLY IDENTIFIED, points to a breakdown in the scientific process. This failure undermines public trust​.
3. Misguided Policy Decisions: Policies based on flawed data risk diverting resources from effective disaster mitigation strategies. By overstating the role of climate change in hurricane damages, these studies obscure the real drivers of vulnerability, such as poor land-use planning and inadequate building codes.

A Call to Correct Course

As Pielke notes, “mistakes happen in science.” What matters is how the scientific community responds when those mistakes are identified. The Frankenstein dataset saga offers an opportunity for course correction:

• Journals like PNAS and JAMC should retract the flawed studies to prevent further misuse of the ICAT17/XCAT datasets.
• The climate science community must adopt stricter standards for data transparency and provenance to avoid similar errors in the future.
• Policymakers should demand higher-quality evidence before enacting costly climate policies based on unverified claims.

This case is not just about bad data—it’s about the integrity of the scientific process. If climate science is to have any credibility and fulfill its proclaimed role in informing policy, it must hold itself to the highest standards of rigor, transparency, and accountability.

Final Thoughts

The Frankenstein dataset is a stark reminder of the dangers of uncritical acceptance in science. While the temptation to fit data to a convenient narrative is strong, true scientific progress requires resisting that impulse. As Pielke’s critique demonstrates, only by confronting and correcting errors can science fulfill its promise as a self-correcting endeavor. Let this be a wake-up call for climate science: integrity must come before ideology.

Sources:

Do Not Use the ICAT Hurricane Loss “Dataset”: An Opportunity for Course Correction in Climate Science

https://twitter.com/RogerPielkeJr/status/1870496128304578675

https://twitter.com/RogerPielkeJr/status/1870873521808871774