Guest essay by Michael Hove [Please note, new update at the end]
In the Spring of 2016, I was updating a North Dakota water resource presentation, which contained a Palmer Hydrological Drought Index (PHDI) chart from April of 2013. I was replacing an older NOAA PHDI chart (May 2005 to October 2013) with an updated NOAA PHDI chart (May 2005 to March 2016). These charts are simple screen captures from the NOAA web site. Then, something caught my eye. I noticed that the summer of 2011 was wetter in the 2016 dataset than in the 2013 dataset, and the summer of 2013 was drier in the 2016 dataset than in the 2013 dataset. I scaled both charts on the Y-Axis to check my first visual impression. The results of this comparison are shown below in Figure 1. This comparison showed that there were several years where the newer 2016 dataset was different from the older 2013 dataset, both wetter and drier. Obviously, NOAA has modified the historic data.
Figure #1
This prompted me to look more closely at some other Palmer Drought Index data I have collected over the years. In 2006 I wrote a report using the Palmer Drought Severity Index (PDSI) and had downloaded the PDSI data from the NOAA website for the time frame of January 1977 to September, 2006. This 2006 dataset was for all nine NOAA climate divisions in North Dakota. Figure 2 shows the North Dakota climate division, which follow county boundaries.
Figure 2
Next, I downloaded (in May 2016) the PDSI dataset for the same time frame as my 2006 PDSI dataset (January 1977 to September, 2006), see Figure 3. I then plotted the two dataset downloads (2006 & 2016) as bar charts for each climate division and also plotted the differences between the two datasets for each climate division. These charts provide a nice visual comparison for how NOAA has modified the historical PDSI data.
Figure 3
To provide for a better visual comparison of the differences for all nine climate divisions, I created scatter plots of the “gray differences bar chart” and calculated the regression line for each of the charts. I then scaled the charts so that all nine of the climate divisions fit on a single page. From the scatter plots and regression you can see that seven of the climate divisions have had their historical data adjusted to produce a “drier” trend line, while two have their historical data adjusted to produce a “wetter” trend line. Also included are histograms of the “differences” data for each climate division.
I welcome any thoughts regarding the trends shown between the “Old Historic” and New Historic” data.
[UPDATE BY WILLIS]: Michael, thanks for your fascinating post pointing out the changes in the PDSI. Like you, I was unaware of them. After reading this post, I wrote to NOAA to ask about the change. Within hours I had the following reply:
On Sep 29, 2016, at 8:27 AM, Derek Arndt – NOAA Federal wrote:
When we switched to nClimGrid/nClimDv as our base US dataset in 2014, we recalculated many of our derivative products including the drought indices.
The methodology is documented in this 2014 article:
http://dx.doi.org/10.1175/JAMC-D-13-0248.1
This pending change and some early comparisons were first shared with the community in 2011.
Click to access GrDD-Transition.pdf
More info:
https://www.ncdc.noaa.gov/monitoring-references/dyk/nclimdiv-tmax-tmin
https://www.ncdc.noaa.gov/monitoring-references/maps/us-climate-divisions.php
Thanks,
Deke
I thanked him for his quick response, and I invite folks to read about the changes in the linked documents. Note that they first wrote about the change back in 2011, three years before making it.
Best to everyone,
w.
[UPDATE 2]: Further information from NOAA:
On Sep 29, 2016, at 9:44 AM, Richard Heim <XXXXX@noaa.gov> wrote:
Hello Willis,
Thank you for your email. In addition to Deke’s response, I wanted to elaborate on a few things.
There are two parts to the answer to your question, Have you recently changed your method of calculating the Palmer Drought Severity Index?
The first part has to do with how Wayne Palmer devised his index back in the 1960s. The Palmer drought index uses an accounting water budget method to compute variables describing precipitation supply, precipitation demand, and a soil moisture component over the entire period of record of data. Then the method goes back and computes “normals” or CAFEC values for these quantities, then goes back and uses the CAFEC quantities to compute standardized indices (what eventually become the PDSI, PHDI, and Palmer Z Index). It also has a “backstepping” process that uses a probability that a drought or wet spell has ended feature. (I discussed this in a 2002 paper which can be found here:
http://journals.ametsoc.org/doi/pdf/10.1175/1520-0477%282002%29083%3C1149%3AAROTDI%3E2.3.CO%3B2)
For the current month and last few previous months, this backstepping feature can (almost always does) result in index values for recent months changing when a new month of data is appended. If a different standardizing period is used for the CAFEC computations, then all of the Palmer index values throughout the period of record will change. NOAA NCEI uses the 1931-1990 standardizing period, so the standardizing period hasn’t changed since the 1990s. (One can see that, due to these characteristics of the Palmer model, a user should never compare current Palmer index values they download currently to historical Palmer index values they downloaded years ago.) A slight programming error was discovered in the Palmer program which was corrected in 2013:
http://www.ncdc.noaa.gov/sotc/national/2013/03/supplemental/page-7/
This resulted in very minor changes to the Palmer index values.
The second part of the answer has to do with the data that go into the computation of the Palmer indices. Historical precipitation and temperature are fed into the program. If these historical values change, then the indices themselves will change. For many decades, NOAA has computed the Palmer index values for climate divisions because the climate division dataset was the only century-scale NOAA spatially and temporally complete dataset. The climate division precipitation and temperature values were computed for many years by averaging the station data in each climate division (for 1931-present) and used a statistical technique to estimate climate division values from statewide values (for 1895-1930 because divisional values didn’t exist for that period). This method of calculating divisional temperature and precipitation values resulted in historical changes (discontinuities) that were due to changes in availability of stations (stations opened and closed over the decades) and changes in methods (the 1930/31 discontinuity), and did not reflect true climate conditions. An improved way of calculating the climate division values was implemented in early 2014 to correct for these deficiencies and it is described here:
http://www.ncdc.noaa.gov/monitoring-references/dyk/climate-division-database-transition
This change in the historical data base was vetted both publicly and in the scientific community and was highly publicized at the time. How this change in temperature and precipitation affected the historical Palmer indices was (and still is) shown here:
http://www.ncdc.noaa.gov/temp-and-precip/divisional-comparison/
I hope this answers your question. If you wish further clarification, I can be reached at XXXXX@noaa.gov
Sincerely,
Richard
I trust this clears things up.
w.
It is a complicated enough procedure that the NOAA should provide a detailed description of the method, but I have not found it on their site. See, for example, an excerpt from
http://journals.ametsoc.org/doi/full/10.1175/1520-0442(2004)017%3C2335%3AASPDSI%3E2.0.CO%3B2
that describes the Palmer method:
“Each month of every year, four values related to the soil moisture are computed along with their complementary potential values. These eight values are evapotranspiration (ET), recharge (R), runoff (RO), loss (L), potential evapotranspiration (PE), potential recharge (PR), potential runoff (PRO), and potential loss (PL). The potential evapotranspiration is estimated using Thornthwaite’s method (Thornthwaite 1948). The calculation of these values depends heavily on the available water holding capacity (AWC) of the soil. The PDSI itself depends on a two-stage “bucket” model of the soil. The top layer of soil is assumed to hold one inch of moisture. The amount of moisture that can be held by the rest of the underlying soil is a location-dependent value, which must be provided as an input parameter to the program.”
There seems to be a lot of room for ‘tweaks’ here (that is the charitable term).
“Is NOAA adjusting data to make [fill in the problem du jour] look worse than they are?”
Yes. Next question?
What this really means is there is no actual data in use, except by chance. If science constantly changes methods and tries to adjust the daylights out of all the other data, then we are dealing with fictitious data. You can say we’re adjusting to match a new standard, but in reality, if you want real data, you don’t change methods and pretend you can alter past data to work with the change. There is no science in this—just data manipulation to try and correct for changes that never should have been made if you want a continuous record. Until we have 50 years of untouched, continous data from identical instruments, all we will have is a fantasy. (Yes, I know there will always be some adjustments, but they should be very minimal. It seems no one understands precision, calibration, continuity, etc. They just rearrange values and declare victory. I just don’t see any science in any of this. Imagine if nuclear science were this laissez-faire.)
Some “just FYI” info.:
(Source: https://wattsupwiththat.com/2014/05/13/checking-the-nca-report-against-real-data-reveals-major-discrepancies/ )
The image above didn’t appear — I think the blockquote prevented it — second try:
The tiny little dates along the bottom of the above image appear (at 400%) to be 1865 or 95 –> 2010.
i got a good big computerscreen and they are labeled as followed: 1895 -> 2010. each step is a 5 year interval thus 1895, 1900, 1905,… till 2010 on the horizontal axis, on the vertical axis the PDSI is labeled from -7 to 0 at the bottom of the zero line and from 0 to +6.at the top of it
hope that helps our readers with smaller screens a bit
Thanks, Frederik!
How insidious NOAA is, huh? Acknowledges a bug in a single line of code when pointed to by some authors. Fixes the bug. Prominently announces the fix long in advance of using the fix. When asked by a random member of the public, within hours responds with all that info.
Yet WUWT publishes conspiratorial accusations without even a link to the NOAA site, not bothering to spend any time at all looking for an explanation, nor even asking NOAA via the obvious means of asking that is provided by NOAA.
That was a question, Mr. Dayton, not an accusation.
The final line of the article, and still, no “conspiratorial accusation” in sight.
Given the context of NOAA’s past behavior
(See, e.g., https://wattsupwiththat.com/2015/06/04/noaancdcs-new-pause-buster-paper-a-laughable-attempt-to-create-warming-by-adjusting-past-data/ ; and https://wattsupwiththat.com/2016/03/02/the-karlization-of-global-temperature-continues-this-time-rss-makes-a-massive-upwards-adjustment/ ), {in case you had not heard, the U.S. Congress is aggressively investigating and has found solid evidence of NOAA wrongdoing}
to be skeptical about NOAA is
the only rational response to what the observations and analysis of the above article revealed.
Anything else is:
1. a non-rational, i.e., emotional, response;
OR
2. a rational, dishonest, attempt to defend a known data fr@udster.
So, which is it, Mr. Dayton? Are you just so blindly loyal to NOAA that you are not thinking straight, here? Or are you intentionally defending known fr@udster NOAA for your own purposes (perhaps, you are a wind power or solar industry “investor” and your money is at stake, thus, you are rational, but dishonest)?
Oh, you went FAR beyond merely saying, “Let’s wait and see what NOAA has to say about this… .”**
**What NOAA says has not, as of yet, been revealed, here. A “bug in a single line of code” would most definitely NOT be considered a “responsive” witness answer in a trial/deposition setting. Far more detail is necessary.
Conclusion: We still do not know. More reading/reporting of what NOAA said in the articles/linked pages sent to Mr. Eschenbach must happen before any conclusions can be drawn about the appropriateness of NOAA’s drought Palmer-data adjustments.
NOAA is a political organization.
I wonder if a pharmaceutical company adjusted their data if it would be okay with folks?
Janice Moore
September 29, 2016 at 9:31 am
“Facts: The ‘record-setting droughts in the 2000’s were not really records at all. …”
I’m not the only one who has recognizes that we live in an age of amorality and a major outcome of this is that ends justify means, things can be whatever we want them to be, lying and obfuscation are legitimate tools to defend something that serves a group (or individual) purpose. These things are what give a stubborn longevity to notions that in an earlier age would have been considered too obviously purposely manufactured to give a thought about.
I can still be surprised despite decades of awareness of this phenomenon. A case in point: the hacking of the DNC emails revealed the most egregious bias in the party against Bernie Sanders (whom, I’m personally glad was defeated – scary to think communism has come that close to becoming respectable in America). But what was all the news about? Instead of being shocked at the content of the emails, the establishment were in a smokescreen fury about Russia hacking into American political stuff. There was no evidence it was Russia at all, but linking to Donald Trump’s sarcastic remarks about hoping Russia would find Hillary’s deleted emails, it evolved into acceptance that it had been Russia, and now the establishment Pols are “sure” it was Russia. What was in the emails soon disappeared as a cause for outrage. Alinski’s Rules have been well studied by the elites running the show.
http://www.bestofbeck.com/wp/activism/saul-alinskys-12-rules-for-radicals
Read them, they will look very familiar to you in today’s world.
“I trust this clears things up.”
Not it doesn’t. it does not explain what has temperature to do with drought when it rains.
maybe that the fallen rain evaporates more quickly when it is 90°F then when it is 40°F once the rain stops?
Why aren’t any of the establishment apologists addressing the point made by EM Smith that the highly opaque Palmer drought index is capable of classifying a location as experiencing simultaneously drought and flood:
https://wattsupwiththat.com/2016/09/29/is-noaa-adjusting-data-to-make-droughts-look-worse-than-they-are/#comment-2309444
As always with the warmist activists who are custodians of climate data, the devil is in the detail. The Palmer index allows higher temperatures to change the precipitation level for drought. And the data manipulation in pushing up current temperatures has been so enthusiastically executed by the current generation of millenial climate taleban footsoldiers that the inflated temperatures put California into virtually permanent drought whatever the precipitation.
The thing about lots of people lying about the same thing is that it becomes necessary to coordinate the lies so that they seem consistent. NASA climate data is well into Fawlty Towers territory in terms of management of a manipulated story.