Guest essay by Michael Hove [Please note, new update at the end]
In the Spring of 2016, I was updating a North Dakota water resource presentation, which contained a Palmer Hydrological Drought Index (PHDI) chart from April of 2013. I was replacing an older NOAA PHDI chart (May 2005 to October 2013) with an updated NOAA PHDI chart (May 2005 to March 2016). These charts are simple screen captures from the NOAA web site. Then, something caught my eye. I noticed that the summer of 2011 was wetter in the 2016 dataset than in the 2013 dataset, and the summer of 2013 was drier in the 2016 dataset than in the 2013 dataset. I scaled both charts on the Y-Axis to check my first visual impression. The results of this comparison are shown below in Figure 1. This comparison showed that there were several years where the newer 2016 dataset was different from the older 2013 dataset, both wetter and drier. Obviously, NOAA has modified the historic data.
This prompted me to look more closely at some other Palmer Drought Index data I have collected over the years. In 2006 I wrote a report using the Palmer Drought Severity Index (PDSI) and had downloaded the PDSI data from the NOAA website for the time frame of January 1977 to September, 2006. This 2006 dataset was for all nine NOAA climate divisions in North Dakota. Figure 2 shows the North Dakota climate division, which follow county boundaries.
Next, I downloaded (in May 2016) the PDSI dataset for the same time frame as my 2006 PDSI dataset (January 1977 to September, 2006), see Figure 3. I then plotted the two dataset downloads (2006 & 2016) as bar charts for each climate division and also plotted the differences between the two datasets for each climate division. These charts provide a nice visual comparison for how NOAA has modified the historical PDSI data.
To provide for a better visual comparison of the differences for all nine climate divisions, I created scatter plots of the “gray differences bar chart” and calculated the regression line for each of the charts. I then scaled the charts so that all nine of the climate divisions fit on a single page. From the scatter plots and regression you can see that seven of the climate divisions have had their historical data adjusted to produce a “drier” trend line, while two have their historical data adjusted to produce a “wetter” trend line. Also included are histograms of the “differences” data for each climate division.
I welcome any thoughts regarding the trends shown between the “Old Historic” and New Historic” data.
[UPDATE BY WILLIS]: Michael, thanks for your fascinating post pointing out the changes in the PDSI. Like you, I was unaware of them. After reading this post, I wrote to NOAA to ask about the change. Within hours I had the following reply:
On Sep 29, 2016, at 8:27 AM, Derek Arndt – NOAA Federal wrote:
When we switched to nClimGrid/nClimDv as our base US dataset in 2014, we recalculated many of our derivative products including the drought indices.
The methodology is documented in this 2014 article:
This pending change and some early comparisons were first shared with the community in 2011.
I thanked him for his quick response, and I invite folks to read about the changes in the linked documents. Note that they first wrote about the change back in 2011, three years before making it.
Best to everyone,
[UPDATE 2]: Further information from NOAA:
On Sep 29, 2016, at 9:44 AM, Richard Heim <XXXXX@noaa.gov> wrote:
Thank you for your email. In addition to Deke’s response, I wanted to elaborate on a few things.
There are two parts to the answer to your question, Have you recently changed your method of calculating the Palmer Drought Severity Index?
The first part has to do with how Wayne Palmer devised his index back in the 1960s. The Palmer drought index uses an accounting water budget method to compute variables describing precipitation supply, precipitation demand, and a soil moisture component over the entire period of record of data. Then the method goes back and computes “normals” or CAFEC values for these quantities, then goes back and uses the CAFEC quantities to compute standardized indices (what eventually become the PDSI, PHDI, and Palmer Z Index). It also has a “backstepping” process that uses a probability that a drought or wet spell has ended feature. (I discussed this in a 2002 paper which can be found here:
For the current month and last few previous months, this backstepping feature can (almost always does) result in index values for recent months changing when a new month of data is appended. If a different standardizing period is used for the CAFEC computations, then all of the Palmer index values throughout the period of record will change. NOAA NCEI uses the 1931-1990 standardizing period, so the standardizing period hasn’t changed since the 1990s. (One can see that, due to these characteristics of the Palmer model, a user should never compare current Palmer index values they download currently to historical Palmer index values they downloaded years ago.) A slight programming error was discovered in the Palmer program which was corrected in 2013:
This resulted in very minor changes to the Palmer index values.
The second part of the answer has to do with the data that go into the computation of the Palmer indices. Historical precipitation and temperature are fed into the program. If these historical values change, then the indices themselves will change. For many decades, NOAA has computed the Palmer index values for climate divisions because the climate division dataset was the only century-scale NOAA spatially and temporally complete dataset. The climate division precipitation and temperature values were computed for many years by averaging the station data in each climate division (for 1931-present) and used a statistical technique to estimate climate division values from statewide values (for 1895-1930 because divisional values didn’t exist for that period). This method of calculating divisional temperature and precipitation values resulted in historical changes (discontinuities) that were due to changes in availability of stations (stations opened and closed over the decades) and changes in methods (the 1930/31 discontinuity), and did not reflect true climate conditions. An improved way of calculating the climate division values was implemented in early 2014 to correct for these deficiencies and it is described here:
This change in the historical data base was vetted both publicly and in the scientific community and was highly publicized at the time. How this change in temperature and precipitation affected the historical Palmer indices was (and still is) shown here:
I hope this answers your question. If you wish further clarification, I can be reached at XXXXX@noaa.gov
I trust this clears things up.