NSIDC's oops moment – uncoordinated changes make for an interesting 24 hours

Many of you are probably aware of some strange goings on over at The National Snow and Ice Data Center (NSIDC) with their Arctic Sea ice graph, specifically, this one here:

You see, up until Tuesday morning, it looked like this:

If you have a keen eye, you might spot the difference, particularly in the proximity of the endpoint of the blue line to the 1979-2000 average line. How does sea ice extent go backwards you ask? Steve Goddard of real-science.com was first to spot it sent out an email notifying many people of his post titled: Breaking News : NSIDC Gets In The Data Tampering Act. I wasn’t convinced there was deliberate tampering going on, because it seemed to me to have all the marks of a processing glitch or something similar, and I made that fact known to many last night.

The two graphs (before and after on April 16th) overlaid look like this:

So not only did the extent change, going backwards, so did the climatology for computing the 2007 line and the 1979-2000 average line. This all came to light about 6PM PST Tuesday night. There was no announcement of this change on NSIDC’s website then.

While it would be easy to start pointing fingers, especially with the timing of the change (right before the extent line looked to cross the average line), I decided the best course of action would be to start asking questions before writing anything.

So I fired off emails to NSIDC’s Dr. Walt Meier and Julianne Stroeve. Strove responded first, within the hour, indicating that she could not see anything wrong, sending the image from NSIDC’s “internal network”, which is the middle graph above. That’s when I sent her the overlay (the bottom image combining the internal image she sent and the web page output image), showing that indeed there was something wrong. The light bulb went on. Walt Meier (who was traveling) responded about an hour later, with this speculation:

Hi Anthony,

Thanks for letting us know. I have a guess at what this might be.

We’re starting to make some changes to our processing to update/improve things, including some you’ve suggested. One thing that we’ve decided to do is to change the way we calculate our 5-day average values. We’ve been doing it as a centered average – i.e., a given day’s value in the plot is actually an average of that day + 2 days before and 2 days after. This caused an issue at the end point because we’d extrapolate to get a 5-day average on the last day, which resulted in wiggles at the end that.

We’re now changing it to be a trailing 5-day average, i.e., a given day’s value in the plot is the average of that day and the 4 preceding days. This will take out the wiggle in the end of the plot (or most of it – there may be some change as sometimes we don’t get complete data and need to interpolate, and later (a day or two) we do get the data and process it.

A key point is that this change doesn’t actually change the data at all; in effect it simply shifts values two days later. In other words, the centered value for Day X is the same as the trailing value for Day X+2.

This change has been implemented in our test environment and we were going to roll it out some time in near future after we tested it for a bit we planned to announce the change. I think that by accident the test code got put into production. I’d need to confirm this, but from the plot differences, this looks like what likely happened.

We’ll look into this and get back to you. I’m traveling tomorrow, but will send a note to people and I or others will get back to you as soon as we can.

walt

That seemed plausible to me, but clearly, both Meier and Strove were caught off guard, and having prominent skeptics alerting you that your most watched public output has gone haywire certainly can’t be comfortable. But, I run a bunch of servers making automated output myself, and I know how things happen. So I gave them the benefit of the doubt, particularly since they were communicating and concerned themselves.

This morning, about 14 hours after the problem was first noticed, this news item appeared on NSIDC’s web site:

Click the image for the story.

That still didn’t explain why Meier and Stroeve were blindsided with news last night from Steve Goddard and I. I queried them more, and as it turns out, they were out of the loop on the implementation. The hand and foot of NSIDC didn’t seem to have coordination on this, and it went online with no notice. Tonight, I got this email from Dr. Walt Meier that explained it:

Hi Steve, Anthony,

I think you’ve probably heard from Julienne and seen the posts we’ve made. But now that I have a chance to respond, I’ll add a few words of explanation and some thoughts. If you want to post these, you’re welcome to.

Thank you to both of you for noticing the issue and bringing it to our attention. Let me clarify (in case it’s not already clear) and provide some context. We are well aware that the daily timeseries plot, as we call it, is closely watched, particularly during the summer melt season. We’ve received various critiques of the plot, which we have taken under consideration to change when we got resources to do it. One them was the “wiggle” in the last two days of the plot. The plot was initially, and by and large still is, meant to provide a simplified glimpse of sea ice extent. The focus was on creating a clean, clear, easy to read, easy to understand graphic. As seen in other plots, the extent is often fairly noisy from day to day. Some of that variation reflects real changes, but much of it is due to limitations in the accuracy of the data or short-term weather effects, such as storm front blowing the ice one direction or another for a short period of time.

Thus, to reduce the noise and better reflect the seasonal trends we decided to use a 5-day average (5 days is a reasonable, though arbitrary, time period to reduce synoptic effects). We chose a centered average because that seemed the most logical. This means the average value is always 2 days behind the latest extent value. However, people wanted to see “today’s” value. So, we decided to provide preliminary values for those last two days by using a simple linear extrapolation. When we got enough data for a full centered 5-day average, we replaced that with the final values. However, this means that the values for the last two days change and one can get a “wiggle” in the data, particularly where there is a day or two of steep change because that day or two gets extrapolated out to 5 days. This can be misleading because, at least for a day or two, the slope may look more extreme than it really is.

I think you’re both familiar with this because it’s been commented on in the past, but I provide the background again for the full context. We refrained from changing it because of three reasons. First, after initial confusion, people understood it, so changing it could cause more confusion. Second, changing the averaging method would slightly change things in comparison with our previous analyses, namely, the date when minimum and maximum extents occur (a shift of two days). This is a minor change, but could cause some confusion. And finally, third, we wanted to make a few other changes and needed to plan resources to do them, so we put this on the list of things to do.

Last week we started to work on some changes. This was simply planning – looking at our processing, assessing what needed to be change. In the process, it was noted that changing the 5-day average would be simpler than we expected and could be done quickly. So I gave the go ahead to do this and was informed a couple days later that it had been done. However, there was some miscommunication. I was expecting that we wouldn’t put it into production immediately, but our developers assumed that it was good to go, so it went into production. Though the change had been discussed amongst all of us, the decision to do it right away happened fairly quickly and I don’t think Julienne was aware that it was in the process of being done.

In any event, what we have now implemented is a 5-day trailing average – in other words, the value plotted for a day is the average of that day and the four previous days. What this means is that there should no longer be a little. The data that we plot on a day should not change and we won’t be doing extrapolation. We think this is a better way to display the data and I think most would agree.

Another issue that wasn’t immediately noticed was that the climatology shifted more than the daily. This is because the climatology used a 9-day average. I don’t remember exactly why this was chosen, but I believe it was to make it look just a bit cleaner, though since it is an average, it already is pretty smooth. And since we were using a centered average, 5-day vs. 9-day, makes little difference. For example, the 5-day average for April 17 is 14.797 million sq km and the 9-day average is 14.801, a difference of 0.004 (4,000 sq km). Effectively, there is no difference because we estimate the precision to be on the order of 0.05 (50,000 sq km). So as long as both the daily and the climatology used a centered average, there was a consistent comparison.

However, when the centered average is moved to a trailing average there is a relative change between the 5-day daily, which slides 2 days, and the 9-day climatology, which slides 4 days. Thanks to Steve for noticing this and pointing it out. We should have it changed to a 5-day by tomorrow so that the comparison plot will again be consistent.

As for the timing of this, as mentioned above, it was mostly simply due to opportunity – we had a chance to make the change, so we decided to do it. Also, knowing that we’re heading toward the summer melt season, it was advantageous to make the change sooner rather than later. As the extent line steepens going through spring and into summer, the “wiggle” is often more noticeable. So making the change now would remove the issue for this summer’s melt season.

The fact that we made this change as the daily extent was nearing the average was entirely coincidental. It never actually entered my mind because I didn’t think it would make any difference (and it shouldn’t once we implement a 5-day average for the climatology). In fact, the change should help because we won’t be using extrapolation that can misleadingly make lines on the plot look closer than what the data really indicate.

Even using a 5-day average, short-term changes in the extent should be taken with some caution. It would be interesting if we did match or exceed the climatology, simply because it’s been several years since it happened. However, the ice near the edge now is all seasonal ice and quite thin and will melt fairly quickly. Any anomaly now will have little to no effect on the summer extent or the amount and thickness of multiyear ice.

As a final, personal note let me make a more general comment. I am saddened that some people have become so cynical about climate scientists and climate data. I can appreciate that scientists have brought some this on themselves. And of course, a healthy dose of skepticism is essential to science. But it is disappointing to see people immediately jump to conclusions and assume the worst. I hope people will take from this explanation that NSIDC, and scientists in general, are working hard to the best we can, both in understanding the science and communicating it. We’re not perfect, we make mistakes. When we find them or hear of them, we try to fix them as quickly as we can and to explain what happened as best we can. I’m proud of our team for working very hard today to address the issues, fix them, and answer questions. I think they did a great job today. And in my experience with other climate scientist, I’ve seen nothing other than that same level of dedication.

Thank you,

Walt Meier

So in a nutshell, NSIDC made a goof in implementation, and in communications. I could find all sorts of criticism for that, but I think they are probably punishing themselves far more than anything critical I might say, so I’ll just let the incident speak for itself.

I will say this though, I can’t even begin to fault them for being upfront and quickly communicative. That is a rare trait in a government agency, so on that basis, they get high marks from me, as well as my thanks. I’m fully satisfied with the explanation.

On Thursday, we’ll likely see this problem rectified, and this time I’m pretty sure I’ll get an email in advance or at the time it happens. I look forward to seeing the changes. On the plus side Dr. Meier tells me that they plan to make the raw extent data available, and that will of course allow us to plot ourselves.

=======================================

UPDATE: 4/19 9AM PST NSIDC has the new corrected graph online – see this story