NSIDC fixes their Arctic Sea Ice graphing problem

As I mentioned yesterday, NSIDC had an oops moment, but with the help of skeptical blogs, was notified of the problem and responded timely and appropriately. They posted this update today:

Update, April 19, 2012: The nine-day trailing average climatology on the daily data graph has been changed to a five-day trailing average, to be consistent with the five-day trailing average for the daily data.

I verified their correction for the climatology was accurate with a new overlay, combining the unaffected graph NSIDC’s Dr. Julienne Stroeve sent me from their internal server storage Tuesday night with the corrected one published on the web today:

http://nsidc.org/data/seaice_index/images/daily_images/N_stddev_timeseries.png

The extent data and the climatology now match, whereas yesterday they did not. The x-axis offset is to be expected given that we are comparing graphs with the temporal data offset due to a trailing average they implemented.

This is what the same comparison looked like yesterday, for graphs made on the same day:

So, problem solved.

Unfortunately, somebody jumped to a conclusion and has already had to issue a correction.

April19: NSIDC graph still appears wrong

now reads:

Correction : April 19 – NSIDC Graph Now Lines Up With April 16

The maxim “haste makes waste” seems appropriate.

The way NSIDC’s Dr. Walt Meier and Dr. Julienne Stroeve handled this should be an example to other agencies that don’t bother to even respond to skeptics.

Advertisements

  Subscribe  
newest oldest most voted
Notify of

NSIDC should take the opportunity to switch to center point averaging. Given the changes this week, switch to center weight immediately.

Ian Hoder

Sorry to be completely skeptical on this, but do you really believe they would have corrected their “error” unless it wasn’t pointed out to them by skeptics? 9 times out of 10 (a number I pulled out of my arse but I bet is close) the “adjustments” favour the CAGW theory.
REPLY: If not by us, eventually it would have been caught. In this particular case, I see no nefarious motives, just simple miscommunications. The graphs match, the wiggle problem is solved, and if the extent line data is reflecting a situation where it will cross the median line, we’ll see it. – Anthony

Anything is possible

Posted this on the previous ice blog, but I think it bears repeating now it has been superceded….
For those of you calling for NSIDC to use a 30-year average : I took the opportunity to raise this issue on Steve’s blog yesterday, and Julienne responded that they were planning to do just that later in the year.
Let’s hope that implementation goes more smoothly than this one…….

Steven Kopits

That standard deviation calculation is way, way wrong. You cannot go from 2 std down back to near average in two months. This would be like going from the bottom of the NBA to second best team in the two months. Doesn’t happen. I don’t know how they calculated that std, but I can tell you with simple visual inspection that they did the math wrong.

Scottish Sceptic

Ok, I have to admit it, they’ve done a good job. Well done.

Rhys Jaggar

Out of interest, does anyone know the longest period of sea ice volume being continually within the statistically ‘normal’ range since the Millennium? If it’s right now, maybe that’s an indication that a slow but steady recovery may be underway?

Which internet gossip blog reports the exact history and status of Watts/Goddard relationship?

GlynnMhor

It’s still going to be wrong to use what they call a ‘trailing average’, since the output of the averaging operator (1,1,1,1,1) is going to be artificially shifted ahead by half of the operator length.
An operator outputs at its centre point, not at its ends.

sailboarder

Its not ice, it is our knowledge of CO2 that is the issue. From Tallblokes workshop:

Sorry to dissapoint everyone, but it looks like we can all go home. We did not cause the big rise in CO2.

Steven Kopits says:
April 19, 2012 at 9:25 am

That standard deviation calculation is way, way wrong. You cannot go from 2 std down back to near average in two months. This would be like going from the bottom of the NBA to second best team in the two months. Doesn’t happen. I don’t know how they calculated that std, but I can tell you with simple visual inspection that they did the math wrong.

I assume they computed the the std the conventional way taking all the extents for a particular day and doing the math on it. There’s no indication the data is at all useful for short term limits as you suggest in you NBA example. To continue that analogy, a team can go from the bottom of the NBA to second best in one year.
What you are expecting would be an interesting display, perhaps you can work on generating it.

Casper

Nice shot, guys!
I can imagine what kind of tricks are applied in the statistical data analysis. Just keep an eye on those cheaters!

John F. Hultquist

Works for me. Thanks Anthony and Drs. Walt and Julianne.
And a “climatology” update is coming this year. Great.

Dear Anthony,
Excuse me, but I see plenty of nefariousness in the graph.
Why else would they hide the actual data and instead post phony “confidence” intervals around a known data set. There is no estimating needed. The facts (the data) are known with 100% “confidence”. I cannot see any reason for the gray zone except to artificially (and nefariously) shrink the actual range of the actual measurements.
Are some years “outliers”? Why does the government (aka Big Brother) need to hide the actual facts? Why vanilla-ize what really happened?
Please forgive me if I hurt somebody’s feelings because I want the actual real data, that I paid through my taxes to have collected. The NSIDC is not the CIA. National security will not be jeopardized. Are the big bad gummit scientists having a pout because some lowlife upstart dared to ask for the ACTUAL DATA!!!
And what’s with the 2007 line? Why is that year the ONLY year graphed? Is there something special about 2007, something so ALARMING such that the paternalistic head-patter gummit mucky mucks need me to see that year and none of the others?
Nefarious, nefarious, nefarious.
Pretty obvious to me that Big Brother is putting out propaganda masquerading as science. Not welcome. Dirty tricksters should be shown the door.

Reblogged this on Climate Ponderings.

kbray in california

I will have a profound sense of relief when those lines cross.
To me it represents the Arctic ice cycle as an oscillation as opposed to a straight line to a hot hell.
It plunges an ice dagger into the heart of CAGW zombieism.
I will pop a cork when it happens.
The ’12 ice appears way more robust that ’07.
A good sign.
However, I am dismayed by the recent flurry of Global Warming “propaganda” that has been
released recently. It seems like “the last gasp” when something is dying. I hope so.
I can’t wait to tell my warmy friends when “Arctic ice is above normal”.
Regardless of which way it goes, the truth will out.
…but I’m rootin’ for the ice.

rgbatduke

Anthony, I don’t know if you or Steve McKintyre have noticed it, but it is very unlikely indeed that the standard deviation for sea ice is correct for late March. It necks in to (eyeballed) half of its extent in early March or mid-April even though the mean is smooth across the two domains. Either there is some extremely odd climate factor that causes a crossover between two essentially independent mechanisms that are responsible for the larger variance before or after or, more reasonably, the standard deviation should be at its fattest all the way across this region. This, by the way, would keep late March in 2007 from being a 5 or 6 sigma event (which is absurd). If one smooths the sd the way it almost certainly should be, March 2007 would be a less than 3 sigma, maybe a 1 in 50 to 1 in a 100 shot (consistent with the whole year).
Oh, and if one adds in the 2007 data, there is no way in hell that the sd is right for any moderate number of sample years. An outlier like this simply bumps the sd to accommodate for few samples (and suggests if anything else a possible bimodal distribution so that the central limit theorem hasn’t yet kicked in, although outliers do happen).
So either the data for late March is “miraculously” tightly clustered compared to expectations from the time immediately before or after or there is something funky about either the data or the way it is being processed into a sigma here. At the very least, something that demands explanation in terms of climate observations over this period, where the ice melts (a tiny bit) consistently where before and after it is melting and growing much more variably.
As they say, “What’s Up With That?”_tm…;-)
rgb

Rob L

NSIDC have done a good job here; listening to criticism, and fixing the problem quickly and openly in a way that we don’t see from the usual AGW suspects. As such I don’t think they deserve any rebuke.
Moving to a 30 year baseline later in the year will certainly screw up the narrative as the confidence intervals will widen and the averages will drop so that relatively low sea ice levels won’t appear anywhere near as impressive as they do with the old baseline.

Sure right … oops, we got caught.

kim

Sumpin’ funky about the last two months.
H/t Bill Illis.
========

Regarding the 1979-2000 average, NSIDC uses it for both Arctic and Antarctic graphs. If 1982-2011 were used, current Arctic extent would probably be above the average, but Antarctic extent would probably be less above the average. I doubt it would make any difference to True Believers and, by definition, objective observers don’t care.
That NSIDC uses 2007 for Arctic reference and 2011 for Antarctic reference is inconsistent. On the one hand, 2007 is a revered year for warmistas but, on the other hand, 2007 has made every succeeding year look like one of cooling.

“Ed_B says: April 19, 2012 at 9:40 am
Its not ice, it is our knowledge of CO2 that is the issue. From Tallblokes workshop
http://www.youtube.com/watch?feature=player_embedded&v=YrI03ts–9I
Thanks for the HT – All I can say is WOW. Everyone needs to watch this presentation.

Sammus

A government agency tried to improve their product, had some internal misscomuniction and growing pains. They then openly admitted this and promptly made corrections. How sad that this is considered so remarkable.

Stacey

The maxim “haste makes waste” seems appropriate.
Dear Anthony
Unlike you to be churlish, at least Mr Goddard corrected his error and he did spot the change in the first place so credit where credit is due. 🙂
Another maxim by a great American “The only man who never makes mistakes is the man who never does anything”
Thank you for all your great work and keeping this blog alive and lively.
S
REPLY: In the original post I gave him full credit for spotting it first, but he also published first, and used a title that said “data tampering” suggesting malfeasance where there was simply a boneheaded mistake brought on by poor communications at NSIDC. He published that without having asked NSIDC a single question, that responsibility fell to me. Steve then compounded his first error by rushing out an opinion on the correction which he had to retract today.
So yes, in this case, “haste makes waste” – Anthony

Kelvin Vaughan

The big question now is will it? wont it?

Do NOT attribute to arravice or malevolence that which may more simply explained by ignorance, stupidity or laziness. Anonomous…
However, I’d add, “Hey, a calculational mistake not immediately caught…as there are a limited number of reviewers!” (Which I think is the case here.)

John Blake

Kudos for simply displaying base-data with integrity? In climate-scientology terms, O Wow. Now we’d like to see RGBatduke’s cogent concerns addressed.

kramer

Funny how this ‘opps’ problem just happened to result in the line going back farther from the mean… I don’t think this just happened on its own.

Honest to Pete, why have “confidence” intervals at all? Why not reveal the actual data?
All you experts out there, maybe you can explain to me why the data should be hidden and an arbitrary gray zone plotted instead?
Does it give you a warm fuzzy to NOT see the data? Do you prefer pablum to steak?

Hi all,
Yes, just to confirm, we are looking to switch to a 30-year climatology. This is more complicated because it doesn’t only affect our daily time series plot, but all of our images. Also, the timing of it is a bit uncertain. We haven’t changed the climo before, so not sure of any issues that may be in the code. Also, we want to do it at non-sensitive time (e.g., near the minimum). We’re tentatively planning to use 1981-2010. We now have final (non NRT) data through 2010, and this is consistent with the NOAA normals. If folks have other suggestions, we’d be happy to hear them and take them under consideration – post a reply here.
The other thing in the previous post that got lot of attention were my brief comments on trust in climate science. I can’t expound on that here, but maybe I’ll try to expand on that sometime when I have time.
Thanks.
Walt Meier
NSIDC

Peter

Might it be remarkable that in a single winter the ice extent from rather low to median.

Ian E

Sure points up the need for transparency! [Are you there, Mann?]

D. J. Hawkins

Mike Dubrasich says:
April 19, 2012 at 10:00 am
Dear Anthony,
Excuse me, but I see plenty of nefariousness in the graph.

Pretty obvious to me that Big Brother is putting out propaganda masquerading as science. Not welcome. Dirty tricksters should be shown the door.

This is the sort of rant that is truly pointless. You can go to NSIDC’s web site and download the data for yourself, for crying out loud. Do anything you want with it, plot it any way you please.
“Hide” the actual data? If you tried to plot every year on the graph it would be a jumble of lines. Try it yourself, if you’re not too lazy.
Why 2007? Seriously, you don’t know why? Try year of minimum extent.
Paranoid, paranoid, paranoid.

Scottish Sceptic

Walt Meier says: April 19, 2012 at 11:21 am
Hi all,
Yes, just to confirm, we are looking to switch to a 30-year climatology.

May I suggest that you just go ahead as soon as you are ready and run the old and the new in tandem.
Alternatively, you could cheat … just be nice to us sceptics and we’ll all roll over on our backs and purr.
But seriously. Just do the right thing, do it well and whilst we might grumble, … we don’t bite.

Tenuk

In the mean-time, Arctic-roos now updated, showing 2012 above the 1979-2006 average for both extent and area!!!
http://arctic-roos.org/observations/satellite-data/sea-ice/ice-area-and-extent-in-arctic. It will be interesting to see if the higher NSIDC average (based 1979 -2000) also crosses in the next few days. It is puzzling how temperature up in the Arctic is all over the place…
http://ocean.dmi.dk/arctic/meant80n.uk.php
Perhaps our quiet sun is starting to have an effect?

I cannot help but get a chuckle out of how the ‘pigbook’ came
out today talking about government waste. It is from the
‘Citizens Against Government Waste.” CAGW

Latitude

…call me cynical
Julienne’s very first post….on Steve’s blog…..she said he’s right…obviously they didn’t know about it……Steve caught it……
===========================================
Julienne Stroeve says:
April 18, 2012 at 7:41 pm
Steve is correct that the switching to a trailing mean resulted in a mis-match in the timing between the climatology and the daily extent since different averaging periods were used (this date mis-match wasn’t a problem in the centered means). We’re working to fix the problem. However, Steve should note that switching to a 5-day trailing climatology mean actually puts the current extent farther away from the climatology.

Peter

Steven Kopits: “That standard deviation calculation is way, way wrong. You cannot go from 2 std down back to near average in two months. This would be like going from the bottom of the NBA to second best team in the two months. Doesn’t happen. I don’t know how they calculated that std, but I can tell you with simple visual inspection that they did the math wrong.”
I think the 2 standard deviations is measured on the 1979-2000 data. So essentially the chart is showing that the sea ice extent for 2007 and 2012 has been outside of (i.e., less than) 2 standard deviations of the mean for 1979-2000. You can’t say by “simple visual inspection” of a chart that “they did the math wrong”. If you think you are smarter than the scientists at NSIDC, then take the 1979-2000 data, calculate a mean and standard deviations for each day and plot it on a chart. But then, I’m sure you won’t be doing that. It’s much easier for you to simply dismiss data that tells you something you don’t want to believe. It’s human nature to just say “that can’t be true” rather than face the facts. It’s also a form of cowardice.

NSIDC fixes their Arctic Sea Ice graphing…

Blue Sky

Thanks Mr Meier for taking time out of your schedule to explain the “oops” and to confirm future plans.

EW-3

What year was it that the Gore and his mob said we’d have an ice free north pole ?
2013 sounds familiar. Anyone know ?

Latitude

Walt Meier says:
April 19, 2012 at 11:21 am
Also, the timing of it is a bit uncertain. We haven’t changed the climo before, so not sure of any issues that may be in the code. Also, we want to do it at non-sensitive time (e.g., near the minimum)
==============================
You have a mean streak………….rotfl
….save yourself, do it sometime in the middle

NZ Willy

SG may have spotted it 1st, but I claim credit for having spotted it 0th in my screed 3 days prior on “Tips & Notes” on the software-model-caused interrelationship of Arctic & Antarctic ice volumes — of which this is an example, as Antarctic anomalies have hopped around in reverse to Arctic during this episode. 🙂

See - owe to Rich

Mike Dubrasich seems to be relatively new, and a little angry, but he is asking for information, so someone should be kind (and patronizing, impossible to avoid). He said at
April 19, 2012 at 11:14 am
“Honest to Pete, why have “confidence” intervals at all? Why not reveal the actual data?
All you experts out there, maybe you can explain to me why the data should be hidden and an arbitrary gray zone plotted instead?”
Well, the gray zone simply surrounds a line in the middle which _is_ the mean data, so look at that and ignore the gray if you want. The gray is somewhat interesting because when the curve stays below it for a long time, as has happened a lot recently, it says (without explanation) “something out of the ordinary is going on here”. That something may be an oceanic warm phase, or it may be CO2, or it may be the sun, and we’re trying to work it all out.
Regarding Mike’s earlier question about 2007, it is displayed because it was the biggest Arctic melt in the recent record, so worth comparing with. Warmists think we are going to go under that curve a lot in the next decade, and sceptics say “really? give me a 3-1 bet on it and I’ll take you on”.
I hope this helps, Rich.

Joseph Bastardi

about the retraction.. let me say this. He forced the issue and was right about everything leading up to it, so we judge him by the last mistake? Cmon. The fact is that NSIDC does a great job, but because of some of the other things going on in this debate one has to be skeptical. I take them at their word, but I have made mistakes with them too. it doesnt help that if my memory serves me correct , the idea of the ice cap death spiral came out of there several years ago, and one of the years mentioned was by 2012. So you cant blame people that look at this for being a bit hesitant at times to trust but no verify.
The fact that the people at the top DID NOT EVEN KNOW WHEN THIS WOULD BE IMPLEMENTED is troubling. If they waited till mid summer it would seem better. I would like to trust, but again given all that is going on in other areas, why now, and why without any message before hand that this was in the works.
As for the retraction, I believe the person has shown he is a good and honorable man. That he is judged by his one mistake, when we see people getting away with things one after another, seems kind of odd to me.
Personally after an honest mistake I made over confusion I stay away from it. Joe Romm went out of his mind, so I watch the Norsex site for my ice fix.
Trust, but verify

Mycroft

If the ice keeps recovering like this, i can see Anthony adding Gore’s and others iceless Arctic to the Climate Fails Page …… LOL

“If you tried to plot every year on the graph it would be a jumble of lines.”
Yes … its hard to tell a narrative if people know how different the data is from year to year.
Better if the “public” think the average line is actual data.

Statistical calculation quibbles aside, I think that the way that this “Ooops” has been handled by Dr. Walt Meier, Dr. Julianne Stroeve and Anthony is an object lesson in how such matters ought to be resolved.
It is called ‘talking with each other’. When lines of communication are not obliterated by political or other considerations simple truths emerge. I, for one, am most cynical when it comes to matters concerning ‘official’ statistics relating to matters climatological but the manner, style and speed in which Dr. Meier responded to this deserves a good deal of respect. Dr. Walt Meier and Dr. Julianne Stroeve, thank you so very much for being so good at your public relations. Would that so many of your colleagues were as open and responsive.

Mike Smith

Like making sausages, the process of research and presenting the results frequently involves more than a little messiness. This was clearly no exception. But, at the end of the day, the system worked perfectly. Some inevitable errors and ambiguities were resolved to the satisfaction of all involved, in an incredibly short period of time. Heck, it was a day or so!
Congratulations to Anthony, and Drs Meier and Stroeve. Keep up the wonderful work!

philincalifornia

EW-3 says:
April 19, 2012 at 11:58 am
What year was it that the Gore and his mob said we’d have an ice free north pole ?
2013 sounds familiar. Anyone know ?
++++++++++++++++++++
The one I remember clearly (although the video has been deleted at Youtube) was at a visit to a German museum in the second week of December, 2008 – 5 years to an ice-free Arctic, so yes 2013.
Also, anyone know what’s going on with the NORSEX plot? It hasn’t been updated since the 5th. I’ve seen this question asked at least a couple of times but haven’t seen an answer. If I missed it, could someone point me to the answer please.

Al Gore`s models predicted there was a 75% chance there would be an ice-free Arctic by the summer of 2014.