Is the NULL default infinite hot?
January 31, 2010 by E.M.Smith
see his website “Musings from the Chiefio”
The empty ocean goes infinite hot on a null anomaly
What to make of THIS bizarre anomaly map?
What Have I Done?
I was exploring another example of The Bolivia Effect where an empty area became quite “hot” when the data were missing (Panama, posting soon) and that led to another couple of changed baselines that led to more ‘interesting red’ (1980 vs 1951-1980 baseline). I’m doing these examinations with a 250 km ’spread’ as that tells me more about where the thermometers are located. The above graph, if done instead with a 1200 km spread or smoothing, has the white spread out to sea 1200 km with smaller infinite red blobs in the middles of the oceans.
I thought it would be ‘interesting’ to step through parts of the baseline bit by bit to find out where it was “hot” and “cold”. (Thinking of breaking it into decades…. still to be tried…) When I thought:
Well, you always need a baseline benchmark, even if you are ‘benchmarking the baseline’, so why not start with the “NULL” case of baseline equal to report period? It ought to be a simple all white land area with grey oceans for missing data.
Well, I was “A bit surprised” when I got a blood red ocean everywhere on the planet.
You can try it yourself at the NASA / GISS web site map making page.
In all fairness, the land does stay white (no anomaly against itself) and that’s a very good thing. But that Ocean!
ALL the ocean area with no data goes blood red and the scale shows it to be up to ‘9999′ degrees C of anomaly.
“Houston, I think you have a problem”…
Why Don’t I Look In The Code
Well, the code NASA GISS publishes and says is what they run, is not this code that they are running.
Yes, they are not publishing the real code. In the real code running on the GISS web page to make these anomaly maps, you can change the baseline and you can change the “spread” of each cell. (Thus the web page that lets you make these “what if” anomaly maps). In the code they publish, the “reach” of that spread is hard coded at 1200 km and the baseline period is hard coded at 1951-1980.
So I simply can not do any debugging on this issue, because the code that produces these maps is not available.
But what I can say is pretty simple:
If a map with no areas of unusual warmth (by definition with the baseline = report period) has this happen; something is wrong.
I’d further speculate that that something could easily be what causes The Bolivia Effect where areas that are lacking in current data get rosy red blobs. Just done on a spectacular scale.
Further, I’d speculate that this might go a long way toward explaining the perpetual bright red in the Arctic (where there are no thermometers so no thermometer data). This “anomaly map” includes the HadCRUT SST anomaly map for ocean temperatures. The striking thing about this one is that those two bands of red at each pole sure look a lot like the ‘persistent polar warming’ we’ve been told to be so worried about. One can only wonder if there is some “bleed through” of these hypothetical warm spots when the ‘null data’ cells are averaged in with the ‘real data cells’ when making non-edge case maps. But without the code, it can only be a wonder:
With 250 km ‘spread’ and HadCRUT SST anomalies we get bright red poles
The default 1200 km present date map for comparison:
GIS Anomaly Map for November 2009
I’m surprised nobody ever tried this particular ‘limit case’ before. Then again, experienced software developers know to test the ‘limit cases’ even if they do seem bizarre, since that’s where the most bugs live. And this sure looks like a bug to me.
A very hot bug…
I recall reading somewhere else that ‘9999’ isn’t a temperature but is instead an error code. Not sure what it’s complaining about, you’d need to look at the code to find out, and there are probably more than one set of conditions that produce that output.
wonder if that model was peer reviewed or beer reviewed?
Reminds me of all that Steigian red from last year’s Antarctic smearing.
So, if the thermometer is dead, it’s red!
http://bastardoldholborn.blogspot.com/2010/02/bbc-and-climate-change.html
fascinating!
Two winners today, no three
(1) Times are a-changin’ when Google Ads shows this
First-rate, too, IMHO. Want to run a thread on it here?
(2) important post here: the ridiculous QC of GISS has got to come out alongside that of CRU
(3) yesterday’s UHI arrived here this morning, just after I sent Warren Meyer a set of pics to think of using to enliven above video, one of which was my rendition of Anthony’s Reno urban transect overlaid on another pic resembling that used for Case 8. Ha, I feel proud.
But I still don’t know how Anthony manages to keep rolling it out, watching other blogs, writing papers, challenging Menne, and putting courteous notes in for the (declining) trolls here. Thank you from the bottom of my heart, moderators too.
If GISS truly uses 9999 for null, it just shows how out-of-date their software is.
It appears they don’t discard or ignore that value, either, when they create an image from the data. Wouldn’t it be better to show those areas as black?
One more indication that the code is ether poorly done or cooked.
Can we test this hypothesis with a bit of reverse engineering of data? Null values in data sets are a common problem I find in my work. (Software dev)
Why don’t they publish the freakin source code? Surely all these scientist types are open source zealots anyway.
So CRU don’t have a monopoly on crap warmmonger software. Who’d have thunk it?
999 or 9999 (and negative versions) are typically used to represent errors, as values such as 0 for no data can be misinterpreted as real data. Blank values can also cause an error in some code, hence the use of a silly low or silly high value, with the intention that the user imediately notice it or that it can be immediately identified as wrong. The code *should* ignore such values, but its not unheard of for these values to be pulled in as someone has used “9999” instead of the “999” value the software ignores.
I understand the code works by smearing hot spots around (homogenisation), so there is potential for one of these large numbers to make it through only to be smeared around between the other low numbers so the error is not immediately obvious to the eye. Hansen et al would expect (or would like) to see warming at the poles, so they make not see this as an error due to their preconcieved solid belief in global warming.
As chiefio states, this is why its so important to test the softwate or code to extremes. I use numerous engineering and hydraulic modelling packages, and reguarly find small and occassionaly large flows. I have a similar habit of testing extreme scenarios so I know where the software can and can not be trusted, but also manually replicating sections of the software engine using their own equations and comparing results – using this method I end up identifying faults that have gone un-missed for years!
Great work! This just demonstrates why we should never stop questioning – even if we are lowly peasent “non-climate scientists” incapable of computer engineering, maths, statistics etc..
The value 9999 is a standard ‘no data’ value. This is nonsense.
Interesting that all the errors of this nature (buggy code, etc) that I’ve seen show warming. I wonder what the odds of that are, if one assumes that they are indeed all errors and no bias was present? My guess is very long odds indeed, and that argues strongly for a non-accidental explanation.
Arizona CJ
Slightly OT, mod, but I need an answer fast and here’s the place to get it.
Australia’s Climate Change Minister, Penny Wong, claimed in an article in today’s The Australian that, “Globally,
Surely that’s a porky.
Slightly OT, mod, but Australia’s Climate Change Minister has claimed the following in today’s The Australian.
“Earlier this month, the Bureau of Meteorology released its 2009 annual climate statement. It found 2009 was the second hottest year in Australia on record and ended our hottest decade. In Australia, each decade since the 1940s has been warmer than the last…
Globally, 14 of the 15 warmest years on record occurred between 1995 and 2009.”
Breaking News;
Glenn Beck just reported on Dr Rajendra Pachauri’s steamy new sex novel and Obama’s new budget that includes $650 billion from carbon taxing, says Beck, ” what does the Obama administration know that we don’t know?”
The world wealthy elite are pushing ahead with their global government carbon taxing schemes.
And if you think this is a joke, it’s not.
I watched the Davos world economic forum interviews and the most powerful people in the world are talking as if the global carbon tax is a given.
Carbon taxing is based on a lie for Christ’s sake.
Just another example of poorly written or intentionally deceptive code.
If a computer model, cannot be audited in public and validated that the code that is reviewed is the actual code being used to run the simulations the output is no better than throwing dice or tossing darts at a wall map, and should never be used in any way shape of form in the development of policy or legislation.
We need a law passed that explicitly blocks use of any model simulation for policy, regulation or legislation without a verifiable audit process.
Any computer programmer that has gotten past “Hello world” level programming knows it is impossible to write error free code containing thousands or millions of lines of code. The best you can do is to thoroughly test for and eliminate the coding and logic errors that are most likely to bite you under unexpected conditions.
A couple decades ago, I was asked to beta test a relatively simple program used by FEMA to process some shelter data. The first think I did was every time it asked for a value I hit every odd ball key on the key board like ! @ # etc. I blew the program up about a dozen times simply because such nonsense key strokes were not tested for and trapped with an error routine that enforced reasonable input limits.
If this code cannot be open sourced or audited by a professional soft ware audit process it should be barred from use.
Larry
You just broke their online widget by asking it to do something that didn’t make any sense. Do you really think the error flag 9999 is entering into real calculations anywhere?
I really don’t understand your fascination with changing the baseline. It’s a waste of time. You’ll just nudge the absolute values of the anomalies up and down, but the trends won’t change.
As for your Bolivia effect, you could probably figure it out with just one day’s effort. GISS finds the temperature at each grid point by adding in surrounding stations, linearly weighted based on the distance from the grid point. Just take the current surrounding stations, and you can confirm their calculation. Then, you can easily check to see how good the interpolation is.
If there is data from Bolivia before 1990 or whenever, just repeat that calculation, pretending that you can’t see the Bolivian stations. Then compare the interpolation to the actual Bolivian data, and you’ll see how good the interpolation is. If you want people to take you seriously that the interpolated values for Bolivia are not good, why don’t you actually do that analysis and find out?
OFF TOPIC –
The British monthly car magazine ‘What Car’ has an article on global warming. At first glance it seems to take the ‘rising CO2 is going to cause dangerous harm – and man is to blame’ claim as factual and even has a 700,000 year graph of CO2 which looks pretty hockey stickish to me at its end.
I hope if needs be someone will correct any of their errors . The magazine claims that current CO2 is at an all time high, but my dim memory thinks it has read something about it being much higher in the far past.
I see you haven’t noticed it doesn’t matter if this part of NASA is correct or not because they just keep getting the money. The moon program is now dead and Obama is rerouting that money to climate monitoring. Hansen and the boys are all grins-That’s what makes me see red!! Obama needs cap and trade for the taxes and the control. As long as these type people are in control we will all be seeing red ,especially in our bank statements. I don’t know when everyone will wake up and see this. Maybe Mass. was a start-we’ll see.
lengthy interview with Pachauri in the Hindustan Times:
Pachauri: ‘They can bend me, but they can’t break me’
Q: Can you provide us revenue for 10 years to prove there is no link between IPCC and TERI?
A: …. I have proved myself in several aspects in the world. Not in eyes of Sunday Telegraph. Fortunately, there are a few people, thank God, like the Sunday Telegraph. But yes, if you want, we can provide the accounts, the payments made over these 10 years.
http://www.hindustantimes.com/They-can-bend-me-but-they-can-t-break-me/H1-Article1-504204.aspx
COMMENT BELOW ARTICLE: Yes, we do want … what are you waiting for? We’ve already asked. Richard North, Sunday Telegraph
also from the interview which needs to be read in full:
Q: Did failure at Copenhagen help climate skeptics?
A: No agreement at Copenhagen in fact encouraged some of the deniers, and those who are financing them with, maybe millions, who knows, billions of dollars.
Hi,
I’ve been digging through the code (just getting it to run on my beefy Linux box). I notice that when they do the griding:
1) They seem to use a linear weighting around the center of each grid, so a station twice as far away from the center contributes half as much to the total – shouldn’t that be some form of inverse square law relationship instead? as the temperature at a given point interacts in all directions – using pure linear is very akin to resistance over distance in a wire.
2) From what I’ve seen in the code (I maybe wrong, being a good many years since I did FORTRAN) they have some weird rule around where a station can be outside 1200km yet still inside the grid – but in this case they give all such stations the weight of the furthest station within the 1200km radius – so pulling such external stations ‘inside’ the 1200km boundary.
once I have the code running I’ll see how this impacts the results.
It seems to me that temperature data analysis methods must accept both positive and negative single or double precision values for input. For this reason, the developer will use an out-of-range value to represent NULL data. If the application is written properly, whatever value is used to represent NULL should be tested for and not be used in any calculation that drives output.
Many modern programming languages now support the NaN value in numeric data types, which can be used to represent NULL or erred data. The beauty of NaN is that it will throw an exception if used in a calculation or used as direct output to any control that expects a real number. That assures that a NaN cannot be accidentally included in analysis, even if by programming error. Me thinks it is time for NASA to upgrade to modern programming languages and systems.
I wonder if they’ll “lose” data for Scotland this winter now that we’ve heard they have had their coldest ever Dec-Jan on record.
Oh, all you skeptic scientists have missed out and I’m sorry. But at least you have kept your dignity and everything else. Keep after them- it just might turn around in your favor. Everyone else help out as much as possible by voting and giving.
9999 is ‘null value’
Smith is graphing a null value and suggesting that NASA preforms the same idiotic procedure that he does.
:eye roll:
Economic Times, India: IPCC claims on Amazon, ice not based on science
The pressure is be on the IPCC to improve its procedures. “The goof ups that are being reported are all from Working Group II. Clearly, evangelism has overtaken science. I am told that there are many things in the summary for policymakers that are not there in the Working Group reports. There is a clear need to distinguish science from advocacy, and the IPCC should stick with science,” environment minister Jairam Ramesh said.
http://economictimes.indiatimes.com/news/politics/nation/IPCC-claims-on-Amazon-ice-not-based-on-science/articleshow/5525992.cms
The color for no data on the GISS maps is Gray not Red and it won’t show up in the temp anomaly scale either:
http://tiny.cc/F20SI
There is a bug in the code that they use for their mapping program. When you add in the SST data the red in the oceans turns white.
@ Michael (14:51:46) :
Oh, that’s just Barry playing his Chicago-style politics, “Pass Cap and Trade or Granny won’t get her medicine and babies will starve.” Etc. Don’t worry, they’ll make the money up with the massive defense cuts once the job is labeled finished in Iraq and Afghanistan. After all, Osama bin Laden agreed global warming is a major threat therefore he is now an ally. Time to shake hands and share the apologies! Oh, and kudos to North Korea and Iran for helping to save the planet with clean energy. Great job, friends!
I doubt the 9999 is getting worked into the calculations, that would make quite a change. But they should have a different shading for the NULL case. Too bad there are no “grey areas” in (peer-reviewed!) Climate Science. And that’s settled!
You know, it might be nice to do an analysis to determine if GISS is letting 9999s bleed into the real observations before accusing them of doing so. Given how much a single 9999 would skew the global temperature (since anomalies are generally ~+/- 5 C max), it wouldn’t be too hard to catch.
I’d suggest poking through the Python version of GISTemp instead of the old FORTRAN mess: http://clearclimatecode.org/
Until you identify an error, this is just blowing smoke. The much more likely explanation of high arctic temps is polar amplification: http://www.realclimate.org/index.php/archives/2006/01/polar_amplification/
It gets even better, when one compares interval 1951-1980 to interval 2009-2009.
Then one gets a bug-map of the area, where the GISS has now no real coverage. One sees directly the geography – where the GISS now avoids any real data – half of the Africa, center of the south America, almost all Greenland and Arctica, big chunks of Canada, all the cold thermohaline suroundings and even the center of Antarctica (didn’t the NASA yet discovered there is a big base on the south pole?…) etc.
Here I made some pictures for a quick comparison:
http://xmarinx.sweb.cz//gissmaperror.JPG
http://xmarinx.sweb.cz//polargissmaperror.JPG
It looks like the area covered by the GISTEMP panel shrinks even much faster than the infamous arctic sea ice… 😉
One then doesn’t much wonder why the GISTEMP has recently so big divergence from UAH, RSS and even HADCRUT3 trends:
http://preview.tinyurl.com/yar6759
Shoddy workmanship? Accidental? I don’t think so. This is NASA.
Errors that have been found with NASA work in the GISS (James Hansen, Gavin Schmidt) department over the years would not happen at a lesser eshtablishment, let alone NASA. Someone along the line would have caught and corrected them. Especially with the particularly embarassing Sep/Oct 2008 “error” you’d think GISS would be keen to root out these issues.
I could speculate that there is a tentacle of politics extending from Washington to GISS that uses these computer “errors” for its purposes. I think other people are speculating the same.
>> Arizona CJ (14:35:57) :
Interesting that all the errors of this nature (buggy code, etc) that I’ve seen show warming. I wonder what the odds of that are, if one assumes that they are indeed all errors and no bias was present? My guess is very long odds indeed, and that argues strongly for a non-accidental explanation. <<
Since they expect warming, the true believers are much less likely to take a second look at an error that increases warming. Errors that show cooling (along with non errors) would be highly scrutinzed.
Zeke,
The python code would be the exact wrong thing to look at.
The graphic displayed on the NASA site is not generated from the code
that NASA has released. Since the CCC project is rewriting THAT code,
your suggestion doesnt help much. Simply the python code doesnt draw the grpahic that EM is pointing to. What code does? Dunno.
have those charts ever been used in publication? Dunno.
Now, if journals had reproducible results requirements The DATA AS USED
and the CODE AS USED to product any result ( table, chart, graph, textual claim) we might be able to tell.
So, think of EMs post as a bug report.
is there a bug?
yes.
Does the bug hit anything other than a graphic drawn on the web page?
dunno.
rbroberg (15:07:35) :
After you have finished rolling your eyes would you have a look at GISTemp temperature values compared to other data sets?
GISTemp shows warmer values than all other sets. GISS is carrying on about hottest year this, 2nd hottest year that. No one else is.
ps. is the sky falling? are you living in a world “terrifying vistas of reality”?
:rolls eyes:
A bit OT, but you have to have a look at the Pachauri video on Richard North’s blog (eureferendum)
Pachauri seems hell-bent on single-handedly destroying the credibility of the IPCC.
I mean, calling Jonathan Leake a skeptic conspirator… who would have thought?
Oh, and the Himalayan glaciers will melt by 2050, not 2035 – from the horse’s mouth.
Chiefo,
Do you have a recommendation on what is an effective way for me, “USA Taxpayer”, to contact GISS for the release of the code by and request more open GISS forums for analysis?
I would greatly appreciate it.
John
@ Zeke Hausfather (15:19:40) :
He said in the article he is not poking through the real code that creates this effect, FORTRAN or otherwise, as it is not publicly available.
Clear Climate Code seeks to emulate the GISTEMP program with easy-to-follow code, and otherwise make reliable code for real climate science. What value is there in playing with their stuff when one is seeking to find out how the real GISS code works?
Oh look, you cited Real Climate as an authoritative source of reliable scientific information. Yeah, that’ll sure win you some points around here.
rbroberg (15:07:35) : 9999 is ‘null value’
Smith is graphing a null value and suggesting that NASA preforms the same idiotic procedure that he does.
Um, no.
The graphs are straight from GISS. I’ve done nothing but download their graph from their software and their data. It’s 100% them, and not about me at all. So you can stop rolling your eyes and try to focus them on the graph…
I’ve been a serious computer programmer for over two decades and I can assure you that null is not 9999. It’s typically 0, but can be other values depending on where it’s uses. I have never seen 9999 for null. Besides, null means literally means unknown in computer programming. So, if 9999 is somehow “bleeding” into the way they are interpolating the readings then it’s proof that climate science is corrupt science. I doubt this is the case but if it is then we would have irrefutable proof that they are committing a fraud as non one, not even a high school student would do this.
Mosh,
I agree with you on how to best debug their online maps. However, the more pernicious accusation that 9999s are bleeding through into the actual temperature data would be much easier to catch in the python implementation unless you really enjoy trying to read through poorly documented Fortran.
Zeke Hausfather (15:19:40) : You know, it might be nice to do an analysis to determine if GISS is letting 9999s bleed into the real observations before accusing them of doing so. Given how much a single 9999 would skew the global temperature (since anomalies are generally ~+/- 5 C max), it wouldn’t be too hard to catch.
Nice idea. So I ran with it. Don’t know how long this GISS map stays up on their site, but I just did 2009 vs 2008 baseline. The “red” runs up to 10.3 C on the key.
http://data.giss.nasa.gov/work/gistemp/NMAPS/tmp_GHCN_GISS_250km_Anom12_2009_2009_2008_2008/GHCN_GISS_250km_Anom12_2009_2009_2008_2008.gif
So unless we’ve got a 10 C + heat wave compared to last year, well, I think it’s a bug …
I’d suggest poking through the Python version of GISTemp instead of the old FORTRAN mess:
Nope. KEY point about debugging. You always test the code being run. the docs are nice, the pseudo code is nice, even some new translation is nice; but they do not exactly capture all the bugs. So for debugging, it’s the real deal only. (Heck, I’ve even had cases where the written code did something different from the binary – compiler bugs…)
@mosher: Does the bug hit anything other than a graphic drawn on the web page?
The bug is seems to be limited to graphics that meet these two condition:
a) NASA is reporting with a 250km Smoothing Radius
b) NASA is reporting an interval where the reporting period = the baseline period
I can’t recall seeing any such graph.
Have you?
Re: rbroberg (15:07:35) :
How do you know the 9999 is a ‘null value’? Do you have access to the code that generates the maps because I dont believe anybody outside of NASA GISS has access.
What if the code that produces the 9999 is something along the lines of:
if val > MAX_VALUE then val = 9999
instead of
if val == NULL then val = 9999
or
if val == ERROR then val = 9999
Because if its the first then it means the code has the potential to produce artificially high anomalies.
What they leave out is that they’re only counting since 1961! Apparently, 1890 was the end of our warmest decade.
Very important point. Also since we have no known version control and most reports do not list exactly what model version was used to generate the run, you have no way of knowing if the results that appear in a given report, are in fact the real output of the software version that allegedly produced the model run.
In that sense you are chasing ghosts regarding older reports where we have no hope of determining exactly what build the model run was made on.
Along with your note about the compiler errors, it is possible that a given code could run differently on different hardware due to such deeply hidden errors. Unless you can run your code on the NASA system there is no way to exclude that possibility.
Larry
John Whitman (15:46:41) : Do you have a recommendation on what is an effective way for me, “USA Taxpayer”, to contact GISS for the release of the code by and request more open GISS forums for analysis?
Well, they have released some of the code. I’ve got a copy from prior to 15 November 2009 that has fixed 1200 km. I’ve seen one report that the new code they installed then as a RCRIT value that is not hard coded and can be either 1200 km or 256 km (but I’ve not seen the code myself… another thing for the ‘to do’ list… down load the new walnut shell and start looking for where the ‘pea’ is all over again … :-{ so it is possible that the 15 Nov 2009 code is what makes the data that feeds this graph, but then we’re still missing the link from that data to the graph. They have contact information on their web site, FWIW. See:
http://data.giss.nasa.gov/gistemp/
where at the bottom is says Hansen does the ‘scientific’ stuff while Ruedy does the webpages and technical So I’d guess it’s him:
http://www.giss.nasa.gov/staff/rruedy.html
contact info is on his page.
*sigh*
Why does moderation involve a LIFO stack instead of FIFO? While it is mildly amusing to see the order in which comments miraculously appear after a long moderator break, this is more than offset by the irritation at seeing the newest posts appear (and get replied to) before the older longer-waiting ones (like mine), with problems following a thread by having to check backwards from the “newest” to see if others have suddenly appeared since the comments were last checked.
Wouldn’t it be easier for the moderators to go to FIFO, besides being easier for us readers?
[Reply: You are right. But WordPress sets the template, and it doesn’t make starting moderation with FIFO simple. It’s a hassle to go back for as many pages as necessary to start at the oldest post. Sometimes I do it that way, if the posts are on the first Edit page. When that’s the case I’ll start doing them from the bottom up from now on. Just speaking for myself here. ~dbs]
@rbroberg (15:07:35) :
“Smith is graphing a null value and suggesting that NASA preforms the same idiotic procedure that he does.”
NASA pre-forming some procedure – looks just about what they might be doing. Hmm, some PERformance!
(Signed, another grammar n*zi)
“Angelina Jolie and Brad Pitt were even said to be thinking about buying Ethiopia.”
Anomanly? Amanoly? Manomanly? Anonaly?
Nasamoly? IPccpnamoly?
“Dubai’s development has long been criticized by environmental activists, who say the construction of artificial islands hurts coral reefs and even shifts water currents. They point to growing water and power consumption.”
…-
“Is it the end of the world? Nasa picture suggests Dubai globe is sinking back into the sea
By Claire Bates
This is how the world looks like according to ambitious engineers in Dubai, but it is starting to look rather ragged around the edges.
The stunning image of the man-made archipelago was taken by an astronaut far above our Earth on the International Space Station.”
http://www.dailymail.co.uk/sciencetech/article-1247651/World-Islands-Is-end-world-Nasa-picture-suggests-Dubai-globe-sinking-sea.html
Over on chiefio.wordpress.com I’ve added two graphs to the posting. While it already has a November graph in it that has a 4 C to 9.9 C anomaly key (see up top) relative to the default baseline, I’ve added two more specific graphs.
For each of these, the baseline is 1998. Since just about everyone on the planet agrees it was hot (and the warmer camp puts it at record hot) it would be hard for us to be ‘warmer than then’ after 12 years of dropping from what they like to assert is a ‘cherry picked hot year’ for comparison. So, OK, in this case it IS a cherry pick. I’m going out of my way to do the cherry pick of the hottest year with comparable thermometers reporting. We have two graphs. First is the 250 km ‘smoothing’ the second is the 1200 km smoothing.
http://chiefio.files.wordpress.com/2010/01/ghcn12-3_giss_250km_anom12_2009_2009_1998_1998.gif
http://chiefio.files.wordpress.com/2010/01/ghcn8-5_giss_1200km_anom12_2009_2009_1998_1998.gif
I know that those will hang around because I’ve saved them on my blog site.
The first one has a 4-12.3 C ‘hot zone’ while the second mutes that while spreading it around another nearly 1000 km down to “only” 4-8.5 C ‘hot zone’.
So I think this points to the ‘bug’ getting into the ‘non-NULL’ maps. Unless, of course, folks want to explain how it is 10 C or so “hotter” in Alaska, Greenland, and even Iran this year: what ought to be record setting hot compared to 1998…
I’ll leave it for others to dig up the actual Dec 1998 vs 2009 thermometer readings and check the details. I’ve got other things taking my time right now. So this is just a “DIg Here” from me at this point.
But it does look to me like “This bug has legs”…
E.M.Smith (15:56:43) :
“So unless we’ve got a 10 C + heat wave compared to last year, well, I think it’s a bug …”
Wow, you’re fast to come to conclusions. Why don’t you download the numbers behind your graph, since the resolution of the color bar isn’t good enough to tell what the red anomalies are.
Then, since you’re basically comparing Dec 09 to Dec 08, it’ll be extremely easy to check against the station data. Pick a few stations in northern Canada and Greenland, and see how much warmer they were in Dec 09 than Dec 08. Some will be quite a bit warmer, as that is what comes with the extreme AO event this winter: weirdly cold at mid latitude NH, and weirdly warm at high latitude NH.
I bet you that it checks out just fine.
Chiefo,
Thanks for your ideas on how to contact GISS.
John
E.M.Smith (16:48:35) :
You keep digging yourself this hole, because you keep using Dec 2009 without realising that it was weirdly warm in Greenland over this time, just as it was weirdly cold further south.
Repeat your 1998-2009 comparison using June instead of December. Please. You’ll see blue where you have been seeing red.
There is no bug here. Just the Arctic Oscillation event of Dec2009/Jan2010.
So I think this points to the ‘bug’ getting into the ‘non-NULL’ maps. Unless, of course, folks want to explain how it is 10 C or so “hotter” in Alaska, Greenland, and even Iran this year: what ought to be record setting hot compared to 1998…M.
The maps that you generating in the comments are basically a diff between two years. The first one was between 2008 and 2009. The second set between 1998 and 2009.
http://data.giss.nasa.gov/work/gistemp/NMAPS/tmp_GHCN_GISS_250km_Anom12_2009_2009_2008_2008/GHCN_GISS_250km_Anom12_2009_2009_2008_2008.gif
http://chiefio.files.wordpress.com/2010/01/ghcn12-3_giss_250km_anom12_2009_2009_1998_1998.gif
In high latitudes, it may come down to a single station representing an entire grid. And when you look at the difference for that [u]one[/u] station for [u]one month[/u] between just two different years, you can get a lot of variability. I notice that you keep mentioning the red side of the scale, but have you noticed that the left side of the scale reaches even further down than the red scale reaches up? The extreme colds are colder than the extreme hots are warmer?
So if just one grid is 10.3C warmer in Dec 2009 than in Dec 2008 that covers the high end. Likewise, at the low end of the scale, if just one grid is -17.4C cooler that pegs the low end. And it probably is just one grid – one grid with one station that averaged 10C higher in 2009 than in Dec 2008, and one other grid with one other station that averaged 17C lower than Dec 2009 than in Dec 2008.
How does your selected Dec 2009 maps compare with other maps. Well here is NOAA’s [Dec 2009 minus baseline(1961-1990)]. Notice the simliar anomalies. Extreme highs in AK, Newfoundland, and Greenland. Extreme lows in Central Russia. It becomes even more similar when you set the NASA baseline to be the same as the NOAA baseline:
http://tinyurl.com/ydp7zuj
( I hope the tinyurl works, just use the NASA mapper and set the baseline to 1961-1990)
I hate not being able to edit posts. 🙁
Here is the NOAA 2009 anomaly graph
http://www.ncdc.noaa.gov/sotc/get-file.php?report=global&file=map-land-sfc-mntp&year=2009&month=12&ext=gif
carrot eater (16:53:11) :
E.M.Smith (15:56:43) :“So unless we’ve got a 10 C + heat wave compared to last year, well, I think it’s a bug …”
Wow, you’re fast to come to conclusions.
Um “I think it’s a bug” is not a conclusion. “It is definitely a bug” is a conclusion. “I think it’s a bug” is a direction of investigation.
Why don’t you download the numbers …
Then, since you’re basically comparing Dec 09 to Dec 08, it’ll be extremely easy to check against the station data.
Since it is so easy, I look forward to your report after you have done the work. Me? I have dinner to make, a sick cat to tend, Open Office to install on 2 platforms, comments on MY blog to moderate, …
So go right ahead and “dig here”. We’ll all be waiting for your answer.
I bet you that it checks out just fine.
So by your criteria that is “coming to a conclusion”. By mine it is a “statement of direction of investigation”. So which are you doing? Jumping to a conclusion or not? (Best to be symmetrical in how you apply the usage…)
Zeke Hausfather (15:52:25) : “However, the more pernicious accusation that 9999s are bleeding through into the actual temperature data would be much easier to catch in the python implementation unless you really enjoy trying to read through poorly documented Fortran.”
Reminds me of the night I lost a quarter at the corner of 1st & Vine.
.
.
(wait for it)
.
.
I looked for it at 1st & Main because the light was better there. 🙂
It might also be worthing taking a look at the Met HADCRUT3 Dec 2009 anomalies.
http://hadobs.metoffice.com/hadcrut3/diagnostics/monthly/anomaly.png
kadaka (16:24:53) : *sigh*
Why does moderation involve a LIFO stack instead of FIFO?
Because all the topics comments are intertwined in one thread and presented as a LIFO. Either you take time searching for the bottom or just start ‘approving’ as fast as you can. (usually less than a minute or two on my blog). Some things go to the SPAM queue and can take longer. Other times multiple moderators may be working different parts of the queue. Oh, and ‘newby’ moderators will approve the clear cases and leave the ‘unsure’ for experienced moderators, so it isn’t even a LIFO or FIFO… And, personally, I’ll go through and “approve” all the “under a dozen” lines ‘sight read’ comments super fast then go back for the ‘ten minutes for this one long one’ comments as a slow slog. Often LIFO for the short ones, then FIFO for the long ones having then found “the bottom” and working back the other way.
Hey, you asked 😉
@ Zeke Hausfather (15:19:40) :
I owe you a huge debt of gratitude. I’m very good with Python, but completely unversed in Fortran, Linux, etc. environments. If you ever make it out my way, your beer is on me.
Zeke Hausfather (15:19:40) :
Would you provide the data that, besides GISTemp, that shows “high arctic temps”?
Steven Mosher:
The color red is a bug?
Um, no.
It does what it needs to do, namely flag missing regions of the ocean. Would you prefer green perhaps?
carrot eater (17:02:59) : You keep digging yourself this hole, because you keep using Dec 2009 without realising that it was weirdly warm in Greenland over this time, just as it was weirdly cold further south.
It isn’t me doing the digging. I can, if you prefer, use a randomly chosen Jan 09 and have the upper bound be 13 C if you like:
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=02&year1=2009&year2=2009&base1=1998&base2=1998&radius=250&pol=reg
Or APRIL and get a 16.5 upper bound:
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=04&year1=2009&year2=2009&base1=1998&base2=1998&radius=250&pol=reg
Would you like me to go through the whole year until I find the highest value? Or just accept that December is the default and not very different from the others either…
But you just keep on digging, you’re doing fine.
There is no bug here. Just the Arctic Oscillation event of Dec2009/Jan2010.
Oh what a wonderful cliff of conclusion you have found to leap off of. I, however, am going to continue to reserve judgment until there is something knowable. For now it remains a ‘direction of investigation’ and anyone with coding / programming experience will tell you that when you have a clearly broken result, especially at an edge case, one ‘reasonable result’, especially at a non-edge case, does not prove ‘no bug here’.
Some very, very quick observations:
After downloading the Python version of the code, it took me about 2 minutes to locate the error handling for values of “-9999”, which in my experience is more often used as a null value than “9999”. The code knows exactly what to do when it encounters a value of “-9999”. It throws it out. There is also error handling for values that are blank.
The parts of the code I’ve scanned thus far turn Fahrenheit into Celsius, so even if a value of +9999 degrees F slipped through, the actual value the code would spit out would be in the neighborhood of 5377 C. I’m pretty sure someone would notice that.
Back into my secret lab….
amino:
How about RSS?
carrot eater (16:53:11) :
“Pick a few stations in northern Canada…”
I think you mean THE station in northern Canada.
Phil M, I used grep to search for 9999.
I was able to confirm it gets used in numerous places, at least in the CCC.
E.g.
step2.py:23:BAD = 9999
and I confirmed that “BAD” then gets used six times within step2.py
Similarly:
step5.py:279: NOVRLP=20, XBAD=9999):
and XBAD then gets used 17 times in step5.py
I found a few cases where they used -9999 instead, but mostly it was 9999.
Using out-of-band values to signify an error is pretty common in computer programming, by the way.
amino:
“Would you provide the data that, besides GISTemp, that shows “high arctic temps?”
http://upload.wikimedia.org/wikipedia/commons/4/43/Atmospheric_Temperature_Trends%2C_1979-2005.jpg
Phil M,
…and without using Wikipedia.
You know why.
E.M.Smith (17:45:31) :
You’re seriously doing it by the upper bound on the color bar? How about picking a certain location.
Did you try June 2009 vs 1998 yet?
rbroberg (17:05:54) :
Anomaly isn’t reliable when dealing with government data. Anomaly, trend, they don’t tell the story.
It’s far more weighty to look at temperatures.
Comparing actual GISS temperatures to other data sets tells the story.
Phil M (18:16:31) :
What does RSS show?
Phil M (18:16:31) :
What set is that?
Phil M (17:51:17) :
Why are you using python???
Phil M (18:16:31) :
Phil,
I asked for “high arctic temps” not temperature trends from a graph in Wikipedia.
Do you have a temperature set that shows high temps in the Arctic?
Did you see this?
“Arctic temperatures above 80°N are the lowest in six years”
http://wattsupwiththat.com/2010/01/23/arctic-temperatures-above-80%C2%B0n-are-the-lowest-in-six-years/
Carrick (18:14:45) :
Yup, I was so giddy with excitement getting my hands on this program in a language I can work with, I just blurted out the first thing I saw.
I guessed I’m biased with the “-9999”; all the times I’ve worked with satellite/climate data that’s been used as a null value.
aMINO aCIDS iN mETEORITES (18:23:26) :
“Why are you using python???”
Because it’s free and open source and rapidly becoming a common platform for scientific computing.
rbroberg (17:03:55) :
If there is no problem with what James Hansen and Gavin Schmidt are doing with the stations then they should switch from using the stations they kept to using the stations they excluded.
According to your lengthy arguments they will be the same.
Phil M (18:38:38) :
see this above:
steven mosher (15:34:16) :
E.M.Smith (17:06:39) :
“Since it is so easy, I look forward to your report after you have done the work. Me? I have dinner to make, a sick cat to tend, Open Office to install on 2 platforms, comments on MY blog to moderate, …”
In about ten minutes of spot-checking:
First station I tried: Goose, Newfoundland.
http://data.giss.nasa.gov/work/gistemp/STATIONS//tmp.403718160005.2.1/station.txt
is 8.6 C warmer in Dec 09 than Dec 08.
Let’s look for other stations in red splotches in Dec 09, compared to Dec 08
Egesdesminde, Greenland 5.1 C
Fort Chimo, Canada. 10 C
Looks like I found your 10 C difference between Dec 08 and Dec 09. Third station I tried. Hence, the range of the colorbar.
Let’s see what else we find.
Danmarkshavn, Greenland. 2.7 C
Godthab Nuuk: 5 C
Inukjuak Quebec: 6.6 C
Coral Harbor: 8.6 C
So I’ve found a bunch of stations that are between 5 and 10 C warmer in Dec 09 compared to Dec 08.
Do you want me to repeat the exercise for the red splotches in Dec 09 compared to Dec 98, or do you get the idea?
aMINO aCIDS iN mETEORITES (18:34:23) :
“Do you have a temperature set that shows high temps in the Arctic?”
From a previous WUWT post:
http://i43.tinypic.com/1zp1q8j.jpg
And here’s 2009:
http://usera.imagecave.com/climatestuff/2009_full.jpg
E.M.Smith (17:45:31) :
carrot eater (17:02:59) :
There is no bug here. Just the Arctic Oscillation event of Dec2009/Jan2010.
Oh, here’s a nice one. The entire winter of 2006 vs 1998. 13 C ‘hotter’.
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=1203&year1=2006&year2=2006&base1=1998&base2=1998&radius=250&pol=reg
Don’t think that is related to last Decembers AO “event”…
Oh, and I’ll save you some time. 2004, 2005, 2007, and 2008 have very similar patterns.
My GUESS at this point is that: it is much more likely to have ‘data drop outs’ in the dead of arctic winter and that GIStemp is not handling a ‘no data’ case properly. It shows up dramatically when the 1951-1980 baseline is compared to self (so lots of cells with zero anomaly to be handled). But in arctic winter there are similar ‘zero anomaly’ cases (possibly from a NULL vs NULL case as a hypothesis) that bias the result. Again: This is a working hypothesis GUESS for guiding debugging and NOT a conclusion.
The conclusion comes after you recompile the fixed code and the problem is gone AND the code passes the QA suite… and maybe even ‘hand process’ a few grid / boxes just to validate that it’s really doing exactly what it ought to be doing.
BTW, the ‘SH Winter” case does not show as much red at all. Given that GIStemp fills boxes and grids “from the top down” I’d further speculate that the N.H. is more prone to whatever the issue is due to being filled first and with less opportunity to benefit from what has ‘gone before’. (The alternative hypothesis would be that the Antarctic thermometers are less prone to drop outs. Easy cross check on that idea.)
Again: All this is speculative. It is what a programmer dose to focus their first dive into the code looking for what the problem really is. Often it gets you to a fix in minutes instead of hours. When it works, it’s great. When it doesn’t, you move on to the next likely idea.
For me, that ‘next likely idea’ would be the odd way GIStemp manipulates data to make seasonal buckets. If this only shows up in NH Winter (or does so much more strongly then) I’d examine things that are different about the way each season is handled. There is this odd ‘reach back to last year’ to start a new winter that could be ‘the issue’. Especially when stations tend to end at year boundaries. (Don’t know why, but thermometer lives are often bounded by the year end / start.)
This graph:
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=0303&year1=2009&year2=2009&base1=1998&base2=1998&radius=250&pol=reg
of March April May 2009 vs 1998 has a very ruby red Antarctic with that odd little red boxes look and a +7.9 C top.
2005 is similar. Both have a very cold peninsula too. So there are some interesting things to investigate here.
Oddly, 2005 FALL has a ruby red Antarctic and Siberia:
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=0903&year1=2005&year2=2005&base1=1998&base2=1998&radius=250&pol=reg
with a 12.5 C hot anomaly upper bound. 2007 and 2008 are similar. This quarter also butts up against the odd ‘seasonal wrap’ that GIStemp does. So is the problem there? Or is this just an odd coincidence? No way to know. Need to go into the code to find out or need to do hand calculations for some of the boxes for comparison.
But I think it’s pretty clear that when we’ve been steadily and profoundly lower than that 1998 peak, and the graphs say we were burning up hot in comparison, the graphs have issues…
Hadley’s CRUTEM3 uses -99.0 for no data, 241,331 null data points out of 1,476,540 or 16.34% of their monthly “average” temps from 1521 stations. Well Vostok got down to −89.2 °C so they had a 9.8 degree buffer between real-world and NULL data! OBTW I wrote a Perl script that loads the CRUTEM3 dataset into a Postgresql relational database, seems to have worked properly but I haven’t checked well enough to be positive, the gziped sql dump sits at 8.6 MiB, someone expressed an interest in one of these so I figure we could connect if they are still interested.
Carrick (17:36:22) : The color red is a bug?
Um, no.
It does what it needs to do, namely flag missing regions of the ocean. Would you prefer green perhaps?
It’s not the color red that’s the big issue, it is the 9999 C attached to that color… Something is just computing nutty values and running with them.
BTW, the “missing region” flag color is supposed to be grey…
EM Smith:
You do know what out-of-band data signifies, right?
It’s obviously an error code, as a quick perusal of the code reveals.
EM Smith:
I think we can concentrate on something more important than having the same color for ocean and land for missing regions.
Seriously, if I had made that code error, I would have left it this way too.
E.M.Smith (16:48:35) :
This is a fun game, after all. Let’s say I want to find the biggest difference between Dec 09 and Dec 98. There are lots of red splotches on the map, and the colorbar has poor resolution. So I’ll download the gridded data and have a look.
Scrolling past all the 9999s for missing data, and I find that I should be looking at some islands north of Russia. I try some station called Gmo Im.E.T, and I get:
Dec 09 is 12.3 C warmer than Dec 98. First try.
Trev (15:02:27),
CO2 varies a lot. Some folks may argue with Beck’s 90,000+ direct chemical measurements accurate to, IIRC, within ± 2.5%.
http://www.biomind.de/realCO2/realCO2-1.htm
The 1940’s weren’t the only time to have high CO2. In the early 1800’s the same thing occurred. This website is controversial, but it’s hard to argue that scientists with reputations, like J.S. Haldane, would all conspire to misread tens of thousands of chemical CO2 measurements. What for?
The planet emits and absorbs varying amounts of carbon dioxide, far in excess of any puny amount that humans emit, and we still don’t know the exact mechanism.
CO2 doesn’t have nearly the effect on temperature that the AGW crowd has been claiming. CO2 spikes in the early 1800’s and the early to mid 1940’s were not followed by temperature spikes. Therefore, CO2 has little effect if any on the temperature. QED
Carrick (19:20:26) : It’s obviously an error code, as a quick perusal of the code reveals.
That is leaping to a conclusion. I would say it is LIKELY a missing data flag (though it could also be any of several other types of failures). But yes, I would suspect most that a ‘missing data flag’ is not being properly handled and is bleeding through as ‘real data’. In the limit case of “no anomalies” it runs wild and we get the Ruby Red World Oceans. In the non-limit case there is an open issue as to ‘does it bleed or not?’. In either case, it’s a bug.
Carrick (19:21:42) : EM Smith:BTW, the “missing region” flag color is supposed to be grey…
I think we can concentrate on something more important than having the same color for ocean and land for missing regions.
Um, no, we can’t. It means that the “missing regions” code is broken in some way. The code knows to assign ‘grey’ to missing regions and screws it up (badly in some cases). “White” is not ‘missing region’ it is ‘no anomaly’. So we have here a bug announcing it’s presence. We do not know the degree to which it matters. Is it just a ‘color mapping fault’ in the web page? Or is it sporadically averaging a 9999 in with a few dozen boxes of real data and making things very red when they ought not to be? We don’t know. But we DO know it must be looked into if you would trust the code to drive policy decisions.
Seriously, if I had made that code error, I would have left it this way too.
And your code would also be unsuited to policy decisions.
9999 is the equivalent of a FITS file NAN. It means no data exists. In a microprocessor, it is a NOP. No operation. Do not read, nothing to do.
So, the Arctic is a NULL, NAN, NOP, 9999 or -9999 as the case may be.
Now, let’s have some fun.
I will subtract the maps and see what gives.
Be right back.
I just love image processing.
Here’s the #2 map subtracted from the #3 map:
http://www.robertb.darkhorizons.org/ghcn_giss_1200km_anom11_2009_2009_1951_1980.jpg
At least the Arctic is now gray. NAN, NOP, -9999
The Antarctic is still beet red, which is another problem.
I looked at several variations of the same year baseline and in every case I tried where the base period is one year and the time interval is all or part of the same year, then NULL grid points become 9999 and the oceans flow with blood.
I assume this bug is in the code that generates the images, not in the code that sausagifies the station data. In which case it’s an interesting bit of coding error and doesn’t amount to much. Just drop Jim and Reto a note and you’ll get a gracious thank you in reply.
Of course if the error lies deeper in the sausagification routines, then it merits a bit of investigation to root it out.
Earle: That sounds right to me.
Make any map that works. Look at the gridded data. All the grey areas for ‘no data’ have 9999 as the temp anomaly.
Probably the code that makes the images is supposed to turn ‘9999’ into grey, but for some odd reason, it messes up if you choose the baseline to be the same as the observed time interval.
But in any case, it’s quite obvious that there is no leakage of the no-data flag into the real data. Whatever anomalies EM Smith thinks are weird, you can quickly find the corresponding stations. You can find them in almost certainly less time than it took to write the original article.
“Using out-of-band values to signify an error is pretty common in computer programming, by the way”
This may be true in research or academic programming but in my experience it is not true in commercial and business software. NULL is a construct, it has no value, is not aggregated and is not numeric. The only statement that can be made is if a variable is null or not.
Phil M (18:56:09) :
Did you see this?
http://wattsupwiththat.com/2010/01/23/arctic-temperatures-above-80%C2%B0n-are-the-lowest-in-six-years/
REPLY: FYI, true then for the moment, but they’ve already rebounded – A
Again I’ll have to say: the stations that have been excluded from the GISS set should be used and the ones that have been used should be excluded.
This would address some of the doubts.
Apparently there are a few people here arguing there is no difference between the two sets. They are not saying that directly but their arguments say it.
The stations dropped should have the same anomaly and temperature result as the ones used—if all GISS apologists are right they should, shouldn’t they?
“Iren (16:09:25) :
What they leave out is that they’re only counting since 1961! Apparently, 1890 was the end of our warmest decade.”
Because the Austraian BoM uses the global average between 1961 – 1990 in all it’s comparisons. So very plump cherry picking going on at the BoM.
OT, but I cought the tail end of a news cast here. Apparently, two teenagers have been charged with starting one of the major bush fires in Victoria in Jan/Feb 2009. Global warming at work I see.
Patrick Davis (21:09:13) :
The choice of baseline is completely irrelevant to the ranking of years or decades.
And if anybody wanted yet another indication that the 9999 values don’t contaminate anything they shouldn’t: Make one of the messy maps with the time period equal to the baseline, and note that the zonal means are all still 0. No sign of 9999 being used. Nor in a regular map, where there are 9999s behind the greyed out areas.
That bug has been there for at least a couple years. I pointed it out at Climate Audit a couple years ago. I don’t think it bleeds into their temperature anomaly though. Just a problem with their plotting software. Of course it does call into question the quality of their programming.
What is supposed to happen if you have a data file that has a certain flag, like a 9999. And your program, written earlier, expects to find -9999 for no data.
That would be operator error. Oops.
Or the program was updated to handle new No data flag types, but fails to do so. That’s a bug.
But, knowing the shennanigans we’ve seen in the past year, who’s to say someone conveniently edited the source code to make the No data flag type read as data. That’s monkeybusiness.
I wonder which one it could possibly be: oops, bug or monkeyfingers.
Phil M (17:25:37) :
Phil M (17:51:17) :
Phil M (18:36:14) :
Carrick (19:20:26) :
So, what have we found out? The actual GISTemp code is not publicly available. However the Clear Climate Code emulation (i.e. an imitation) does do proper error handling, and looks so good people somehow think they have the real thing, except written in Python.
Thus we have learned an equivalent amount about GISTemp that we could learn about the coding of Microsoft’s Solitaire game(s) from studying a “virtually identical” freeware version, which is basically nothing.
=============
To dbs and E.M. Smith, thanks for the replies on the moderation issue.
Mr. Smith, extra thanks for your work on the black box reverse engineering. Maybe some day we’ll get a successful FOIA request on the actual code, although I’d put better odds on another “helpful hacker” releasing it first. Until then this will do, and it is quite revealing and much appreciated.
“carrot eater (21:43:48) :
The choice of baseline is completely irrelevant to the ranking of years or decades.”
Yes I totally agree the 1961 – 1990 global average used as a yardstick by almost all official weather services around the world is completely irrelevant.
kadaka:
What do you mean?
The GISTemp source is here.
Ouch. Looks like very dirty programming, I’m suspecting that they’re filtering out the null values (999.9 in the GISS data files) AFTER they’ve done the actual computation, but that their way of doing this breaks down when the only available values are 0 and 999.9.
Still, it doesn’t seem to me that it really affects the non-pathological cases. The Arctic WAS unusually warm in December (e.g. Svalbard was much milder than southern Norway most of the month). But one thing to keep in mind, is that the unusually “warm” weather in the Arctic really wasn’t *warm*. Does it really matter whether the temperature at the north pole is -35 or -45? It will be interesting to follow the global temperatures in the next few months. I think the harsh winter in Europe, Siberia, China, the US, etc. may be followed by a slower than usual spring, because of the snow albedo and the frozen ground.
Back to GISS: One thing which puzzles me, is the apparent special treatment they give South Pole measurements. Look at the big orange circle here: http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=1212&year1=2009&year2=2009&base1=1951&base2=1980&radius=250&pol=pol
I asked for 250 km smoothing, and they did it, except around the south pole where there seems to be a circle of 1000 km radius? Why?
Should be possible to test to see if the blood is bleeding through. Set up a program to change all non-9999 values to the value giving white, and replot. I can see the logic, would do it if I had the expertise, but it’s not my job at this point…
Carrot Eater, Phil M and Carrick. While you are visiting from RC, don’t forget to go to the other threads and offer your opinions – for instance you could tell us why the IPCC chairman’s porn novel doesn’t further discredit him.
Or why Phil Jones’ missing Chinese stations are not an issue.
Or why cleaning your boots when visiting Port Lockroy is relevant to global warming.
Not forgetting those “melting” glaciers too, of course.
I haven’t seen you on those threads yet.
Too busy at Wikipedia?
Correction: “Phil M” turns up at porngate to admonish Anthony for not blocking a Google advert for something called “50 Sexiest Women of Science Fiction”. Well done Phil! In true warmist fashion, even when talking about the IPCC porn novel, he offers no defence for his spiritual leader, he just attacks Anthony about something else.
Wow. This is curious.
See this map.
Data Sources:
Land: unadjusted 1880-1999
Ocean: none
Map Type: Anomalies
Mean Period: Annual (Dec-Nov)
Time Interval: Begin 1951 — End 1980
Base Period: Begin 1951 — End 1980
Smoothing Radius: 1200 km
Projection type: polar
Now to me, studying the downloaded pdf, in the right-side Southern Hemisphere view, between 90 and 180 deg W longitude there is a large block of yellow (.2 to .5) in the Pacific Ocean. Note the sharp termination line, the protractor says it is at 179 deg W (181 E). Note also that there is no yellow in the NH map, only the SH. The block of yellow also terminates right at the equator.
Also, there appears to be only three colors, the yellow, red (2 to 4), and dark red (4 to 9999). On 400% zoom there is a noticeable small patch of white (-.2 to .2? Left edge not marked, -.2 on regular projection) in the SH at about 85 deg E near the equator, which would be about 10 deg east of and south of the tip of India.
Switching back to regular projection, there is a lot of white, dark red, and no yellow. And something else that’s curious. The SH white patch? On regular projection it looks diamond shaped but dark red, same shape seen on the 400% zoom of the SH polar projection in white.
Changing the time intervals to 1970-1999 with polar projection, looks the same. Only 1970 for all year fields, same.
Experimenting further, I tried the other end, using 1880 to 1910. Which generated the following error message: I cannot construct a plot from your input. Time interval must begin in 1880 or later. Yup, it thinks “1880” is not 1880 or later.
So, switching to 1881 to 1910, some different colors, but still the curious cooler section that terminates along 179 W and the equator. It is very noticeable how at 179 W one clearly-defined region suddenly transitions from dark orange (1-2) to white (-.2-.2?) while another goes yellow (.2-.5) to white.
Checking by single years to see when the diamond first showed up, I noticed for 1953 the 179 W transition was in the NH but not the SH, an odd flip. Looking back, the flip is in 1952. For 1951, there is the 179 W transition in the NH, but now in the SH the transition is reversed, marking a change to warmer instead of cooler.
At 1950 the 179 W transition is only in the SH.
The off-India diamond appears to first show up in 1954, with the linked polar projection showing a dramatic NH to SH color difference. 179 W transition is only in SH, again.
All this clearly shows to me there is something wrong.
I could accept the problem lies in the bit of programming that draws the polar projection maps. Except the 179 W artifact is not consistent. From 1951 to 1953 inclusive it was showing in the NH, with 1951 also showing in the SH but with a reversed effect. That indicates to me two things. One, the code that makes the polar projection maps is showing me something in the data, it’s not all in the map drawing. I cannot imagine how there could be an error in that code that decides to single out those three years for special treatment. Two, around the early 1950’s some sloppy adjustments were made to the data, with 1951 particularly messed up. It might be in the code that feeds data to the map drawing program, with encoded adjustments for those specific years, but it also is highly possible the changes are in the “unadjusted” data.
I would also like to know why the off-India diamond appeared in 1954.
Offhand I don’t think many people look at the polar projections, at least this way. To me it looks blatantly obvious something is screwed up, I’m surprised it didn’t somehow show up before.
Although I’m even more surprised that the GISTemp mapper doesn’t recognize 1880 as being 1880 or later. Come on, that is absolutely lame. Whomever missed that deserves a career encoding form layouts in COBOL.
In closing, I will simply note what is says on the bottom of the relevant web pages:
Responsible NASA Official: James E. Hansen
Patrick Davis (22:45:37) :
Please explain why you think the choice of baselines is important. It is not. All it does is change the absolute values of the anomalies. The trends, which is what’s important, do not change as you change the baseline.
kadaka (22:35:12) :
This is silly. All you don’t have is the code behind the online map plotter for the website.
aMINO aCIDS iN mETEORITES (20:44:16) :
“Did you see this?
http://wattsupwiththat.com/2010/01/23/arctic-temperatures-above-80%C2%B0n-are-the-lowest-in-six-years/
REPLY: FYI, true then for the moment, but they’ve already rebounded – A”
I did see it, but as Anthony demonstrated, that was weather, not climate. A clear example of why one should not eagerly await monthly satellite anomolies and use that as proof of a trend. Or point to a single extreme weather event,”hot” or “cold”, and attempt to infer something about a climate trend.
Sorry – OT but unable to post on Tips and Notes for some reason.
One for the weather not climate department?
Met Office confirms Scotland had coldest December + January for almost 100 years:
http://news.bbc.co.uk/1/hi/scotland/8492333.stm
Note – this does not mean coldest winter, which is Dec + Jan + Feb.
Here’s some of the documentation from Python version of the code (step2.py). I have not gone through the code line by line to verify that it does what it says, but the documentation appearing at the beginning of each class “def”inition follows the standard Python conventions. Also, it appears the translation of the code from Fortran (or whatever) to Python was done by an outside consultant. Everthing that follows is taken directly from the code; the last class is for all the consipiracy theorists. 🙂
def urban_adjustments(anomaly_stream):
“””Takes an iterator of station annual anomaly records (*dict*,
*series*) and produces an iterator of urban adjustment parameters
records. The urban adjustment parameters describe a two-part
linear fit to the difference in annual anomalies between an urban
station and the set of nearby rural stations.
The algorithm is essentially as follows:
For each urban station:
1. Find all the rural stations within a fixed radius;
2. Combine the annual anomaly series for those rural stations, in
order of valid-data count;
3. Calculate a two-part linear fit for the difference between
the urban annual anomalies and this combined rural annual anomaly;
4. Yield the parameters of this linear fit.
If there are not enough rural stations, or the combined rural
record does not have enough overlap with the urban record, try
a second time for this urban station, with a larger radius.
If there is still not enough data, do not produce an
adjustment record.
“””
def get_neighbours(us, rural_stations, radius):
“””Returns a list of the stations in *rural_stations* which are
within distance *radius* of the urban station *us*. Each rural
station returned is given a ‘weight’ slot representing its
distance fromn the urban station.
“””
def apply_adjustments(stream, adjustments):
“””Applies the urban adjustment records from the iterator
*adjustments* to the station records in *stream*. Returns an
iterator of adjusted station records.
Rural stations are passed unchanged. Urban stations without an
adjustment record are discarded. Urban stations with an
adjustment record are adjusted accordingly, to remove the modelled
urban-heat-island effect.
An urban adjustment record describes linear and two-part linear
fits to the difference between an urban station annual anomaly
series and the combination of annual anomaly series for nearby
rural stations. The linear fit is to allow for a linear urban
heat-island effect at the urban station. The two-part linear fit
is to allow for a model of urbanization starting at some point
during the time series.
“””
def flags(fit, iy1, iy2):
“””Calculates flags concerning a two-part linear fit. Also
accumulate statistics. Not well documented or understood, but
note that in adj(), below, the two-part fit will be disregarded if
the flag value is not either 0 or 100.
“””
aMINO aCIDS iN mETEORITES (17:26:48) :
Would you provide the data that, besides GISTemp, that shows “high arctic temps”?
Will the UAH satellite record work for you?
http://vortex.nsstc.uah.edu/public/msu/t2lt/uahncdc.lt
Arctic temps can be found under ‘NoPol’. The trend for that region is at the bottom, in case you’re interested in that too.
apparently the “problem” is fixed…and just in the last 10 minutes
@ Carrick (23:49:15) :
and
carrot eater (04:01:16) :
GISTemp updates:
Example 1:
Example 2:
Example 3:
Three different times, they changed THEIR codes, THEN changed the publicly-available source codes. If what was released was THE real GISTemp they themselves were running, then they would only have to upload the new code for distribution, there would be no need for other modifications.
November 13 2009 may be another example, but there they said
thus it does not outright say the public version was modified separately. There may be other good examples hidden among the careful wording of the updates, but I didn’t meticulously scrutinize all the change notes.
barry (05:24:27) :
These are not high
to say they are high is an exaggeration
Phil M (05:04:13) :
funny, you guys keep going back to python
no surprise
Phil M (04:47:48) :
but weather actually is climate
the cooling of the last 5 years in the earth is why ‘weather’ is happening across the earth
warming is not happening
Steve M. (07:22:30) :
So it is; that was fast by GISS. I wonder how many people tried to make the weird maps over the last day.
kadaka (07:26:28) :
Give me a break. If you’re that determined to believe that the publicly available source code is different from what they run, then download that code, run it, and see if you can find any differences between its results and the published results.
Steve M. (07:22:30) :
apparently the “problem” is fixed…and just in the last 10 minutes
Yeah, isn’t the power of WUWT and the Chiefio amazing?
That’s why for my long post (03:41:38) I saved the pdf’s of the regular and polar projection maps as well as the text file versions of the gridded data for all the examples I linked to. If someone connected to either site wants to host them, I can email the data. Alternately if you know of a free hosting site, I could dump them there and post the links here.
PS, it still won’t take 1880 as a start year.
@ carrot eater (07:59:14) :
If it looks the same and gives the same results it must be the exact same thing?
Have you ever sold Rolex watches on a streetcorner?
Carrick (17:36:22) :
“The color red is a bug?
Um, no.
It does what it needs to do, namely flag missing regions of the ocean. Would you prefer green perhaps?”
Ah, when the usual color for MISSING DATA is gray and when Red is used to denote a high anomaly, then ya it is a bug.
Is it of any consequence? I dunno. I’ve not seen it in any papers.
It is kinda funny though. I really respect the clearclimatecode guys. They find little bugs. they report them. NASA fixes them. Its really quite simple.
So, rather than everybody spending a bunch of time trying to figure out this “bug” ( scare quotes mean I accept your objection) somebody should just drop Reudy a note.
Crap, even I cant take my own advice and have spent more time telling people what to do than actually doing it.
E.M. Smith I would suspect most that a ‘missing data flag’ is not being properly handled and is bleeding through as ‘real data’.
It is is difficult to understand why you stubbornly cling to your suspicion (if you still do). It is obvious that the high and low end of the scales in the 2009 – 2008 diff as well as the 2009 – 1998 diff come from specific stations.
So to make it as obvious as possible. The 2009-2008 diff map shows a lower bounds of -17.4C and an upper bounds of 10.4C (anomalies). The following data comes from GHCN RAW (v2.mean) which, as you know, is the major input into GISTEMP. Remember, it is not that any station will set an endpoint. But a remote station that is the only one its grid and more than 250km from a neighbor can set an extreme by itself. I have not verified that the following indeed are a) the only station in its grid and b) more than 250km from its neighbors, but this theory is far more likely than your bleed over theory (which doesn’t explain the cold end points)
Here is the first map you selected (2009-2008, Dec) and asked for an explanation of the high end of the scale (ignoring the low end):
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=12&year1=2009&year2=2009&base1=2008&base2=2008&radius=250&pol=reg
Here is the GHCN data explaining the endpoints of the scale (-17.7, +10.3)
SALEHARD stnid 22223330000
Dec 2009: -30.3
Dec 2008: -12.9
Diff: -17.4
DAWSON, Y.T stnid 40371966000
Dec 2009: -19.3
Dec 2008: -29.6
Diff: +10.3
Here is another map you selected (2009-1998, Dec) and asked for an explanation of the high end of the scale (ignoring the low end):
http://chiefio.files.wordpress.com/2010/01/ghcn12-3_giss_250km_anom12_2009_2009_1998_1998.gif
Here is the GHCN data explaining the endpoints of the scale. (-17.7, +12.3)
TURUHANSK stnid 22223472000
Dec 2009: -39.1
Dec 1998: -22.0
Diff: -17.1
GMO IM.E.T. stnid 22220046000
Dec 2009: -15.7
Dec 1998: -28.0
Diff: +12.3
Your April 2008 map will require a little more work. The low end of the anomaly (-8.9C) comes from GHCN station 70089055000 CMS”VICE.DO.M. But the high end anomaly (+16.3) is in the Antarctic and GISTEMP uses the SCAR data set for Antarctica.
http://www.antarctica.ac.uk/met/READER/surface/stationpt.html
http://www.antarctica.ac.uk/met/READER/temperature.html
Steve M. (07:22:30) :
Error
I cannot construct a plot from your input. Anomalies for any period with respect to itself is 0 by definition – no map needed
apparently the “problem” is fixed…and just in the last 10 minutes”
Oh great. Well if there was a problem and it was pointed out by someone them I hope they did the right thing by putting up a change notice, thanking the person who pointed it out. Also, you can expect people to now start wondering if this “bug” ( deference to carrick) has had any impact that is now hidden.
ps.. Python is cool
http://arstechnica.com/open-source/news/2008/01/openmoko-freerunner-a-win-for-phone-freedom.ars
Arrg. ok the dream behind that phone was cool.
@ Steve M. (07:22:30) :
Hey look! Mapping an 1881-1882 interval with an 1881-1883 base still shows the 179 deg W artifact in SH polar. So does 1881-1920 with 1881-1921. Likewise 1920/22, 1920/30. 1940-61 with 1940-1962 shows it very well. Seems like any periods similar enough to practically cancel out will yield the artifact.
Yup, it’s still in there.
rbroberg (08:28:05) : Looks like you one-upped me by finding the lowest anomalies as well. But I’m pleased that we both honed in on GMO IM.E.T. for the Dec 09 to Dec 98 difference.
Seriously, it took me all of five minutes to find that station.
@E.M. Smith:
Hope your cat is feeling better.
@carrot eater: Looks like you one-upped me by finding the lowest anomalies as well.
Yeah. The idea of looking at the data first seems obvious once you take off the ‘its all a fraud’ hat. And EM’s insistence of looking at only one end of the scale seems to reveal a certain explanatory bias as well.
aMINO aCIDS iN mETEORITES (18:45:04) :
“rbroberg (17:03:55) :
If there is no problem with what James Hansen and Gavin Schmidt are doing with the stations then they should switch from using the stations they kept to using the stations they excluded.
According to your lengthy arguments they will be the same.”
That is the best argument I have heard in a long time. If there was no bias or “cherry picking” involved the excluded RAW data should show a warming trend also when compared to the same baseline as the data that was keep.
“And EM’s insistence of looking at only one end of the scale seems to reveal a certain explanatory bias as well.”
I think this is a very important thing to notice.
Gail Combs (10:54:56) :
“If there was no bias or “cherry picking” involved the excluded RAW data should show a warming trend also when compared to the same baseline as the data that was keep.”
Somebody would have to repeat the effort that was made when the GHCN was started, and track down, collect and organise all the unreported data since 1990. I do think it’s time for that to happen, but it was a big effort. That said, if somebody wanted to spot-check a certain region, they could probably gather that data from SYNOP reports and check it out.
In the meantime, the other obvious thing to do is to simply build one record from stations that continued, and a separate record from stations that stopped. That can be fairly easily done, and has indeed been done.
http://www.yaleclimatemediaforum.org/2010/01/kusi-noaa-nasa/
Second figure; the analysis would be improved by taking the spatial mean, but it’s a start. Before the terminated stations dropped off, they were doing about the same thing as the stations that continued.
Simply saying that something might have had an effect is not that great a contribution. One has to go into the data and actually see if it did have an effect.
Red is used to show hot areas.
Using red to show areas with no data makes it look like there are more hot areas.
It is just “Jedi” mind tricks to bias the graphic.
Move along – nothing to see here.
after reading some of the commenters here (apparently they’re from RealClimate) I feel like I need a shower
carrot eater (12:12:30) :
This is what you said: blah, blah, blah, blah
” they were doing about the same thing as the stations that continued.”
The words “about the same thing” are relative. Relative terms are not allowed in science.
What is really happening is fudging to get a desired result.
Or is it sophistry to say “about the same thing”?
Who knows, because it’s all relative.
Harold Blue Tooth (14:01:37) :
If you want something more quantitative, then repeat the exercise, calculate the spatial means, and compute the difference in trends between continuous and terminated stations. And there you’ll have it.
If you are truly interested in this issue of station drop-off, you should be requesting the people leading that discussion to do that very analysis.
Alternatively, you could go back to the 1980s, use non-Bolivian stations to calculate the estimated temperature at some grid point within Bolivia, and then check against the actual observations. That’ll give you an idea of how well the interpolation works. This could probably be done in about a day, if somebody were interested in the matter.
OT, but back in 2006, when the election in Mexico happened, I had reason to be interested. I was able to watch the latter part of the results coming in. This was a supreemely close election, with the eventual official margin being 0.58%. Lopez Obrador had a led of about 2% when I started keeping track, with about 81% of the vote in. I refreshed the counter every 2-5 minutes. In every single refresh, the margin moved in favor of Calderon. I could not see what precincts or whatever, but the increases on both sides each time were VERY small. Yet in this election decided by barely 1/2%, every single small update was in favor of Calderon. At least while I was watching, which we pretty much right up until the end. For the final 19% of the vote (out of 40 million votes), Lopez Obrador was barely edged out, so that his 2% lead became a loss. That not ONE of those refreshes showed a gain in his favor, to me, that reeked of the fix being in, someone tampering with the results. I refreshed probably 100 times.
In a 50-50 vote, that is like flipping a coin 100 times and every time it came up heads. What is 2^100th power?…lol
When I read that every error was one way, I know it is like those coin flips. 2 to the nth power is a big number, after all.
Peter –
I DID see in the HADCRU emails that they DO use 9999 and 999 as values for missing data. It is in several places, and specifically in HARRY_READ_ME.txt.
As to whether GISS is using it, I can’t say, but it is IN the data. And CRU shared with GISS and NOAA.
carrot eater (14:36:35) :
don’t need all that
I want James Hansen to use the stations he’s been dropping and drop the stations he’s been using—and completely open up what he is doing to everyone.
That’s a more accurate way. No fudging in the stations he’s been using in the picture then.
From the “Sweep the dirt under the rug” department. If you now try to do a “self vs self” anomaly map you get this message:
Surface Temperature Analysis: Maps
Simple reminder
Anomalies for any period with respect to itself is 0 by definition – no map needed
Guess it was easier to hide it than to actually fix the code. A reasonable short term HACK but hardly good QA.
“carrot eater (04:01:16) :
Please explain why you think the choice of baselines is important. It is not. All it does is change the absolute values of the anomalies. The trends, which is what’s important, do not change as you change the baseline.”
The 1961 – 1990 “baseline” is cherry picked and not nearly wide enough. Trends would be important if one did not remove 75% of the recording devices nor “add vaule” to the raw data, and then lose the raw data.
rbroberg (09:48:21) :
@E.M. Smith:
Hope your cat is feeling better.
The cat passed away about 11am.
And EM’s insistence of looking at only one end of the scale seems to reveal a certain explanatory bias as well.
Pardon? I’m exploring ALL ends of the scale. This was an unexpected excursion on my way to benchmarking the baseline:
http://chiefio.wordpress.com/2010/02/02/giss-benchmarking-the-baseline/
I’d fully expected to get a white land / grey ocean map. That is what ought to have happened. To ignore an obvious bug in their web based maps would be to put the entire benchmark in doubt.
Need I point out a line of text from my comments earlier:
So we have here a bug announcing it’s presence. We do not know the degree to which it matters. Is it just a ‘color mapping fault’ in the web page? Or is it sporadically averaging a 9999 in with a few dozen boxes of real data and making things very red when they ought not to be? We don’t know. But we DO know it must be looked into if you would trust the code to drive policy decisions.
Hardly a statement of expectation of any given outcome.
And if you bother to look into it, you will find that “edge cases” are the most important ones in debugging and QA. I always test the zero and infinity cases if at all possible. And I usually do start at zero since the infinity case often does not exist. So you start at zero and work your way up. In this case, zero is ‘baseline against baseline’.
You will also find that if you want to test for the presence of bugs you must behave as though you expect them to be there. This is the most common failure of programmers in not catching bugs (and why the folks who do QA for a living are often specialists at it). You simple must look for broken things if would ever find them. That does not mean you want them.
I managed the QA department for a C compiler tool chain for a few release cycles. The “can’t ever happen” cases in the QA suite were a large percentage (and the frequency with which they caught bugs shows their worth). You just never ignore a bug until it’s investigated and found benign; and you just never assume the code is good. It isn’t. There are always bugs. The purpose is to make sure none of them are really “material” and stamp out any that are.
Sidebar: On one occasion, a programmer had finished a very nice input screen that carefully checked for out of bounds data and many other cases. The manager walked up, looked at the very pretty screen layout, read the “spec”. And proceeded to reach over and “mash” the keyboard. (He was simulating someone leaning on the keyboard with an arm/ elbow). The “system” crashed. Took me quite a while to figure out what very unexpected characters had caused the crash… but the final product DID pass QA (even the “key mash”).
That does not mean folks expect to do key mashing all day long. It means that that is how you find bugs and how you make “robust” code.
Not by hiding your bugs and not by assuming they do not matter.
@ E.M.Smith (17:15:16) :
I am sorry to hear about your cat. I lost one last winter. Such losses can be hard.
I agree with you about testing boundary conditions. No argument there.
I agree there was a display bug and that it now appears to be corrected.
I disagreed with your claim that the display bug for 9999 was ” bleeding through as ‘real data’.
You asked for an explanation of the +10C and higher anomalies when diffing 2009-2008 and 2009-1998.
I demonstrated that the endpoints for many of your maps, both high and low, were obviously matching the monthly averages for isolated high latitude stations.
But I haven’t seen you yet disclaim or any in way moderate your position that the ‘display bug’ was ‘bleeding through to the “real data.”‘
Do you still hold that view?
Edit for above:
I wrote: I disagreed with your claim that the display bug for 9999 was ” bleeding through as ‘real data’.
I should have written” I disagreed with your suspicion that the display bug for 9999 was ” bleeding through as ‘real data’.
An interesting observation was made by “boballab” here:
http://boballab.wordpress.com/2010/02/02/blood-red-oceans/
He noticed that the upper right hand corner of the graph gives the average anomaly for the map. For example, the present Dec 09 anomaly is 0.67 in that map. On the “Blood red oceans” one at the top, the average anomaly is given as 5344.1 C so we now know that the Bogus Nines are used in some calculations.
We still don’t know if this is just in the creation of the web page or not. (I could see a case where it was generated by the graphing package, but that would be a bit odd given that the GIStemp software also calculates a value, but the GISS code has done odd things before. If they do it both ways they could at least print them both as a crossfoot validation. A “check if equal” for those two values in the display code would have flagged this some time back, assuming it is in the graphing part).
BTW, to the folks posting that the GIStemp code is available for download at the GISS link: As I’ve pointed out before, that is SOME code, but not THIS code. There is no graphics presentation module in the code (unless they added it on at the last update) and you don’t get to play with all the knobs…
Unfortunately, with the only visibility of this bug now swept under the rug, we may never find out what it really is.
Espen (02:14:49) :
I asked for 250 km smoothing, and they did it, except around the south pole where there seems to be a circle of 1000 km radius? Why?
Fascinating… I would GUESS (in all caps since some folks here don’t seem to see words like “speculate”, “suspect”, “expect”, “line of investigation” etc. unless really really loud…) that the way GIStemp calculates boxes is from the “top down” and the end ‘box’ may be an ‘everything left over’ that happens a bit too soon. If I get a chance I’ll look at the code that is published and see if it does that or if this is another one that is ‘between the code and the screen’…
On the question of “does the baseline matter” the benchmark I did of the baseline says “yes it does”. How much is for another more detailed study, but the benchmark showed that the baseline has features (very unusual features like a hot/cold antarctic pair) that “bleed through” into the present year reports. I’ll have to add a “1971-1990” decade graph to the presentation for the non-GIStemp folks…
kadaka (03:41:38) : Experimenting further, I tried the other end, using 1880 to 1910. Which generated the following error message: I cannot construct a plot from your input. Time interval must begin in 1880 or later. Yup, it thinks “1880″ is not 1880 or later.
Most likely you are set to the “year” that wraps from Nov to Oct (the default) rather than the Jan to Dec that is at the very top of the list. So the code is trying to use 1879 Nov & Dec to fill out the “year” that is mostly in 1880. Yeah, a ‘human factors’ issue in the interface design. But if you go out of your way to choose the J-D year it will work.
rbroberg (08:28:05) : It is is difficult to understand why you stubbornly cling to your suspicion (if you still do).
A few decades of debugging code and running a code QA department and being Build Master for another product (web based integrated email, web, etc. server w/ DHCP services and routing). After a while you learn that you NEVER know what the bug really is until the code is FIXED and passes QA. (And even then you keep your ‘suspects’ list in case the bug comes back…)
Dismissing “suspects” too early is a major reason for not finding a bug. Yeah, you rank them in probability order. Yeah, you take the most likely (and sometimes the easiest to prove / disprove) and whack on them first. But you NEVER ‘dismiss’ a suspicion until the Fat QA Suite sings… Too many times thinking “This time for sure!”… and watching everyone else do the same thing.
So as of now my “suspects” list would be headed by a “web graphics” package that we can’t look at. But everything ELSE stays on the list until the end. (And yes, ranked from highest to: “you’ve got to be kidding, no way”… but it stays on the suspect list.)
SIDEBAR: I once was writing a cost accounting package. Tests fine. Add new FUNCTION and function call. Old code breaks. Hours later, change old code name from FOO to FOONEW preparing to try new idea. Code works… Leave it alone, go back to New Function BAR. Add a few lines. FOONEW breaks. Change name back to FOO. It works… Called the vendor. Compiler bug made their compiler sensitive to names of functions sometimes…
SIDEBAR2: I once wrote code of the form:
If A goto ALAB
if NOT-A goto WAYDown
Print “You can’t get here – Compiler error”
and had it print …
So after a few of those, you stop assuming that you EVER know you can dismiss out of hand anything as ‘not the problem’. That, and running QA on a compiler tool chain for a while 😉
Further, it is a bit of the cops dictum. If you suspect and look, you may catch the problem. If you look and all is well, fine. But if you never suspect, you will never look, and you will never catch the problem. So suspect first, then look. Then keep suspecting until the court closes the case. And maybe keep an eye on the guy even if the case gets dismissed.
(Yeah, I’ve hung out with cops a lot of my life and had some amount of ‘police training’ – directing cars is fun, marching in formation not so much, when to draw your gun and fire is spooky [ everyone killed a couple of innocent people in the film when they did ‘stupid but innocent’ things.] Film was designed to teach you that. To teach you to suspect always, but leap to conclusions never.)
It has served me well over the years. I find bugs others miss. I spend a bit more time investigating some dead ends some times, but more often get to the ‘real problem’ faster than the ‘one line fix’ guys. I also tend to write really solid code.
Basically, you have a search tree. Don’t prune it too fast and don’t erase it when you take a path. Keep an eye on the other paths and be ready to jump there if some new data shows up. But don’t erase that other path…
FWIW I have the same “suspicion” about every bit of code I write.
I think it is more solid than most, but at the drop of a hat I’d start tearing it apart if it did something even a little bit unexpected.
So evidence is FINE and it helps change the weighting on the search tree, but it does not erase a branch. Just puts a toll booth on it 😉
Everyone that is saying that ‘silly’ ‘extreme’ data values are used to report errors could not be more wrong. Or at least, has been wrong since about 1990 when professional programmers started taking Y2K seriously. All professional programmers know not to pass errors or messages in data. Ergo, this software is either ancient or written by incompetents.
“George (23:50:00) :
Everyone that is saying that ’silly’ ‘extreme’ data values are used to report errors could not be more wrong. Or at least, has been wrong since about 1990 when professional programmers started taking Y2K seriously. All professional programmers know not to pass errors or messages in data. Ergo, this software is either ancient or written by incompetents.”
What was serious about Y2K? I mean, countries like Romania (With nuclear power stations) and Ethipoia (Ok to be fair Ethiopia had their Y2K in 2007 in our callendar) didn’t have the resources to deal with the “problem”, which didn’t happen (In countries which did not “invest” in the Y2K “problem”).
Well, some people made lot of money out of that “scare”.
George:
I must not be a professional programmer.
#define EOF -1 int c; while ((c = getchar()) != EOF) { // do something }That Richtie guy was a real slacker, wasn’t he?
The Y2K problem had nothing to do with passing out of band data, by the way. It was a schlock mistake that would have gotten points taken off if it had appeared in first semester programming code. I know this because I was taking points off students projects for errors like this well before the Y2K scare.
Actually I’ve always taken Y2K as a model for how people come to exaggerate climate change impact. The worst scientists hide out in the climate change impact area (the good ones are in the physical science part which is mostly unrelated). Same goes for Y2K, I turned down a great paying job involving Y2K. So did most other competent people I know.
There’s an announcement at CCC related to this:
Science advances in odd ways.
Score another victory for open source!
Has anyone suspected a “ghost in the machine”?????? Like Hitler’s ghost. I have see many movies where Hitler is in one of his rages going “9999”. Any correlation?????
This is serious!!!
George says:
No.
Firstly, this software is old, but not really ancient. Most of it was written in the 1980s or early 1990s.
Secondly, this software was written by scientists, not by professional programmers. This is the sort of software that scientists write. That does not make them incompetents.
Thirdly, in most scientific datasets, whether or not they are produced by software written by professional programmers, you will find missing data identified by specific in-band values, often as +/-10^n -10^m (that is, numbers such as 9999, 999.9, -9.999, 0.99999, etc). It’s just the way it is done in that field. It has various advantages, including simple and compact file formats and the ability to read in a huge array of floats in one line of Fortran. It’s a tradition going back several decades, to really ancient software.
In ccc-gistemp, we have a notion to switch eventually from the current missing data values (for which we follow the Fortran in using the variable name BAD) to an IEEE floating-point error value, possibly a non-signalling NaN value. This will allow us to retain the advantages of using a single-typed array while also getting the infectious properties of NaNs (any operation with a Nan operand yields a NaN result).
Any of E.M.Smith’s software war stories, I can match. That’s what you get for working on compiler and runtime projects for years. The garbage collector bugs were the worst (and were, specifically, my job to fix).
Lastly, I’d like to reiterate the standing invitation to any programmer who wants to actually contribute, rather than simply sniping, to come over to the ccc-gistemp project and write some code.
Incidentally, the do_GISS_analysis driver script, available from the GISS website, answers the Sooper Sekrit Mystery about where the 250-km data comes from:
# do the gridding
cd ../../STEP3
do_comb_step3.sh 250
do_comb_step3.sh 1200
That was hard.
The graphics, I think, are produced by NCAR graphics. Probably driven by the STEP6 code, which I haven’t looked at (because none of these maps are actual science products).
The most basic task regarding that dear selled GISS project does not consist in engaging a bunch of (unpayed) volunteers, mandated to shift Dr. Hansen’s convoluted code base from a typed language like Fortran to an interpreted and essentially untyped one like Python, but in putting up a well designed database of raw temperatures, which also provides station meta-data and a detailed history record for each and every imported value, much like it is standard for databases supporting security, medical, pharmaceutical, financial or forensic applications.
Regarding your CCC project, I see that even Python may be used to parse flat files. While this is a rather small step for the average programmer, it may be considered to be a big step by everyone else. However, in what respect is it appropriate to eulogize such a filiopietistic effort as “another victory for open source!”? The absolute progress here is more like Lenin once summarized the attitude of the Menshevik fraction: one step forward, two steps back.
You might like to work more on sentence construction and less on ten-shilling words. If you think ccc-gistemp is filiopietistic, you either don’t understand the project or don’t know the meaning of the term.
Which “dear selled GISS project”? Glory? ISCCP? What? “dear selled” how?
Your first sentence boils down to “The most basic task … consists in putting up a database of raw temperatures ….” Fine. If you think that is important, you go ahead and do it. I would have thought that was rather outside the GISS remit, and more a job for organisations like NOAA and the WMO, both of whom seem to do a reasonable job.
For myself, I think that GISTEMP is important, because of the prominence of its results, and latterly of the code itself, in the public discourse. I also think that code clarity is important. So that’s what I choose to work on, together with the people who have joined the project. If you don’t think GISTEMP is important, why are you commenting on this thread?
aMINO aCIDS iN mETEORITES (16:32:15) :
My post directly addressed what you want to get at. Using existing data, just run an analysis of stations that were dropped, vs the ones that were not. Zeke Hausfether’s already done a preliminary version of that, just without spatial averaging. But even then, it looks like the dropped stations behave just as the not-dropped ones.
As for Hansen dropping things: he isn’t. GISS takes the station data from the NOAA, who get it from the from the individual countries. Stations only appear to drop off because those stations aren’t continuously reported by the individual countries in the monthly format.
Patrick Davis (16:48:23) :
This doesn’t make sense. For an individual station, I could pick any baseline I wanted, and I’d still get the same trend at that station.
EM Smith:
You have used a lot of verbiage here, but I still see no response to rbroberg’s question:
“But I haven’t seen you yet disclaim or any in way moderate your position that the ‘display bug’ was ‘bleeding through to the “real data.”‘
Do you still hold that view?”
You were suspicious of any and all red splotches on the maps, especially the ones around the Arctic. With a minimum investment of time, both rbroberg and I found the specific stations that resulted in those red splotches. So it’s quite clear to us that those red spots are due to actual stations, not any grid point using a 9999 in its calculation. Do you have a reply to that?
carrot eater (09:40:49) : Do you have a reply to that?
Yes: You don’t seem to read very well.
I’ve clearly stated it was a suspicion, an early ‘line of investigation’ of a bug. NOT a conclusion. You seem to have trouble figuring that out. Not my problem. I’ve also stated that now, with the further information given, it looks more like a graphing issue: but the numbers do get used in the calculation of the average anomaly in the upper right corner, so we don’t know were else they might be used. There really isn’t anything I can do to make that more clear to you.
And I’ve also said that you never fully remove a suspicion from from the debugging list until the code is patched and that fixes the problem. (NOT just blocking the test case as NASA has done. Doing that just assures we will never know when it is fixed.) All you do is change the weighting on the probability. It is simply not possible to know what the bug is from the outside; or even by inspection of the code (since there can be compiler bugs). So until you patch what you think is wrong and run the QA suite AND it comes up clean, all suspicions stay on the ‘things to check’ list. (Heck, even after that they go into the QA suit to keep checking for those suspected error cases forever. That’s what a QA suit is all about: checking what you suspect might happen even when you believe it is not happening; because some times it does happen.)
Further, we don’t know the limits that tickle this particular bug. So any samples of individual maps that are shown valid DO NOT prove that all data on all maps are valid. If the bug can be shown localized to the graphics display, then you can start to have confidence in the numbers, but until that happens it is simply not possible to know what happens to the 9999 data in all cases. The idea that “one valid result validates the code” is broken reasoning. The same kind of broken reasoning shown in the “Harry Readme” file when they “fix the code” by taking a bad value out of the input data instead of preventing a float overflow in the first place. (I showed how to fix it properly on my site). On the next bad data item they will have a float overflow again. Similarly here. We don’t know what’s wrong, so we can’t know when it’s fixed or what it impacts. We can only hope that the guys at GISS do find the problem and fix it; and not just paper over it by preventing the test case. IFF we could run the “baseline against self” test and have a grey / white map; then we would know their fix worked. With no test case, we will be left with perpetual doubt that they have ever fixed anything. And “hope” is not a debugging strategy nor a QA suite.
You seem to want to leap to the conclusion that everything is just perfect. So go ahead. Just don’t expect me to follow you off that cliff of conclusion.
“Hammy (10:50:06) :
Has anyone suspected a “ghost in the machine”?????? Like Hitler’s ghost. I have see many movies where Hitler is in one of his rages going “9999″.”
Ahem. “9” – nine – sounds like the german word “Nein” meaning “No”. No numerology here.
Surely the “fact” table that is to be presented to the rendering tool shouldn’t contain any rows where the measure value carries either a null or a null susbstitute. Nor any rows where an identifying attribute needed by the rendering logic is null or carrying a null substitute.
If the rendering engine accepts manual input the same should be appllied by the input code, in addition there should be some range limits on input (e.g. start with -273.1C <= temperature <= 100 C as the initial ballpark for air and sea temperatures then hone it down by experience…)