Is the NULL default infinite hot?
January 31, 2010 by E.M.Smith
see his website “Musings from the Chiefio”
The empty ocean goes infinite hot on a null anomaly
What to make of THIS bizarre anomaly map?
What Have I Done?
I was exploring another example of The Bolivia Effect where an empty area became quite “hot” when the data were missing (Panama, posting soon) and that led to another couple of changed baselines that led to more ‘interesting red’ (1980 vs 1951-1980 baseline). I’m doing these examinations with a 250 km ’spread’ as that tells me more about where the thermometers are located. The above graph, if done instead with a 1200 km spread or smoothing, has the white spread out to sea 1200 km with smaller infinite red blobs in the middles of the oceans.
I thought it would be ‘interesting’ to step through parts of the baseline bit by bit to find out where it was “hot” and “cold”. (Thinking of breaking it into decades…. still to be tried…) When I thought:
Well, you always need a baseline benchmark, even if you are ‘benchmarking the baseline’, so why not start with the “NULL” case of baseline equal to report period? It ought to be a simple all white land area with grey oceans for missing data.
Well, I was “A bit surprised” when I got a blood red ocean everywhere on the planet.
You can try it yourself at the NASA / GISS web site map making page.
In all fairness, the land does stay white (no anomaly against itself) and that’s a very good thing. But that Ocean!
ALL the ocean area with no data goes blood red and the scale shows it to be up to ‘9999′ degrees C of anomaly.
“Houston, I think you have a problem”…
Why Don’t I Look In The Code
Well, the code NASA GISS publishes and says is what they run, is not this code that they are running.
Yes, they are not publishing the real code. In the real code running on the GISS web page to make these anomaly maps, you can change the baseline and you can change the “spread” of each cell. (Thus the web page that lets you make these “what if” anomaly maps). In the code they publish, the “reach” of that spread is hard coded at 1200 km and the baseline period is hard coded at 1951-1980.
So I simply can not do any debugging on this issue, because the code that produces these maps is not available.
But what I can say is pretty simple:
If a map with no areas of unusual warmth (by definition with the baseline = report period) has this happen; something is wrong.
I’d further speculate that that something could easily be what causes The Bolivia Effect where areas that are lacking in current data get rosy red blobs. Just done on a spectacular scale.
Further, I’d speculate that this might go a long way toward explaining the perpetual bright red in the Arctic (where there are no thermometers so no thermometer data). This “anomaly map” includes the HadCRUT SST anomaly map for ocean temperatures. The striking thing about this one is that those two bands of red at each pole sure look a lot like the ‘persistent polar warming’ we’ve been told to be so worried about. One can only wonder if there is some “bleed through” of these hypothetical warm spots when the ‘null data’ cells are averaged in with the ‘real data cells’ when making non-edge case maps. But without the code, it can only be a wonder:
With 250 km ‘spread’ and HadCRUT SST anomalies we get bright red poles
The default 1200 km present date map for comparison:
GIS Anomaly Map for November 2009
I’m surprised nobody ever tried this particular ‘limit case’ before. Then again, experienced software developers know to test the ‘limit cases’ even if they do seem bizarre, since that’s where the most bugs live. And this sure looks like a bug to me.
A very hot bug…
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
And if anybody wanted yet another indication that the 9999 values don’t contaminate anything they shouldn’t: Make one of the messy maps with the time period equal to the baseline, and note that the zonal means are all still 0. No sign of 9999 being used. Nor in a regular map, where there are 9999s behind the greyed out areas.
That bug has been there for at least a couple years. I pointed it out at Climate Audit a couple years ago. I don’t think it bleeds into their temperature anomaly though. Just a problem with their plotting software. Of course it does call into question the quality of their programming.
What is supposed to happen if you have a data file that has a certain flag, like a 9999. And your program, written earlier, expects to find -9999 for no data.
That would be operator error. Oops.
Or the program was updated to handle new No data flag types, but fails to do so. That’s a bug.
But, knowing the shennanigans we’ve seen in the past year, who’s to say someone conveniently edited the source code to make the No data flag type read as data. That’s monkeybusiness.
I wonder which one it could possibly be: oops, bug or monkeyfingers.
Phil M (17:25:37) :
Phil M (17:51:17) :
Phil M (18:36:14) :
Carrick (19:20:26) :
So, what have we found out? The actual GISTemp code is not publicly available. However the Clear Climate Code emulation (i.e. an imitation) does do proper error handling, and looks so good people somehow think they have the real thing, except written in Python.
Thus we have learned an equivalent amount about GISTemp that we could learn about the coding of Microsoft’s Solitaire game(s) from studying a “virtually identical” freeware version, which is basically nothing.
=============
To dbs and E.M. Smith, thanks for the replies on the moderation issue.
Mr. Smith, extra thanks for your work on the black box reverse engineering. Maybe some day we’ll get a successful FOIA request on the actual code, although I’d put better odds on another “helpful hacker” releasing it first. Until then this will do, and it is quite revealing and much appreciated.
“carrot eater (21:43:48) :
The choice of baseline is completely irrelevant to the ranking of years or decades.”
Yes I totally agree the 1961 – 1990 global average used as a yardstick by almost all official weather services around the world is completely irrelevant.
kadaka:
What do you mean?
The GISTemp source is here.
Ouch. Looks like very dirty programming, I’m suspecting that they’re filtering out the null values (999.9 in the GISS data files) AFTER they’ve done the actual computation, but that their way of doing this breaks down when the only available values are 0 and 999.9.
Still, it doesn’t seem to me that it really affects the non-pathological cases. The Arctic WAS unusually warm in December (e.g. Svalbard was much milder than southern Norway most of the month). But one thing to keep in mind, is that the unusually “warm” weather in the Arctic really wasn’t *warm*. Does it really matter whether the temperature at the north pole is -35 or -45? It will be interesting to follow the global temperatures in the next few months. I think the harsh winter in Europe, Siberia, China, the US, etc. may be followed by a slower than usual spring, because of the snow albedo and the frozen ground.
Back to GISS: One thing which puzzles me, is the apparent special treatment they give South Pole measurements. Look at the big orange circle here: http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=1212&year1=2009&year2=2009&base1=1951&base2=1980&radius=250&pol=pol
I asked for 250 km smoothing, and they did it, except around the south pole where there seems to be a circle of 1000 km radius? Why?
Should be possible to test to see if the blood is bleeding through. Set up a program to change all non-9999 values to the value giving white, and replot. I can see the logic, would do it if I had the expertise, but it’s not my job at this point…
Carrot Eater, Phil M and Carrick. While you are visiting from RC, don’t forget to go to the other threads and offer your opinions – for instance you could tell us why the IPCC chairman’s porn novel doesn’t further discredit him.
Or why Phil Jones’ missing Chinese stations are not an issue.
Or why cleaning your boots when visiting Port Lockroy is relevant to global warming.
Not forgetting those “melting” glaciers too, of course.
I haven’t seen you on those threads yet.
Too busy at Wikipedia?
Correction: “Phil M” turns up at porngate to admonish Anthony for not blocking a Google advert for something called “50 Sexiest Women of Science Fiction”. Well done Phil! In true warmist fashion, even when talking about the IPCC porn novel, he offers no defence for his spiritual leader, he just attacks Anthony about something else.
Wow. This is curious.
See this map.
Data Sources:
Land: unadjusted 1880-1999
Ocean: none
Map Type: Anomalies
Mean Period: Annual (Dec-Nov)
Time Interval: Begin 1951 — End 1980
Base Period: Begin 1951 — End 1980
Smoothing Radius: 1200 km
Projection type: polar
Now to me, studying the downloaded pdf, in the right-side Southern Hemisphere view, between 90 and 180 deg W longitude there is a large block of yellow (.2 to .5) in the Pacific Ocean. Note the sharp termination line, the protractor says it is at 179 deg W (181 E). Note also that there is no yellow in the NH map, only the SH. The block of yellow also terminates right at the equator.
Also, there appears to be only three colors, the yellow, red (2 to 4), and dark red (4 to 9999). On 400% zoom there is a noticeable small patch of white (-.2 to .2? Left edge not marked, -.2 on regular projection) in the SH at about 85 deg E near the equator, which would be about 10 deg east of and south of the tip of India.
Switching back to regular projection, there is a lot of white, dark red, and no yellow. And something else that’s curious. The SH white patch? On regular projection it looks diamond shaped but dark red, same shape seen on the 400% zoom of the SH polar projection in white.
Changing the time intervals to 1970-1999 with polar projection, looks the same. Only 1970 for all year fields, same.
Experimenting further, I tried the other end, using 1880 to 1910. Which generated the following error message: I cannot construct a plot from your input. Time interval must begin in 1880 or later. Yup, it thinks “1880” is not 1880 or later.
So, switching to 1881 to 1910, some different colors, but still the curious cooler section that terminates along 179 W and the equator. It is very noticeable how at 179 W one clearly-defined region suddenly transitions from dark orange (1-2) to white (-.2-.2?) while another goes yellow (.2-.5) to white.
Checking by single years to see when the diamond first showed up, I noticed for 1953 the 179 W transition was in the NH but not the SH, an odd flip. Looking back, the flip is in 1952. For 1951, there is the 179 W transition in the NH, but now in the SH the transition is reversed, marking a change to warmer instead of cooler.
At 1950 the 179 W transition is only in the SH.
The off-India diamond appears to first show up in 1954, with the linked polar projection showing a dramatic NH to SH color difference. 179 W transition is only in SH, again.
All this clearly shows to me there is something wrong.
I could accept the problem lies in the bit of programming that draws the polar projection maps. Except the 179 W artifact is not consistent. From 1951 to 1953 inclusive it was showing in the NH, with 1951 also showing in the SH but with a reversed effect. That indicates to me two things. One, the code that makes the polar projection maps is showing me something in the data, it’s not all in the map drawing. I cannot imagine how there could be an error in that code that decides to single out those three years for special treatment. Two, around the early 1950’s some sloppy adjustments were made to the data, with 1951 particularly messed up. It might be in the code that feeds data to the map drawing program, with encoded adjustments for those specific years, but it also is highly possible the changes are in the “unadjusted” data.
I would also like to know why the off-India diamond appeared in 1954.
Offhand I don’t think many people look at the polar projections, at least this way. To me it looks blatantly obvious something is screwed up, I’m surprised it didn’t somehow show up before.
Although I’m even more surprised that the GISTemp mapper doesn’t recognize 1880 as being 1880 or later. Come on, that is absolutely lame. Whomever missed that deserves a career encoding form layouts in COBOL.
In closing, I will simply note what is says on the bottom of the relevant web pages:
Responsible NASA Official: James E. Hansen
Patrick Davis (22:45:37) :
Please explain why you think the choice of baselines is important. It is not. All it does is change the absolute values of the anomalies. The trends, which is what’s important, do not change as you change the baseline.
kadaka (22:35:12) :
This is silly. All you don’t have is the code behind the online map plotter for the website.
aMINO aCIDS iN mETEORITES (20:44:16) :
“Did you see this?
http://wattsupwiththat.com/2010/01/23/arctic-temperatures-above-80%C2%B0n-are-the-lowest-in-six-years/
REPLY: FYI, true then for the moment, but they’ve already rebounded – A”
I did see it, but as Anthony demonstrated, that was weather, not climate. A clear example of why one should not eagerly await monthly satellite anomolies and use that as proof of a trend. Or point to a single extreme weather event,”hot” or “cold”, and attempt to infer something about a climate trend.
Sorry – OT but unable to post on Tips and Notes for some reason.
One for the weather not climate department?
Met Office confirms Scotland had coldest December + January for almost 100 years:
http://news.bbc.co.uk/1/hi/scotland/8492333.stm
Note – this does not mean coldest winter, which is Dec + Jan + Feb.
Here’s some of the documentation from Python version of the code (step2.py). I have not gone through the code line by line to verify that it does what it says, but the documentation appearing at the beginning of each class “def”inition follows the standard Python conventions. Also, it appears the translation of the code from Fortran (or whatever) to Python was done by an outside consultant. Everthing that follows is taken directly from the code; the last class is for all the consipiracy theorists. 🙂
def urban_adjustments(anomaly_stream):
“””Takes an iterator of station annual anomaly records (*dict*,
*series*) and produces an iterator of urban adjustment parameters
records. The urban adjustment parameters describe a two-part
linear fit to the difference in annual anomalies between an urban
station and the set of nearby rural stations.
The algorithm is essentially as follows:
For each urban station:
1. Find all the rural stations within a fixed radius;
2. Combine the annual anomaly series for those rural stations, in
order of valid-data count;
3. Calculate a two-part linear fit for the difference between
the urban annual anomalies and this combined rural annual anomaly;
4. Yield the parameters of this linear fit.
If there are not enough rural stations, or the combined rural
record does not have enough overlap with the urban record, try
a second time for this urban station, with a larger radius.
If there is still not enough data, do not produce an
adjustment record.
“””
def get_neighbours(us, rural_stations, radius):
“””Returns a list of the stations in *rural_stations* which are
within distance *radius* of the urban station *us*. Each rural
station returned is given a ‘weight’ slot representing its
distance fromn the urban station.
“””
def apply_adjustments(stream, adjustments):
“””Applies the urban adjustment records from the iterator
*adjustments* to the station records in *stream*. Returns an
iterator of adjusted station records.
Rural stations are passed unchanged. Urban stations without an
adjustment record are discarded. Urban stations with an
adjustment record are adjusted accordingly, to remove the modelled
urban-heat-island effect.
An urban adjustment record describes linear and two-part linear
fits to the difference between an urban station annual anomaly
series and the combination of annual anomaly series for nearby
rural stations. The linear fit is to allow for a linear urban
heat-island effect at the urban station. The two-part linear fit
is to allow for a model of urbanization starting at some point
during the time series.
“””
def flags(fit, iy1, iy2):
“””Calculates flags concerning a two-part linear fit. Also
accumulate statistics. Not well documented or understood, but
note that in adj(), below, the two-part fit will be disregarded if
the flag value is not either 0 or 100.
“””
aMINO aCIDS iN mETEORITES (17:26:48) :
Would you provide the data that, besides GISTemp, that shows “high arctic temps”?
Will the UAH satellite record work for you?
http://vortex.nsstc.uah.edu/public/msu/t2lt/uahncdc.lt
Arctic temps can be found under ‘NoPol’. The trend for that region is at the bottom, in case you’re interested in that too.
apparently the “problem” is fixed…and just in the last 10 minutes
@ur momisugly Carrick (23:49:15) :
and
carrot eater (04:01:16) :
GISTemp updates:
Example 1:
Example 2:
Example 3:
Three different times, they changed THEIR codes, THEN changed the publicly-available source codes. If what was released was THE real GISTemp they themselves were running, then they would only have to upload the new code for distribution, there would be no need for other modifications.
November 13 2009 may be another example, but there they said
thus it does not outright say the public version was modified separately. There may be other good examples hidden among the careful wording of the updates, but I didn’t meticulously scrutinize all the change notes.
barry (05:24:27) :
These are not high
to say they are high is an exaggeration
Phil M (05:04:13) :
funny, you guys keep going back to python
no surprise
Phil M (04:47:48) :
but weather actually is climate
the cooling of the last 5 years in the earth is why ‘weather’ is happening across the earth
warming is not happening
Steve M. (07:22:30) :
So it is; that was fast by GISS. I wonder how many people tried to make the weird maps over the last day.
kadaka (07:26:28) :
Give me a break. If you’re that determined to believe that the publicly available source code is different from what they run, then download that code, run it, and see if you can find any differences between its results and the published results.
Steve M. (07:22:30) :
apparently the “problem” is fixed…and just in the last 10 minutes
Yeah, isn’t the power of WUWT and the Chiefio amazing?
That’s why for my long post (03:41:38) I saved the pdf’s of the regular and polar projection maps as well as the text file versions of the gridded data for all the examples I linked to. If someone connected to either site wants to host them, I can email the data. Alternately if you know of a free hosting site, I could dump them there and post the links here.
PS, it still won’t take 1880 as a start year.
@ur momisugly carrot eater (07:59:14) :
If it looks the same and gives the same results it must be the exact same thing?
Have you ever sold Rolex watches on a streetcorner?
Carrick (17:36:22) :
“The color red is a bug?
Um, no.
It does what it needs to do, namely flag missing regions of the ocean. Would you prefer green perhaps?”
Ah, when the usual color for MISSING DATA is gray and when Red is used to denote a high anomaly, then ya it is a bug.
Is it of any consequence? I dunno. I’ve not seen it in any papers.
It is kinda funny though. I really respect the clearclimatecode guys. They find little bugs. they report them. NASA fixes them. Its really quite simple.
So, rather than everybody spending a bunch of time trying to figure out this “bug” ( scare quotes mean I accept your objection) somebody should just drop Reudy a note.
Crap, even I cant take my own advice and have spent more time telling people what to do than actually doing it.
E.M. Smith I would suspect most that a ‘missing data flag’ is not being properly handled and is bleeding through as ‘real data’.
It is is difficult to understand why you stubbornly cling to your suspicion (if you still do). It is obvious that the high and low end of the scales in the 2009 – 2008 diff as well as the 2009 – 1998 diff come from specific stations.
So to make it as obvious as possible. The 2009-2008 diff map shows a lower bounds of -17.4C and an upper bounds of 10.4C (anomalies). The following data comes from GHCN RAW (v2.mean) which, as you know, is the major input into GISTEMP. Remember, it is not that any station will set an endpoint. But a remote station that is the only one its grid and more than 250km from a neighbor can set an extreme by itself. I have not verified that the following indeed are a) the only station in its grid and b) more than 250km from its neighbors, but this theory is far more likely than your bleed over theory (which doesn’t explain the cold end points)
Here is the first map you selected (2009-2008, Dec) and asked for an explanation of the high end of the scale (ignoring the low end):
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2009&month_last=12&sat=4&sst=0&type=anoms&mean_gen=12&year1=2009&year2=2009&base1=2008&base2=2008&radius=250&pol=reg
Here is the GHCN data explaining the endpoints of the scale (-17.7, +10.3)
SALEHARD stnid 22223330000
Dec 2009: -30.3
Dec 2008: -12.9
Diff: -17.4
DAWSON, Y.T stnid 40371966000
Dec 2009: -19.3
Dec 2008: -29.6
Diff: +10.3
Here is another map you selected (2009-1998, Dec) and asked for an explanation of the high end of the scale (ignoring the low end):
http://chiefio.files.wordpress.com/2010/01/ghcn12-3_giss_250km_anom12_2009_2009_1998_1998.gif
Here is the GHCN data explaining the endpoints of the scale. (-17.7, +12.3)
TURUHANSK stnid 22223472000
Dec 2009: -39.1
Dec 1998: -22.0
Diff: -17.1
GMO IM.E.T. stnid 22220046000
Dec 2009: -15.7
Dec 1998: -28.0
Diff: +12.3
Your April 2008 map will require a little more work. The low end of the anomaly (-8.9C) comes from GHCN station 70089055000 CMS”VICE.DO.M. But the high end anomaly (+16.3) is in the Antarctic and GISTEMP uses the SCAR data set for Antarctica.
http://www.antarctica.ac.uk/met/READER/surface/stationpt.html
http://www.antarctica.ac.uk/met/READER/temperature.html