The View from Down Here Isn’t Much Different

James A. Schrumpf

If there’s one belief in climate science that probably does have a 97% — or more — consensus, it’s that the raw temperature data needs meaning, least-squaring, weighting,  gridding, and who know what else, before it can be used to make a proper data set or anomaly record.  I’ve been told several times that without some adjustments or balancing, my calculations will be badly skewed by the imbalance of stations.

So the null hypothesis of temperature data all along has been, “Just how bad/far off /wrong are the calculations one gets by using raw, unadjusted data?”

NOAA provides access to unadjusted monthly summaries of over 100,00 stations from all over the world, so that seems a good place to begin the investigation.  My plan:  find anomaly charts from accredited sources,  and attempt to  duplicate them using this pristine data source.  My method very simple.  Find stations that have at least 345 of the 360 records needed for a 30-year baseline.  Filter those to get stations with at least 26 of the 30 records needed for each month.  Finally, keep only those stations that have all 12 months of the year.  That’s a more strict filtering than BEST uses, as they allow fully 25% of the 360 total records to be missing.  I didn’t see where they looked at any of the other criteria that I used.

Let’s see what I ended up with.

Browsing through the Berkeley Earth site, I ran across this chart, and decided to give it a go.

clip_image002

I ran my queries on the database, used Excel to graph the results, and this was my version.

clip_image003

BEST has more data than I, so I can’t go back as far as they did.  But when I superimposed my version over the same timeline on their chart, I thought it was a pretty close match.  Mine is the green lines.  You can see the blue lines peeking from behind here and there.

clip_image005

Encouraged, I tried again with the contiguous 48 US states.  Berkeley’s version:

clip_image006

My version, superimposed:

clip_image008

That’s a bit of a train wreck.  My version seems to be running about 2 degrees warmer.  Why?  Luckily, this graphic has the data set included.  Here is BESTs average baseline temps for the period:

% Estimated Jan 1951-Dec 1980 monthly absolute temperature (C): %
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
% -4.07 -1.75 2.17 8.06 13.90 18.77 21.50 20.50 16.27 9.74 2.79 -2.36
% +/- 0.10 0.09 0.09 0.09 0.09 0.09 0.10 0.09 0.09 0.09 0.09 0.09

Here are mine:

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
-1.26 1.02 5.2 11.13 16.22 20.75 23.39 22.58 18.68 12.9 6.09 .98
.03 .03 .02 .02 .02 .02 .01 .01 .02 .02 .02.02

It’s obvious by glancing at the figures that mine are around two or so degrees warmer.  I guess it’s due to the adjustments made by BEST, because NOAA says these are unadjusted data.  Still, it’s also obvious that the line on the chart matches up pretty well again.  This time, the data set for the BEST version is provided, so I compared the anomalies rather than the absolute temperatures.

The result was interesting.  The graphs were very similar, but there was offset again.  I created a gif animation to show the “adjustments” needed to bring them into line.  The red line is BEST, the black is mine.

clip_image009

At the moment, the GISS global anomaly is up around 1.2, while BEST’s is at 0.8.  The simple averaging and subtracting method I used comes closer than another, highly complex algorithm that uses the same data.

My final comparison was of the results for Australia.  As with the US, the curve matched pretty well, but there was offset.

clip_image011

The same as with the US comparison, the BEST version had a steeper trend, showing more warming than did the NOAA unadjusted data.  For this experiment, I generated a histogram of the differences between the BEST result for each month and the simple method.

clip_image012

The standard deviation was 0.39, and the vertical lines mark the 1st and 2nd standard deviations.  Fully 77.9% of the error fell within one standard deviation of the BEST data.and 98.0% within 2.

What does this mean in the grand scope of climate science, I don’t know.  I’m certainly no statistician, but this is an experiment I thought worth doing, just because I’ve never seen it done.  I’ve read many times that results of calculations not using weighting and gridding and homogenization would produce incorrect results — but if the answer is not known in advance, like how the “heads” result of a million coin tosses will come very close to 1 in 2 is known, then how can it be said that this simple method is wrong, if it produces results so close to those from to more complex methods that adjust the raw data?

The climate data they don't want you to find — free, to your inbox.
Join readers who get 5–8 new articles daily — no algorithms, no shadow bans.
0 0 votes
Article Rating
78 Comments
kevin kilty
May 8, 2019 8:43 am

Arguing over mean temperature time series has become very, very stale. I am old enough to recall Gilbert Plass’s many publications about warming trends in northern Europe pre 1950. His graphs were probably sufficient to demonstrate a warming planet, but we know now that the trends these portrayed reversed after the early 1950s for the next several decades. So, time series were sufficient to show how CO2 was warming the planet until they weren’t. Of course, there is a ready explanation, actually several, for why they reversed that preserves “appearances”. The climate system is complicated enough that one can always find an explanation for why a trend does whatever it does.

People making statistical adjustments to time series never mention, perhaps because they aren’t aware, that adjustments come with uncertainties of their own. Without carefully examining each station’s time series, and data about station moves, encroachment, and so forth, one can’t be certain that statistical adjustments are helping matters. Many criteria by which we judge the correctness of time series and our corrections are ad hoc. There is no basis for even believing that corrections resulting in a residual “anomaly” that is a series of random, normally distributed numbers, means the corrections are valid. Weighting criteria are ad hoc, too.

One thing that climate scientists (I am not not one) are trying to accomplish with regard to mean temperature is to demonstrate that the Earth is out of equilibrium with regard to energy balance, and hence that some agent like CO2 or feedback is warming the planet. But this seems a pretty pointless goal when one considers that heat is being carried constantly toward the poles and toward the top of the troposphere–transport means we are always out of equilibrium, perhaps quite far from equilibrium, and we don’t fully understand and cannot model this transport accurately.

The thing activists are trying to accomplish is change to the political and economic structure of the world using a “climate crisis” as justification. Temperatures are in no way fit for this grand and frankly scary purpose.

What one thinks of the mean temperature record is dictated largely by one’s prior beliefs. If someone is predisposed toward belief in climate troubles, then a temperature time series means quite a lot. If one is skeptically inclined, it means much less. Looks much like belief in the paranormal doesn’t it?

May 8, 2019 9:43 am

This is all a crock. There can be averages of temperatures from various locations. There is no way to measure Global Average Temperature. The attempt is subject to manipulation such that no objective observer would ever trust the “results.” One single thermometer in the Arctic used to show temps across 1200 kilometers? Please.

Any dozen widely spaced long continuous records of temperature would show an effect from increasing CO2, if there is one. There is no evidence of one so far. The endless pontification about the global average, whatever they are calling it now, completely unscientific.

I did not spell “pontification” wrong, what is your SpellCheck doing?

Much Ado About Nothing…

Dr Francis Manns
May 8, 2019 12:40 pm

The temperatures at Berkeley record a number of volcanic cool periods and a number of ENSO cycles. The most obvious on your graphs is Krakatoa in 1883. The temperature prior to 1875 had already reached current range. There were 21 serious (cooling) eruptions between 1875 (Askja, Iceland) and 1934 (Robock, 2000) that are recorded on NOAA thermometers. I suggest you look at a more remote station like Key West. FL. or Death Valley, CA.

The modelers are misinterpreting regression toward the norm after the Little Ice Age. Regression toward the norm explains the hiatus. Climate models ought to be deleted.

Dr Francis Manns
May 8, 2019 4:34 pm

The temperatures at Berkeley record a number of volcanic cool periods and a number of ENSO cycles. The most obvious on your graphs is Krakatoa in 1883. The temperature prior to 1875 had already reached current range. There were 21 serious (cooling) eruptions between 1875 (Askja, Iceland) and 1934 (Robock, 2000) that were recorded on NOAA thermometers. I suggest you look at a more remote station like Key West. FL. or Death Valley, CA.

The modelers are misinterpreting regression toward the norm after the Little Ice Age. Regression toward the norm explains the hiatus. Climate models ought to be deleted.

Bindidon
May 8, 2019 4:37 pm

James Schrumpf

You seem, if I understand you well, to doubt about the usefulness and correctness of latitude and longitude gridding of absolute or anomaly-based station data before averaging it into time series.

Maybe you look at various graphs showing you why you are wrong.

Here is a graph showing for CONUS the yearly average of daily maxima per station in the GHCN daily data set:
(1) https://drive.google.com/file/d/1qGV5LfKw_lFKNdZMlq15ZHz6sA1CA294/view

And here is the same for the Globe:
(2) https://drive.google.com/file/d/1GMuNs9ptRzDd7KxFQbKv0o5ySR5VNc9b/view

Maybe you think: „Aha. The Globe is like CONUS“.

But… here is the Globe without CONUS:
(3) https://drive.google.com/file/d/1UcLK3usYjICeHeAsAb5ivcusW0Y0EdNe/view

What happens in (2) is that you have in GHCN daily worldwide about 18000 US stations competing with 18000 non-US stations: this means that about 6 % of the Globe’s land surface has the same weight as the remaining 94 %.

When you exclude the US stations, you irremediably get another picture of the Globe.

If you now average, in a preliminary step, all stations worldwide into grid cells of e.g. 2.5 degree (about 70000 km² each) you obtain the following graph:
(4) https://drive.google.com/file/d/1TFdltVVFSyDLPM4ftZUCEl33GmjJnasT/view

Yes: it looks like the preceding graph (3) in which all US stations had been excluded.

This is due to the fact that now, about 200 CONUS grid cells compete with 2000 grid cells outside of CONUS. Thus CONUS’ weight is reduced to about its surface ratio wrt the Globe. Fair enough!

Ironically, the gridding method gives, when applied to CONUS itself, a graph
(5) https://drive.google.com/file/d/16XwogXkjltMuSzCdC5mPKqC1KNev4ayy/view

showing that the number of daily maxima per station per year above 35 °C even decreases. The reason is again that the gridding gives more weight to isolated stations whose ‘voice’ was diluted within the yearly average of the entire CONUS area.

Like an ungridded Globe looks like CONUS, an ungridded CONUS looks like the sum of its West and East coasts.

Regards
J.-P. D.

Bindidon
May 9, 2019 4:04 am

James Schrumpf

I’m wondering why you cherry-pick some small parts of the Globe, like Europe, CONUS or Australia, instead of checking your simple [?|!] method against BEST’s entire land-only data:

http://berkeleyearth.lbl.gov/auto/Global/Complete_TAVG_complete.txt

Please download and enter it into Excel (you seem to be an excellent Excel user), enter the data you processed (I suppose you use GHCN daily), and proceed to a fair, complete comparison.

Maybe you see then that you possibly made more errors than BEST did adjust.

Your goal: to come as near to BEST as possible:

https://drive.google.com/file/d/1EymLmp1mHweXOnrVIRL0DDjnHabgt_yr/view

and compare your result with the little layman work I made, and a with more complex approach performed by Clive Best:
http://clivebest.com/blog/?p=8464

You will easily see there that Clive Best started the BEST/GHCNd comparison much earlier.

But my lack of knowledge and experience in statistics made the comparison with the professional work in earlier times simply hopeless because of the paucity of station data.

Regards
J.-P. D.

Frank
May 9, 2019 1:19 pm

James: Many scientific fields, we design experiments to determine what we want to know. Let’s imagine we are time travelers and could go back in time and measure temperature anywhere on the planet. For GHG-mediated global warming, we believe that GHGs slow radiative cooling to space, leaving more energy in the environment and therefore a higher temperature. So perhaps we would design an experiment to measure the temperature and average local heat capacity per unit area, and then see how much energy has built up over time in terms of W/m2. The ARGO buoys are already doing a pretty good job of measuring how much energy is accumulating in the ocean, so let’s give ourselves the job of measuring energy accumulation over land. Since there is the same weight of air over every square meter, the heat capacity of the atmosphere is the same over every square meter. If we assume the lapse rate remains constant (a reasonable approximation), then warming everywhere in the atmosphere will be the same as warming 2 m above the surface. So, maybe we would design an experiment where we placed one temperature station in every 1, 10, 100, 1000 or 10,000 km^2. If we intended to compare with a climate model, we might place one temperature station at the center of every land grid cell in bottom layer of the climate model. However, I doubt we would scatter temperature stations at random, or put them mostly in cities, or near the coast, or near airports.

Unfortunately, we aren’t time travelers. We must work with the data we have, despite the fact that the thermometers aren’t ideally distributed. Nevertheless, we can still make a grid and average all of the thermometers in each grid cell, so we can have one temperature for every (say) 10,000 km^2 grid cell. Then we can average all of the grid cells. BEST has a better way of doing this: kriging, which is a way of using the data to determine a temperature “field”. That field is continuous: T(x,y) is the temperature at point x,y. Gridded temperatures are discontinuous.

All of the temperature indices calculate temperature per unit area and average, because that is what climate scientists most want to know from an energy accumulation perspective.

You may have a different perspective. You may care about how much warming is hurting people design an experiment with one thermometer in the center of every 100,000 people. Or put the thermometers where both people and agriculture are. However, whatever you want to know, thermometers probably have not been placed at the right location to give you a MEANINGFUL answer, so simply averaging is unlikely to provide the best answer. As best I can tell, simple averaging serves no purpose.

With the advent of satellite altimetry, we can measure risings sea level everywhere in the ocean. However, I only care about SLR at the coast where people living. That happens to be where tide gauges are located. In this case, simple averaging would be more useful than the global average we get from satellite altimetry. However, a better answer would be to weight the tide gauge data by the value of the land at risk around the tide gauge. (Climate scientists need the global value to prove thermal expansion, ice cap melting, glacier melting, ground water depletion, and reservoir-filling explain the rise we see.)

Bindidon
May 12, 2019 3:28 pm

James Schrumpf

I lacked the time to go back into the center of our discussion about GHCN daily. But inbetween here are some related infos concerning similar things when processing sea level data in about the dsame manner as temperature time series:

https://www.psmsl.org/data/obtaining/rlr.monthly.data/rlr_monthly.zip

Below you see a graph comparing two evaluations, the one without and the other with gridding:
https://drive.google.com/file/d/1LNrib0i7EoapqaHyjt3S93h5eqc__8dc/view

How important it is to do this gridding you see here:

https://drive.google.com/file/d/1Nm-BGX6jeymj_fsqkvmpe8QrSj9ZRR66/view

In this graph you have 1513 tide gauge stations. The smallest points show grid cells with up to 4 gauges.
The bigger points show stations with up to about 20 of them.

12 % of the grids totalise 40 % of the stations!

But… this is not the end of the story, because our simple-minded baselining eliminates lots of stations having provided data in earlier times, whose data then is absent because they lack sufficient baselining data.

This leads to a strange bulge in the time series which very probably is not contained in a correct baselining.

A very first trial to solve the problem was to generate 4 time series baselined wrt different reference periods, each about 30 years long, and to rebaseline 3 of them wrt the main one (1993-2013):

https://drive.google.com/file/d/1a0iG06y9mISWaXnrzWLNQkKUy4Ti6X52/view

This motivates to an automated solution which, if successful, could be brought back to the temperature data processing.

It seems that Church and White, whose work has been subject to harsh criticism on this blog, can’t have been that wrong, isn’t it?

In a few days, I’ll come back again and present similar data concerning GHCN daily.

cu
J.-P. D.

Bindidon
May 14, 2019 6:06 pm

James Schrumpf

Apologies for this reply, but… to write “my method” about what has very certainly been designed and implemented 40 years ago is a bit (too) hard. Indeed, it is “about as simple as can be”, I tested it 5 years ago, but quickly did understand that it is by far too simple.

And no, Mr Schrumpf: ” calculating an anomaly for each station individually, then averaging those results”, does NOT have “the same effect as gridding.”

I thought you would understand that when looking at sea level data processing. You didn’t.

So let us come back to temperatures.

I generated yesterday two time series with absolute data (out of GHCN daily, but I could have done the same job out of GHCN monthly V3 or V4):
– one without gridding;
– one with gridding.

The average temperature over all months from Jan 1880 till Dec 2018:
– no gridding: 10.2 °C
– gridding: 15.2°C.

How is that possible?

The GHCN daily data processing told me this:
– 35537 stations contributed during the period above;
– these stations were/are located within 2567 grid cells.

This gives in average about 14 stations per grid.

The 413 grid cells containing more than 14 stations made a total of 29116 stations. This means that no more than 16 % of the cells in the world contain 83 % of the stations.

And that’s how it looks:
https://drive.google.com/file/d/17ZgjmYUL43320EoLQ5bL0Hs3aYwas-gt/view

As you easily can see, the 413 stations are nearly all located in the US and in Southern Canada, in Northern Europe and in South East Australia.

Thus if you don’t grid, your global average temperature will look like an average of these three corners, and not like an average of the Globe. How could 17 % of the data manage to beat 83 % of it?

This you can see in this graph below, in which no gridding was applied:
https://drive.google.com/file/d/14wlJi4thfapzR70jU2VOH4zkEOY7A074/view

As you can see, the blue line looks very flat. No wonder: its data shows a trend of 0.077 °C / decade.

The red line (whose data does not contain any US station data) has a well steeper slope with 0.15 °C / decade, i.e. nearly twice as much.

And now have a look at the chart below, which compares gridded data for the Globe with and without the US:

https://drive.google.com/file/d/1uQoPuALu8TkUzJAMlBLYZk-sDff4aFCG/view

Now, due to the fact that we have 413 grid cells competing with 2154 cells, instead of having 29564 stations competing with 5973 stations, the comparison becomes quite different: the two plots are nearly similar, their trends are equal with 0.10 °C / decade.

Your words: „It seems that by reducing the values to be averaged to an anomaly first, and then averaging those reduces– or even eliminates — any benefits of gridded data.“

Sorry, Mr Schrumpf: this is simply wrong.

May 15, 2019 10:42 am

Hi Bendidon,

You might have got different results if you had reduced the values to anomalies first, and then averaged those. From the description of what you did, you again averaged absolute temperatures in some manner and then turned those into anomalies. I’m guessing you took the stations within one grid and averaged those temps to get a grid average temperature. Then you used those grid averages in some way to produce anomalies to average into your global average anomalies — perhaps with some extra processing as well.

You should take the blue line from your non-gridded chart (globe with US data) and lay it along the lines from your gridded data chart. They match almost perfectly back to around 1953 or so — around 66 years.

Why should non-gridded data chart so perfectly over gridded data for two-thirds of a century?

I’ll post some more when I’ve had more time to look your and my data over.

May 15, 2019 6:44 pm

Seems to me you are ignoring the fact that the non-gridded, US-included line falls right on top of your gridded lines for two-thirds of a century. You seem unable to explain it and prefer to handwave it away as irrelevant.

It also appears to be lost on you that your blue, ungridded, US data had a trend of 0.077C/ decade, while your gridded data had a trend of 0.10C/decade.

The difference in those trends is only 0.023C. What was your standard error? It only needs to be 0.012 for that difference to be statistically insignificant. At any rate, its hardly the vast gulf you make it to be.

It’s too bad you decided to give up and go home. It could have been interesting if you didn’t believe what you couldn’t explain was unimportant.