Thanks for an interesting analysis of the recent temperature data. My only comment is for people like Lee: Please tell us when a temperature trend is long enough/clear enough to infer that the trend is real. There is no statistically significant trend in average global temperature since 2001. How long would the current lack of statistically significant warming have to continue before it would be reasonable for someone to doubt IPCC’s 2007 projections of warming (and resulting environmental disruptions) due to carbon dioxide over the coming two decades? 3 more years? 5 more years? 20 more years? Under what circumstances could people reasonably conclude that the IPCC projections of 0.2C per decade are simply wrong? Would a falling temperature trend for the next 5 years be enough?

The researchers at the Hadley Center have demonstrated the courage of their convictions, and have predicted no warming until the end of 2009 (due mostly to ENSO), followed by a return to “rapid warming” in 2010 and beyond.

This prediction by the Hadley Center is enormously helpful, for it begins to place the climate modelers in the same boat as everyone else who works in science: good science makes accurate predictions, while bad science makes incorrect predictions. With the Hadley Center on record about the next several years, reasonable people will be able to evaluate the validity of the climate models the Hadley Center is using to make their predictions.

If there are no circumstances under which we can conclude from incorrect predictions that the models are wrong, or if the modelers simply refuse to make or be judged by the model predictions, then the modelers have left the field of science and entered the field of theology….. and they should just be ignored by scientists (and everyone else).

Then why do you constantly point to those questions?

The usual – the primary – assumption with any data series is that a.) it will have errors, and b.) those errors will be randomly distributed.

Every analysis made on this site is based on those assumptions. The statistical techniques waved at every opportunity here as some sort of Better Business Bureau badge of approval, just flat out don’t work without the assumption of random distribution of errors. That is a fact of life.

But for apparently agenda related reasons, you – and others – constantly insinuate that the errors are systemic – in one direction only, rather than having the random character assumed in your own analysis. Yet you use bogus application of statistical techniques to support those insinuations.

A claim of unidirectional error in all data series – that all errors in the measurement of temperature are positive, with none negative – is an extraordinary claim, and you better supply extraordinary evidence. Muttered insults aren’t going to cut it.

If you believe there are systemic errors you should make that case, not just say ‘abracadabra’ with statistical magic and predigestation, while stroking your beard in false concern.

You can’t have it both ways. You can’t on the one hand make arguments from data, and then hint that the data is suspect. Make an honest argument from the data available, or alternatively question – with real argument, not just concern trolling – that the data should not be relied on.

Otherwise you’re just substituting major league noise and nitpicking for honest discourse.

I’m looking at using LAD (least absolute deviation) to do the regressions. This will produce a straight line, which everybody seems to be able to understand, but will not be as influenced by outliers the least squares regression is. I may return to the use of it with the same four metrics this discussion has focused on, but right now, I’ve turned my attention to the IPCC’s claims about what has happened over the hundred year period 1906-2005.

Steven Mosher,
:)

REPLY: We’ll take a look. – Anthony

In fact they are waiting for the next El nino to bring the “trend” back to the
norm.

There is no climate trend. there is the mean of a calculation. we never observe
the climate. It does not exist.

http://www.climateaudit.org/?p=2868#comment-225274

– and you appear to have invented the cosmic ray detector that can be seen in temp data. Seems to confirm that temp cycles look to be the same as cosmic ray cycles… then again it could also be that cosmic ray detection equipment is temp sensitive… whoops. :-)

(Here’s a prediction for the future. As we move forward in time, the trend from a straight line regression through the data since the beginning of the satellite period, 1979, will drift downward…)

Well… Duh. I could tell you that without the fancier versions of showing statistical trends. I’d guess though that you could treat 1998 as an outlier and ignore it and conclude similarly. Or.. you could calc 2nd derivatives and plot regressions on that; it would do the same thing, especially if you tossed 1998 as an outlier. I merely wanted to point out that your plots didn’t add any more than what I already knew or could derive using simpler means.

On the other hand refer back to the cosmic ray and neutron plots and now you have a statistical tool that tells me something I didn’t already know.

But in all the discussion of whether it makes any sense to imply a trend from the average cumulative change of a series, or concern about too much weight being given to the downturn at the end of the period of analysis, a significant observation made in Part II has largely gone overlooked. There I noted:

“Incidentally [a lower implied decadal average than what results from a , this not entirely owing to fitting a downward trend through the data since 2001. Separate slope and constant dummy variables are also included for the 1998 El Nino, and this accounts for some of the difference. In fact, somewhat surprisingly, when a constant dummy is added for the 1998 El Nino, it reduces the slope (trend) for the non-El Nino part of the time series through 2001. We usually expect a constant dummy to affect the model constant term, not the slope. But in every case here it reduces the slope in a significant way as well, so some [maybe most?] of the difference in the “dT” and the result we’d get from a straight trend line owes to the effect of controlling for the 1998 El Nino.”

For anyone still interested in trying to learn something from all of this, I invite you to look carefully at the following chart, for HadCRUT, which superimposes all three trending/smoothing methods on the data, paying particular attention to around 1995 and 1996, in the months before the 1998 El Nino:

http://i26.tinypic.com/51cwaw.jpg

First, note how poorly the HP smoothing captures what is happening at that point: it begins trending upward while the actual anomalies are moving downward before they begin the rapid climb to the 1998 peak. In the same way, the straight linear trend, that so many seem convinced is the only way to do this, is well above the x-axis at a time when the anomalies are moving downward below the x-axis. Both the straight line trend, and HP smoothing, are excessively influenced by the 1998 El Nino, i.e. “pulled upward” by it, in a way that is unjustified by the data.

Of the three techniques, only the one that models 1998 El Nino as a shock (and “change point” also), avoids this bias and accurately measures the straight line trend in the period prior to the break point used to control for the 1998 El Nino. And that trend, taken directly from my regression output, and not some “dodgy” number I’ve calculated from endpoints (though it would be exactly the same, rendering moot the concerns or criticisms about my “dodgy” technique), converted to decadal form, is 0.118C/decade, well below the 0.159C/decade derived from the straight line for the entire period. The latter is unduly influenced and biased by the 1998 El Nino.

Here’s a prediction for the future. As we move forward in time, the trend from a straight line regression through the data since the beginning of the satellite period, 1979, will drift downward as we move away from 1998 El Nino. When the El Nino becomes part of the earlier half of the data series, those looking for evidence of AGW will finally see the wisdom of controlling for it somehow, because at that point it will actually become a negative influence on the overall trend, being on the other side of the fulcrum point as it were.

Again, thanks for all the dialog.

I’ve replied to you over on your own web page.

Basil

Basil is simply saying that a straight linear progression maybe leaving out possible information about a larger overall trend. To be honest we do not know what this larger overall trend portains to, but in looking a a smoothed dataset you can see where the nuiances of said trend maybe occuring rather then looking at the trend as a whole.

Look at the DOW over the last year. If this where the average temperature the trend on linear regression would be positive. This is true even though we are now basically at the same amount of money that the DOW is now vs a year ago. Now if you were to look at this as a smoothed trend you would see that it was positive, and now it is negative. You get a bettter idea of what is going on really by the smooth trend.

Again sorry if I am speaking out of turn, just trying to help explain what is going on by attempting to give an outside example that may not have people as biased in thier thinking… If that is what is going on.

I think we are in violent agreement. The descriptions I hear from climate scientists lead me to believe that the underlying model ( reality) is not
linear. So, fitting the temps to a linear trend is excel easy and not very
informative. That said, I scratch my head when pressed for an alternative.
Corrections for serial correlation are a good start..

I liked your approach, I just need to wrap my skull around it.

This simply demonstrates that this smoothing technique, like most others, can be quite unstable near the beginning and end of a time series.

By the way, I made a donation to Watts’ tip jar , today and recommend that readers give what they can.

REPLY: Thanks Tom!

Is that what they call damning with faint praise? :) I appreciate the attitude, but you are still not getting it.

Let’s you (and Lee) put aside the smoothed series in Part III for a moment, and think back to the linear trends estimated in Part II. How do you propose calcuating the “average” trend for a trend line which slopes up modestly for a while, shoots sharply up and back down before resuming the modest trend, then jumps up and begins to trend down after 2001? Everybody knows how to read the trend of a straight-line regression. How do you read the “average” trend of a trend line that varies like the trends plotted in Part II? Don’t suggest that I fit a straight line trend through the trend (which is basically what you are proposing for the smoothed series in Part III).

I took a short cut and calculated it from the delta in the end points. But let’s do for the trends in Part II what I did for Lee on the smoothed lines in Part III: first difference the trend, and take the average of the first differences. For HadCRUT in Part II, this produces an “average” trend, per month, of

0.00093967

This compares to the straight line linear trend of

0.00132630

The first number is lower than the second number because of the declining rate of growth at the end of the series. Multiply these numbers by 120 for decadal equivalents. BTW, in my reply to Lee where I calculated the equivalent HadCRUT number for the smoothed series,

0.00086229

I said “Now multiply it by 349, and divide it by 120, and see what you get.” That was incorrect. For these numbers, we just multiply by 120. We divide by 349 when using the delta from the endpoints, which should just give us the same numbers we have above. I.e., if we do the math right, it makes no difference whether we compute the average from the end points, divided by 349, or from the average of the monthly first differences. The numbers should be the same.

Everybody understands the 0.0013230 from a straight line regression, because that is what it is, and can be read from the output of a statistical regression. That doesn’t make it a better, or more correct number than the other two, which I think people are having a hard time grasping because of the novelty of the technique used to derive them. But the technique is sound, and nobody has yet to show otherwise.

In the end, the “trend” is no more meaningful than the delta from the end points. One is simply an estimate of the average monthly change, and the other is a measure of the cumulative change. Choosing which model is “best” has nothing to do with how I’ve calculated the cumulative or monthly or decadal change.

Paul Clark,

I don’t know the answer to your question. I would be surprised, though. Roger Pielke Jr. has a little write up today over at http://www.icecap.us on what Lucia is doing to validate some IPCC projections. If you haven’t been following what she’s doing, head over to her web site (http://rankexploits.com/musings/) and take a look.

JamesG,

I understand your point. When I teach anything having to do with statistics, I usually begin “If you torture the data long enough it will confess, even to crimes it did not commit.” I’m sure that’s what Lee thinks I’m doing here, but I’m not. I’m asking some questions of the data that don’t appear to have occurred to many, but not for the purpose of forcing it to confess to any preconceived notions of what it should be saying.

Stan Needham, Gary Gulrud, Josh, Enochson,

Thanks for the kind words. Gary, your comment (“I especially respect your limiting the discussion to features of the data rather than leaping ahead to causation which statistics seldom informs.”) especially made my day when I first read it.

jd,

Yes, it has been going up. But by how much, “on average?” That’s obviously a question that a lot of people are interested in, and I don’t think the answer is as simple as what you get from fitting a straight line through the data.

Stephen Mosher,

“Fitting monthly or yearly temperature data with a linear regression is just plain stupid. That’s a theorem somewhere. ok, it’t not stupid. It’s easily communicated.”

I wouldn’t call it “stupid,” but it is often done to data that don’t deserve it, or which should be analyzed more carefully before concluding too much from the slope of a straight line trend.

Ian,

The .001 is applies to the decadal form of the estimate. Take 0.00086229 and multiply it by 120, and you have 0.1034748. The latter is what the .001 applies to.

As mosher points out a regression is probably the worst way to do this except for all of the other methods.

That being said, bear in mind that although I disagree with your conclusions, I applaud the post series. Certainly, by looking at the data from a different persepctive, this gives food for thought… meaning you have made your point, even if I figure you are wrong. This is reminiscent of Edison in a way; he said he’d discovered 2000 ways to not make light bulbs but never once failed.

Congratulations on a thought provoking series!

Regardless of our individual levels of success those who aspire to word-smithery do so to communicate well. It’s a work-in-progress.

I find your tendency toward criticism of your cohort, “people on our side”(here and in adjoining threads), PC.

Might I suggest ‘Eeyore’ as a more fitting, exuse the pun, nom de plume?

I’m less certain about alternatives. A piecewise linear ( say Tammy) is a cheap
and easy fix. Basils approach, I havent got to the bottom of. Beware the cookbook chaps. However, eschew the cookbook at your peril Dr. Mann.

I’d prefer to model the system and devise a technique to detect trend changes
in that system. hmm

Atmoz has an interesting post on Short term cycles ( like ENSO) and the trend excusions you see during these periodic episodes.

At the bottom. This cooling weather phase is probably a very good thing for climate science. Why?

You mentioned the avg monthly first difference of 0.00086229, and a confidence interval (I believe for this avg) of +/-0.001. That gives a range of 1.86229 to -0.13771. What, if anything, should I conclude from that (esp. that the range takes in 0)?