Doug L. Hoffman
Friday, Dec 18th, 2009
While climate skeptics have gleefully pointed to the past decade’s lack of temperature rise as proof that global warming is not happening as predicted, climate change activists have claimed that this is just “cherry picking” the data. They point to their complex and error prone general circulation models that, after significant re-factoring, are now predicting a stretch of stable temperatures followed by a resurgent global warming onslaught. In a recent paper, a new type of model, based on a test for structural breaks in surface temperature time series, is used to investigate two common claims about global warming. This statistical model predicts no temperature rise until 2050 but the more interesting prediction is what happens between 2050 and 2100.
David R.B. Stockwell and Anthony Cox, in a paper submitted to the International Journal of Forecasting entitled “Structural break models of climatic regime-shifts: claims and forecasts,” have applied advanced statistical analysis to both Australian temperature and rainfall trends and global temperature records from the Hadley Center’s HadCRU3GL dataset. The technique they used is called the Chow test, invented by economist Gregory Chow in 1963. The Chow test is a statistical test of whether the coefficients in two linear regressions on different data sets are equal. In econometrics, the Chow test is commonly used in time series analysis to test for the presence of a structural break.
A structural break appears when an unexpected shift in a time series occurs. Such sudden jumps in a series of measurements can lead to huge forecasting errors and unreliability of a model in general. Stockwell and Cox are the first researchers I know of to apply this econometric technique to temperature and rainfall data (a description of computing the Chow test statistic is available here). They explain their approach in the paper’s abstract:
A Chow test for structural breaks in the surface temperature series is used to investigate two common claims about global warming. Quirk (2009) proposed that the increase in Australian temperature from 1910 to the present was largely confined to a regime-shift in the Pacific Decadal Oscillation (PDO) between 1976 and 1979. The test finds a step change in both Australian and global temperature trends in 1978 (HadCRU3GL), and in Australian rainfall in 1982 with flat temperatures before and after. Easterling & Wehner (2009) claimed that singling out the apparent flatness in global temperature since 1997 is ’cherry picking’ to reinforce an arbitrary point of view. On the contrary, we find evidence for a significant change in the temperature series around 1997, corroborated with evidence of a coincident oceanographic regime-shift. We use the trends between these significant change points to generate a forecast of future global temperature under specific assumptions.
Read the rest of the article here.
E.M. Smith
Always bet on the engineer over the theoretician.
Spot on.
This is exactly why, when I switched from science to music as a carreer, I went to a music college (conservatoire), not a university.
My first piano teacher there was an international competition winner; the other 2 had been child prodigies (one in Britain, one in Greece).
One of my pupils went to Oxford. His considerable practical skills counted zero to his degree (1st).
I remember when the universities were complaining about maths students who had no ‘feeling’ for numbers. Perhaps this is why the AGWers can’t spot the silly answers.
PS. The science I did learn was invaluable, technically, when I had to start teaching myself – the thinking process.
E.M.Smith,
Interesting post on trading. However, what trading tools do is capture the psychology of the market and display it in graphic form, as in MACD and other oscillators, and moving averages. Thus, certain patterns play out over and over because they capture certain moments of market mood. Eg, the blow off top becomes a top because it represents, firstly extreme exuberance (the blow off), followed by the reality (there are no more buyers left), followed by profit taking (as early long positions are closed), followed by dismay (as new longs see the price fall), followed by triggering of new stop losses. It’s all very interesting, but climate is not the collective sum of human psychology.
I believe the researchers have hit on something interesting, based on chaos theory. It is known that chaotic systems have embedded repeating patterns (great attractors). If the climate is driven towards great attractors then there should be a fingerprint of patterns within it. This approach, I think, goes some way to trying to tease out this pattern using statistical tools, without attempting to define a physical mechanism.
The main weakness is that it relies on unreliable data, which in itself, may invalidate the whole exercise.
PS, got any tips for the stock market?
The Phil Jones econometric technique uses a data deficit to produce climate inflation.
But if the test is run on doctered data, then of course the results mean nothing.
I’d like to know which data for Australia were used, as we know that the CRU data for Darwin are under suspicion.
Vincent (03:45:30) : Interesting post on trading. However, what trading tools do is capture the psychology of the market and display it in graphic form,
That is the interpretation of some of the indicators. The reality is that the indicators only look at the actual data available. Opening price, daily high, daily low, and closing price. Volume. The other indicators are substantially just various ways of combining these in ever more interesting ways. (ignoring things like sentiment surveys, that are not really in the same class).
So you can interpret a sudden surge in price and volume as an over exuberance followed by an emotional let down, but that is a projection we put on the data. It is just as possible that it was a commonly shared “program trade” computer hitting the same decision point on a herd of desks… (IIRC something like 70% of all trades now are some form of computer driven trade… hard to ascribe emotions to that 😉
So some indicators are called sentiment indicators (such as RSI) , and some are called other things, but they are actually just price / volume manipulations, by and large.
One could just as easily get a “momentum” in volume from completely mechanical things (like when a trader is ‘caught out’ and gets a margin call they can not meet). CHK got slammed like this when an executive who had margined his stock, about $150 M IIRC, found that with the price cut in the market drop he could not meet the margin call. His “forced selling” in size caused a continued drop (thus more margin calls and more sales) until the price had been cut about in half. All mechanical processes (though I’m sure the guy getting his account sold out by his broker did not feel good about it ;-0 the actual process is mechanical)
So be careful with what you attribute to emotion and what the indicator actually does.
as in MACD and other oscillators, and moving averages.
And here you lump three completely different types of indicator together into one group. Bad Idea. Moving Averages smooth the trend and lag the response. All they show is price change as a function of time, smoothed. It says nothing more. Does a price rise because of good earnings? Or good news (sentiment)? Or because it was added to an index? Or from insiders buying (often in mechanically determined quantities) from fixed investment programs like 401k plans? Yes to all… So the SMA just lets you see that effect (and the accumulations of those effects). Nothing more.
MACD is just a way to get the derivative (and a somewhat sloppy one at that) on two different time scales; then try to find the “jerk” (change of acceleration over time). It is just an attempt to remove some of the time lag that the moving average puts in. So all the things that move SMA move MACD, just sooner and faster (and hopefully still smoothed).
And oscillators (there are dozens, so this is necessarily vague) are attempts to take a “signal” and make it bigger and easier to see. They put more hysteresis into some other basic indicator process to take out some of the “hard to make a decision it’s in the middle ground” state. (Generally, I don’t like oscillators, I want to know when the ‘call’ is lacking conviction…)
Again, notice none of this talks of psychology… It almost entirely speaks to changes of velocity or momentum in price (and for some indicators, volume). Properties that have significant analogs in velocity, momentum, and volume of things like air and ocean…
Thus, certain patterns play out over and over because they capture certain moments of market mood. Eg, the blow off top becomes a top because it represents, firstly extreme exuberance (the blow off), followed by the reality (there are no more buyers left), followed by profit taking (as early long positions are closed), followed by dismay (as new longs see the price fall), followed by triggering of new stop losses.
That is the interpretation put on it in trade books. And there may well be some truth to that. Or it might just be folks rationalizing the fractal nature of price changes…
Over the long haul, price tends to follow things like increased earnings. In the very short time periods, it tends to be driven by (sometimes minor) changes in trade volume. I’ve seen a stock price move by $1 on a $20 stock from ONE order (mine). It was a thinly traded stock on a slow day and the market maker got gamed into a bad move 😉
So is it REALLY that psych game you described? Or just that one reservoir of energy (stock volume) gets drained (market makers slowly selling out at ever higher prices) while another gets filled (long term investor tanking up) until a hysteresis point is reached, and at that point the energy (price) needed to get that last bit out and ‘flip the switch’ shows as a faster final rate of change?
Don’t be so quick to toss out a tool that works just because your interpretation of why it works does not map; the underlaying processes may still, in fact, be analogs.
SIDEBAR; earlier someone had said program trading with MACD always fell apart. Yup. Because MACD indicates best in markets with a relatively strong trend. When there is very little trend, it waffles badly and you get sporadic and sometimes wrong signals. In trendless markets, Slow Stochastic works better (it gives a ‘faster twitch’ that captures short “blips” better, but will trade you out of a good position in a strongly trending market). A third indicator, DMI, tells you how much trend you have. So when DMI is over 25, use MACD. Under 20, use Slow Stochastic. Between 20 and 25 they both work, but look at direction of change of DMI and trade toward the future. I.e. if at 22, headed up, swap to MACD; but at 22 headed down, use Slow Stochastic. Basically, the “isolate to test just one indicator” academic analysis will structurally fail to find what indicators work, because no indicator works for all market conditions. But a collection of indicators can.
It’s all very interesting, but climate is not the collective sum of human psychology.
Nor is the stock market, despite what the “learn to trade quick” books say. As noted above, there is a significant mechanical component.
An early observation (by Jevons and another guy before him…Herschel?) was that markets followed sunspots. The “theory” was that sunspots informed about weather, that informed about crop yields (and thus trading company profits, bread maker profits, insurance profits, etc.) There is much correlation to support this… One creepy point is that there is a “crash’ about every 10 or so years. Gee, didn’t we just have a synchronized stock market and sunspot number crash? …
So on the 10 to 50 year scale, markets tend to integrate long term reality, not human emotion, yet on the hourly basis are often more ‘random’. In fact, they look quite fractal. Which touches on what another poster noted above: There are islands of stability or trend, even in fractal things. To the extent temperatures are somewhat fractal (and stock prices are somewhat fractal) and to the extent the indicators just fine where you are in that fractal series: the same indicators will work for both. (Even if the fractal nature of price at some level is driven by some human emotion… Things as diverse as EKG / heart rhythm and blood vessel growth are fractal, so why not the product of our fractal brain structure?)
This approach, I think, goes some way to trying to tease out this pattern using statistical tools, without attempting to define a physical mechanism.
As do indicators. It is only the user that brings a “physical mechanism” interpretation, not the tool.
The main weakness is that it relies on unreliable data, which in itself, may invalidate the whole exercise.
Interesting enough, most indicators are modestly robust to bad data. It’s fairly common in stock trading.
PS, got any tips for the stock market?
Yes: Never trade on tips in the stock market.
No, honestly.
If you are not willing to learn what to do, and why to do it, you will just get killed. You are trying to get someone else to keep your spouse happy for you. It does not turn out well because your spouse wants YOUR attention, or will turn on you: one way or the other.
I do put up a posting (nominally once a week, but when things are ‘uninteresting’ it can stretch out to two or three) of what I’m doing so folks can learn how to do it if they wish. But you can’t have someone else eat your dinner for you, nor visit the “little room” for you; nor tell you what to buy…
At least in part because the most important decision is when and what to sell, not what to buy. And even for buying: “when” is more important that “what”. (Again, exactly contrary to what investing books tell you. Trading is NOT investing. Though Ben Stein has a good book on this, IIRC it is “Yes, You CAN Time the Market”)
If you just want something to buy and forget about for a long time, you want active management. A good mutual fund (or a company like Warren Buffets investment vehicle Birkshire Hathaway BRKA / BRKB ) A simple S&P 500 index fund beats about 65% to 75% of all money managers and strategies, so it is your benchmark. When in doubt, use it (ticker SPY or just about every company on the planet has a fund) and only move to something else when very well justified.
But even there, the 10 year ‘business cycle’ really means you ought to be ‘in 8 years, out 2’ (“Out 2” ended a bit ago at the market crash bottom).
What most folks do, though, is exactly wrong. Like right now just about everyone has stuck their money in bonds or annuities. Just in time for the Fed rate to be near zero. It can’t exactly go down much from here, so we know it will be going up next. And that kills bond prices. So all the folks who rand to bonds and muni’s are going to get whacked again as the FED raises interest rates… (Now is the time to be selling bonds, not buying them. Oh well…) Bonds ARE NOT SAFE and can drop a lot in value as inflation kicks in and the Fed raises rates. Bond FUNDS are worse as a fund never matures so you can’t just wait for a return of principle at maturity.
If you want to know more, hit my site. In my experience, most folks claim they want to know how to invest, but in reality just want lazy gain. Well you can’t get excess gain while being lazy… But since most folks don’t want to learn the complicated bits, it would be inappropriate to put more here and bore everyone else.
But a final thought: Just remember that prices are a somewhat chaotic and fractal time series of Min and Max. In that context, that a tool developed to deal with that, might also work in another somewhat chaotic and fractal time series of Min and Max is not so far fetched…
“Vincent (03:45:30) :got any tips for the stock market? ”
When your colleague tells you at the coffee machine that oil is getting scarse and can only go up in price from now on: Sell.
“Grant Hillemeyer (21:19:46) :
First, thank you, Anthony for this great website. It has been for me as a layman a good jumping point for the literature regarding the great debate. As for this prediction and others like it they are not very useful bordering on the laughable. I do have one question for you, I have looked far and wide, what is the evidence of a corelation between climate warming and CO2 levels?”
Some notes of mine. Ferdinand is an expert on CO2 measurement AFAIK.
http://wattsupwiththat.com/2010/01/01/no-increase-of-atmospheric-carbon-dioxide-fraction-in-past-160-years
Ferdinand Engelbeen (17:36:40) :
There is an extreme good correlation between accumulated emissions and the increase in the atmosphere over the past 46 years.
In itself a strong indication that the emissions cause the increase, as there is no natural process capable to follow the
emissions in ratio to such a degree:
http://www.ferdinand-engelbeen.be/klimaat/klim_img/acc_co2_1960_2006.jpg
The graph is for Mauna Loa and the South Pole trends, as these show that the increasing emissions in the NH give an
increase in lag of the SH CO2 trend.
The trends compared with the temperature trend over the period 1900-2004 (CO2 levels in ice cores up to 1960):
http://www.ferdinand-engelbeen.be/klimaat/klim_img/temp_emiss_increase.jpg
Which shows that temperature is not the driving force for CO2 levels: CO2 still increases in ratio with the emissions,
even if the temperature doesn’t increase in the 1945-1975 period and in the current period since 1998.
Breakdown of correlation Temp – Co2:
http://www.c3headlines.com/2010/01/satellite-confirms-that-global-temps-continue-decline-trend-a-minus-151f-per-century-rate.html
How can you call an unexplained trend change analysis as a model and how many unexplained trend changes are we going to have in the next 40 years? And if these unexplained trend changes werent predictable before, how can you be sure that it will then warm up again as predicted by the old disgraced models. This looks like a way to save AGW face until we are all dead.
Richard Holle:
I would hope people would not just roll their eyes when reading your post and go on to the next one. I had the good fortune to listen to presentations by Theodor Landscheidt in the early 90s. I also was able to visit with him after the presentations. I suspect that there are a fair number of scientists who were relieved by his death, as he predicted the last three El Ninos prior to his death with no predictions that did not occur. As to how this pertains to the subject of this thread, I suspect that when more research is done, it probably will be realized that the forces he studied are responsible for most of the climate variations.
One more prediction I distinctly remember. In the early 90s he said that equity yields would be low worldwide due to high inflation around 2011.
After visiting with him, I was encouraged to write an article on cycles in tree ring data in the Midwestern US. It was published in CYCLES magazine. He responded to me that he thought the article was interesting. If the major cycle I found is valid, the corn belt should see serious drought in the years surrounding 2014-2015. I assume then that the global warming alarmists will be very vocal about how we have resumed the upward temperature trend.
E.M.Smith:
You make some very good observations. I have made a substantial part of my living modeling psychology in the markets I study. (Not the stock market, however, as I feel I have no real edge.) I do know that it is the psychology that I model. I discovered that what I had been doing for many years was the same as what Soros called reflexivity. But, I also have been fascinated by seeing very similar patterns in other seemingly unrelated areas, like weather. One example I use when discussing this is how things like a thunderstorm is similar to how markets work, and how divorces unfold. Something starts small, slowly building, and at some point it begins to gather energy at an ever increasing rate. There is a feedback loop driving it until it exhausts itself. I suspect this exponential growth is common in almost everything, as Ray Kurzweil wrote about so well in “The Singularity is Near.” (BTW, I highly recommend that book to anyone interested in climate science, if for no other reason than to show clearly how linear thinking is so invalid today. I also highly recommend James Gleick’s book “CHAOS: Making a New Science.” Both are must reading, IMHO.)
I might disagree with you on using the DMI in conjunction with the MACD. Again, trying to quantify these in NeuroShell Trader is extremely easy – and very frustrating. But market analysis is very similar to climate analysis in this way, isn’t it? The key is to predict when a market (climate) is going to trend or not. (Of course, first we need to define what a trend is, don’t we.) But if we can predict a binary filter – zero or one – trend or no-trend, the rest is trivial (as to the tools we use).
I enjoyed your observations. Thanks.
Saddly all based on the HadCRU results now known to be fudged.
George E. Smith (15:13:23) :
It might be useful to note that the last time that atmospheric CO2 levels were as low as they are today, was about 267 million years ago; go figure.
Yes and the plants DO NOT like low, they much prefer 1000ppm. The idea that the Vostok Ice cores show the correct geological range of CO2 from below 200ppm to 300ppm is laughable. As someone here once pointed out if the CO2 levels were below 200ppm during the last Ice Age all there would be is grass because everything else would have died out. I have seen information stating growth stops at 180ppm and all plants are dead by 150ppm with grasses being the most hardy and trees the least.
Stomatal response to CO2 comes up with much higher numbers than the Ice Cores.
Changes in the number of stomata suggest atmospheric CO2 concentrations duringan the Early Devonian had a high of 10-12 times its present value. see: http://aob.oxfordjournals.org/cgi/content/abstract/76/4/389
Charles Rossi (15:44:35) :
Look where mathematics got us in the financial world!
REPLY:
The same place it gets us with Cap & Trade…. Richer bankers and poorer peons.
Who do you think was behind the Enviromental/AGW scam in the first place? The banker/oilmen of course. Tie up the resources (scarcity), regulate the competition (pack the agencies with YOUR puppets) and now the crowning piece – Cap & Trade.
Actually it has been an absolutely brilliant piece of public manipulation and has netted certain folks millions if not billions. They must be really ticked about Climategate and all the snowstorms.
Anyone know what the units of Temperature are?
This chart seems to indicate that they are in Temperature anomaly units.
Plot you data against absolute units folks. The only reason this chart looks even remotely interesting is that we have zoomed in on it so heavily.
Starting from a consistent zero point and maintaining ranges is something we need to do to make the science in this are better. I just read a newer post about analysis of temperature data comparing Urban and Rural temperatures and the ranges were different. Glancing at the graph you get a the impression they are close to the same, but because the ranges were different they really said 2 different things.
Charting is a wonderful tool for finding data. It is also a wonderful tool for fooling yourself.
The greatest sin the climatologists have committed is fooling themselves with their charts of data on nonsensical axes.
Every engineer here knows what happens to calculations if you use non-absolute temperatures — you get nonsensical answers (unless of course you are dealing with delta t problems.
Kermit (15:59:19) :
Not only do they not have good data for either the independent variables or the dependent variable, they don’t even know what the independent variables should be in the models.
Boy that is the best one sentence summations of “Climate Science” I have seen yet. A very pretty way of saying garbage in garbage out.
Thanks, E.M.Smith, for the considered reply to my posting. You have made me reconsider my original thoughts on indicators, which is a good thing. In hindsight, it was over simplistic to describe the market as just the sum of human emotions, even though emotions are a part. It is a very complex thing and I don’t expect to reach your level of understanding. I do trade on the NYSE but I seem to spend most of the day on WUWT, so I guess this is where my interest lies.
I will take a look at your site however, and see if I can learn anything.
EVERYONE LISTEN UP:
Richard Holle (23:18:58) “We will never be able to sort out their compounded signals, if we just look at the larger forces at work as background noise […]”
AGREE 100%.
Kermit (08:08:22) “I had the good fortune to listen to presentations by Theodor Landscheidt in the early 90s. I also was able to visit with him after the presentations. I suspect that there are a fair number of scientists who were relieved by his death […]”
Agree.
Kermit (08:08:22) “[…] as he predicted the last three El Ninos prior to his death with no predictions that did not occur. As to how this pertains to the subject of this thread, I suspect that when more research is done, it probably will be realized that the forces he studied are responsible for most of the climate variations.”
Instead of “forces” & “responsible”, I would say “variables” & “related” – there’s so much confounding …and it’s not necessarily physicists who have the skills & background to initially pull the patterns together.
Kermit (08:08:22) “Ray Kurzweil […] “The Singularity is Near.” […] if for no other reason than to show clearly how linear thinking is so invalid today.”
It’s good to see so many people cluing in.
Kermit (08:08:22) “[…] James Gleick’s book “CHAOS: Making a New Science.””
What is needed soon is a book that illustrates (for a lay audience) the utility of recurrence & wavelet methods in deciphering the deterministic aspects of chaos. There need to be chapters on conditioning and deeper chapters on complex conditioning, illustrating many examples of how pure determinism generates low linear [amplitude] correlation. It needs to be emphasized that complex conditioning often looks like chaos. Phase-aware analysis methods need to go mainstream – and soon. Perhaps paradox can be rendered approachable by talented, patient communicators.
If you want thrill, chills, and unending (ten+ years and counting) melodrama, take a look at Rambus (RMBS), a company involved in a “Jarndyce and Jarndyce”-type litigation. Here’s a link to the wild and woolly Rambus discussion page on Investors Village:
http://investorvillage.com/smbd.asp?v=1&category=R7&dValue=&rValue=&nmValue=&pmValue=&nhValue=&NewCount=&topicsOnly=1&pt=m&mb=3666&SearchFor=&Subject=&DatePostedMin=&DatePostedMax=&RecommendedBy=&AuthoredBy=&MinRecs=&FilterType=Top+100+Recs%3A+7+Days
Is it just me, or does the graph in this post indeed show that the temperatures from the last 10 years fall in general above the extrapolated trendline from the 20 years before? So there’s nothing strange about the last 10 years of temperature development? And that this doesn’t suggest at all that we should expect a pause in the warming (let alone a decline)?
Same conclusion more or less as here: http://tamino.wordpress.com/2009/12/07/riddle-me-this/
phlogiston:
“This reminds me of Asterix and the Soothsayer – the bogus soothsayer’s favourite quote was “this I had forseen” after everything that happened. The peasant villagers were in awe of him!”
I hope we’ll soon get to the part where everybody’s finally convinced he’s a fraud, and he is asked “did you foresee this?” as he is attacked.
The test is not unknown in statistics. Basically, a significance test between the slopes of two regression lines to determine whether the underlying process has shifted. (It’s easy if one of the lines is slope=0. All sorts of processes experience phase shifts.