Guest Post by Willis Eschenbach
Among the papers in the Copernicus Special Issue of Pattern Recognition in Physics we find a paper from R. J. Salvador in which he says he has developed A mathematical model of the sunspot cycle for the past 1000 yr. Setting aside the difficulties of verification of sunspot numbers for say the year 1066, let’s look at how well their model can replicate the more recent record of last few centuries.
Figure 1. The comparison of the Salvador model (red line) and the sunspot record since 1750. Sunspot data is from NASA, kudos to the author for identifying the data.
Dang, that’s impressive … so what’s not to like?
Well, what’s not to like is that this is just another curve fitting exercise. As old Joe Fourier pointed out, any arbitrary wave form can be broken down into a superposition (addition) of a number of underlying sine waves. So it should not be a surprise that Mr. Salvador has also been able to do that …
However, it should also not be a surprise that this doesn’t mean anything. The problem is that no matter how well we can replicate the past with this method, it doesn’t mean that we can then predict the future. As the advertisements for stock brokers say, “Past performance is no guarantee of future success”.
One interesting question in all of this is the following: how many independent tunable parameters did the author have to use in order to get this fit?
Well, here’s the equation that he used … the sunspot number is the absolute value of
Figure 2. The Salvador Model. Unfortunately, in the paper he does not reveal the secret values of the parameters. However, he says you can email him if you want to know them. I passed on the opportunity.
So … how many parameters is he using? Well, we have P1, P2, P3, P4, F1, F2, F3, F4, N1, N2, N3, N4, N5, N6, N7, N8, L1, L2, L3, and L4 … plus the six decimal parameters, 0.322, 0.316, 0.284, 0.299, 0.00501, and 0.0351.
Now, that’s twenty tunable parameters, plus the six decimal parameters … plus of course the free choice of the form of the equation.
With twenty tunable parameters plus free choice of equation, is there anyone who is still surprised that he can get a fairly good match to the past? With that many degrees of freedom, you could make the proverbial elephant dance …
Now, could it actually be possible that his magic method will predict the future? Possible, I suppose so. Probable? No way. Look, I’ve done dozens and dozens and dozens of such analyses … and what I’ve found out is that past performance is assuredly no guarantee of future success.
So, is there a way to determine if such a method is any good? Sure. Not only is there such a method, but it’s a simple method, and we have discussed the method here on WUWT. And not only have we discussed the testing method, we’ve discussed the method with various of the authors of the Special Issue … to no avail, so it seems.
The way to test this kind of model is bozo-simple. Divide the data into the first half and the second half. Train your model using only the first half of the data. Then see how it performs on the second half, what’s called the “out of sample” data.
Then do it the other way around. You train the model on the second half, and see how it does on the first half, the new out-of-sample data. If you want, as a final check you can do the training on the middle half, and see how it works on the early and late data.
I would be shocked if the author’s model could pass that test. Why? Because if it could be done, it could be done easily and cleanly by a simple Fourier analysis. And if you think scientists haven’t tried Fourier analysis to predict the future evolution of the sunspot record, think again. Humans are much more curious than that.
In fact, the Salvador model shown in Figure 2 above is like a stone-age version of a Fourier analysis. But instead of simply decomposing the data into the simple underlying orthogonal sine waves, it decomposes the data into some incredibly complex function of cosines of the ratio of cosines and the like … which of course could be replaced by the equivalent and much simpler Fourier sine waves.
But neither one of them, the Fourier model or the Salvador model, can predict the future evolution of the sunspot cycles. Nature is simply not that simple.
I bring up this study in part to point out that it’s like a Fred Flintstone version of a Fourier analysis, using no less than twenty tunable parameters, that has not been tested out-of-sample.
More importantly, I bring it up to show the appalling lack of peer review in the Copernicus Special Issue. There is no way that such a tuned, adjustable parameter model should have been published without being tested using out of sample data. The fact that the reviewers did not require that testing shows the abysmal level of peer review for the Special Issue.
w.
UPDATE: Greg Goodman in the comments points out that they appear to have done out-of-sample tests … but unfortunately, either they didn’t measure or they didn’t report any results of the tests, which means the method is still untested. At least where I come from, “test” in this sense means measure, compare, and report the results for the in-sample and the out-of-sample tests. Unless I missed it, nothing like that appears in the paper.
NOTE: If you disagree with me or anyone else, please QUOTE WHAT YOU DISAGREE WITH, and let us know exactly where you think it went off the rails.
NOTE: The equation I show above is the complete all-in-one equation. In the Salvador paper, it is not shown in that form, but as a set of equations that are composed of the overall equation, plus equations for each of the underlying composite parameters. The Mathematica code to convert his set of equations into the single equation shown in Figure 2 is here.
BONUS QUESTION: What the heck does the note in Figure 1 mean when it says “The R^2 for the data from 1749 to 2013 is 0.85 with radiocarbon dating in the correlation.”? Where is the radiocarbon dating? All I see is the NASA data and the model.
BONUS MISTAKE: In the abstract, not buried in the paper but in the abstract, the author makes the following astounding claim:
The model is a slowly changing chaotic system with patterns that are never repeated in exactly the same way.
Say what? His model is not chaotic in the slightest. It is totally deterministic, and will assuredly repeat in exactly the same way after some unknown period of time.
Sheesh … they claim this was edited and peer reviewed? The paper says:
Edited by: N.-A. Mörner
Reviewed by: H. Jelbring and one anonymous referee
Ah, well … as I said before, I’d have pulled the plug on the journal for scientific reasons, and that’s just one more example.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
RJ actually did do a forecast with a fraction of the data and he doesn’t claim his model is right (as it isn’t — it fails the simplest diagnostics). RJ does these models for fun. He’s not a political activist. It’s unfortunate that he got tangled in this whole PRP mess. I would have advised him to steer well-clear of publishing in PRP had I known he was doing so, as it has been obvious for many months that a blow-out like this was going to be inevitable. (Anthony: I don’t know how you didn’t see it coming. You must have had blinders on.)
Dr. Svalgaard and Willis, since the purpose of these exercises ultimately, is to predict temperature changes by using solar phenomena, I ask, a non-scientist, how accurate a proxy for solar activity are sunspots? If they are not accurate, are there other proxies for solar activity besides sunspots? If such phenomena exist, have there been efforts to try to correlate these other solar phenomena with temperatures, if so, how credible are they? Thanks.
Chuck L says:
January 22, 2014 at 6:28 am
how accurate a proxy for solar activity are sunspots?
The microwave flux from the Sun is a good index of solar activity and the modern sunspot number is a good proxy for the flux: http://www.leif.org/research/SHINE-2010-Microwave-Flux.pdf
There are indications that over the past decade the official sunspot number has been a bit too low compared to the flux, but that is a second order effect.
Underlying physical model — No
A few well defined fitting parameters — No
A load of feces — Yes
Willis says “The way to test this kind of model is bozo-simple. Divide the data into the first half and the second half. Train your model using only the first half of the data. Then see how it performs on the second half, what’s called the “out of sample” data.
I bring up this study in part to point out that it’s like a Fred Flintstone version of a Fourier analysis, using no less than twenty tunable parameters, that has not been tested out-of-sample.
More importantly, I bring it up to show the appalling lack of peer review in the Copernicus Special Issue. ”
From the paper :
4 Forecasting
To test if the model has forecasting ability, we can redo the
correlation with data only up to the years 1950 and 1900 and
determine the forecast for the next 50 and 100 yr to see if the
model can predict the sunspot data we have already experi-
enced.
Figure 5 gives a forecast for the period 1950 to 2050 made
from the correlation of the model with data up to 1950.
“Figure 6. A comparison of monthly sunspot numbers from 1900 to
2000 (in blue) with the absolute value of the correlation model (in
red), derived using data only up to 1900 and the extended forecast
to 2000.”
Jeezus Willis you’re at it again. Read the frigging paper before shouting off on WUWT.
I’m not saying I find this very convincing but if you want to rip something apart at least read it first.
“In fact, the Salvador model shown in Figure 2 above is like a stone-age version of a Fourier analysis. But instead of simply decomposing the data into the simple underlying orthogonal sine waves, it decomposes the data into some incredibly complex function of cosines of the ratio of cosines and the like … which of course could be replaced by the equivalent and much simpler Fourier sine waves.”
Willis you are in danger of talking above you pay grade.
If you have a modulation of two cosines, it will appear as three peaks in a Fourier spectrum. Each with its own phase amplitude and frequency. You could then link some of the phase and amplitude terms so avoid needing extra parameters but it certainly would not be “much simpler”.
http://climategrog.wordpress.com/2013/09/08/amplitude-modulation-triplets/
Willis,
You stated in the main text of your article that the formula you wrote was the one used by Slavador. Putting a contrary statement in the footnote? Without even a marker in the main text that there was a note to your “handiwork”?
And it’s my problem that I missed the note in the end credits following the listings of Best Boy and Fluffers?
You couldn’t simply have written: “The author’s equations can be expanded to …” in the main text.
I don’t need no steenking Mathematica to do simple algebraic substitution and expansion. Not that I would begin to do so in this case because it’s superfluous effort and it obscures the physical parameters. Parameters which, if left “pristine”, provide additional insight while working with equations. (*)
FWIW: I’d have been happier if Salvador had left symbols to represent each of the period components in his equation and if he’d not referred to “years” as frequencies.
I wasn’t “impressed” by the number of parameters. After all, you need that many to describe the simplified physical behaviour of the system with that many “degrees of freedom”. They’re not arbitrary parameters, They’re derived from measurements of the physical world.
Salvador was looking for particular “spectral content” within the sunspot record at the frequencies of interest. If the number of significant degrees of freedom and their relative “directions” (characteristic frequencies and phases) is wrong, then the other parameters are likely to be substantially different for different sample sets of sunspot data. Salvador mentions that his model won’t work without considering the 21.005 quarter Uranus period.
mentions the difficulty with being accurate regarding the longer cycles as the detailed sunspot record is comparatively short… the longest period in his analysis is 1253 years.
A Fourier analysis would have told Salvador nothing about the physical world. His method tests a hypothesis which has some physical basis in the real world. It’s not just a “wiggle match”.
(*) I knew the origin of the elephant quote from “offline” sources. IIRC, von Neuman also urged Feynman to not lose sight of the physics when ploughing through formulae.
P.S. The Figure 1 carbon 14 data (not dating) illustrates a proxy. (Let Willlliam Connnnolllley illustrate it and Tallbloke set it straight.)
Notwithstanding the above, I agree with Willis’ view that this is largely curve fitting. However, the out of data tests do suggest there may be something worth further study.
As I’ve pointed out to a couple of this team in personal communication (not this author) there is a danger of this kind of approach becoming numerology if you are ready to arbitrarily accept any combination harmonics sub-harmonics beats, resonances and amplitude modulations of any planetary periods.
I might buy the possibility of Ian Wilson’s VEJ idea but when I see
“– one-quarter Uranus orbital frequency equal to 21.005
–two modulating frequencies of 178.8 and 1253 (forming
a beat frequency of 208 yr).”
I start to to think , hang on. Without a concrete reason to suggest a link with U and then why the 4th harmonic but not 1st , 2nd or 3rd , the word numerology springs to mind.
You need to write out all the base frequencies all the possible permutations within in your scheme of combinations and you’d probably realise that you have enough numbers to start your number system. In that context the “planetary” constants are essentially random numbers.
¹⁴C has been used as a proxy for solar activity for a while. (Science: Changes in atmospheric carbon-14 attributed to a variable sun. 1980)
It’s data. Not dating.
Greg Goodman says:
January 22, 2014 at 6:48 am
Figure 5 gives a forecast for the period 1950 to 2050…
“Figure 6. A comparison of monthly sunspot numbers from 1900 to
2000 (in blue)…
Both look like failures to me.
Thanks Willis. A good article on a difficult subject.
I know of many methods to get a small signal out of a lot of noise, and it’s a good thing that they can be rapidly changed and transformed until one seems to work.
To prove that your noise-reduction black box is working you extract something meaningful, like a conversation. But if to extract that conversation you must have a transcript of it before it occurs, then your black box is useless.
How about getting funding for your black box factory?
lsvalgaard says: Both look like failures to me.
Clearly the fit is not as good. My point was the W was ripping into both the authors and the editors in no uncertain terms, for not doing something that was in fact in the paper.
I should add that most probably, predicting solar cycles is as difficult as predicting weather or climate cycles.
Also that difficult is no enough of a word, this is a convoluted and often obfuscated subject.
Willis has Occam’s Axe.
It’s an “open access” journal. This means it is a way of extracting “publication fees” from people who otherwise can’t survive a legitimate peer review. In any case, the journal is being pulled off the virtual “shelf” probably because it is too obviously fraudulent
http://www.pattern-recogn-phys.net/volumes_and_issues.html
The standard model has a similar problem.
Tallbloke, as you’re watching, can you enlighten us on the Uranus thing?
Dan, no, it doesn’t necessarily mean that, there are excellent open access journals that are free for both author and reader, e.g. http://jmlr.org/ .
Excellent demolition job! Unfortunately, it means CAT scans and MRI’s don’t mean anything, either.
This is an excellent example of curve fitting. This article should be used as an example for students in mathematics and statistics.
Curve fitting is a very common error to make. It is easy to believe that you have found a good formula for predicting the future when you in reality have just performed curve fitting on historical data.
I remember when I worked in Telecom that a small company called us and told that they had found an unbelievable accurate formula for projecting future telephone traffic for each telephone trunk.
I watched them present their solution, and to no surprise, it was a curve fitting exercise just like this.
A journal which publishes things like this cannot be called scientific.
/Jan
Greg Goodman says:
January 22, 2014 at 6:51 am
In that context the “planetary” constants are essentially random numbers.
=============
Kepler tells us otherwise.
To the extend the solar cycles are cyclical, curve fitting can have predictive value. The three best known examples are perhaps the day-night cycle, the cycle of annual seasons, and the cycle of ocean tides. These were all understood first as a result of curve fitting, long before we understood the underlying process.
The ocean tides are perhaps the most informative, whereby we are able to accurately calculate the future state of a chaotic system years in advance. This doesn’t mean the current paper has it right. Only that it might. My suspicion is that the curves may be over fit. To my eye there likely should be more noise in the observations vs the model.
dikranmarsupial,
“Open Access” is a “wild west” environment for authors and readers alike. Maybe some are OK, but many are just scams that extract “fees” and publish junk – a lucrative business model given the oversupply of Ph.D.s who will sell their soul to beef up their CVs. My problem with “open access” is the same as the internet in general: lots of interesting stuff but you have to know how to avoid wasting time on the incredibly larger amount of junk masquerading as quality material. Maybe we need Google to invent an algorithm to score “open access” papers (a substitute for peer review) in order to assist in this process of finding the quality needles in the paper glut haystack.
dikranmarsupial, fair point but the same is now true of PR with the tens of thousands of journals spewing out hundreds of papers each every year.
Even the previously “highly respected” titles now print garbage, so science is basically screwed.
fredberple: Kepler tells us otherwise.
No fred, you missed the point. I’m talking about the plethora of beats, interferences, periods, harmonics and sub-harmonics that are used as feed stock in this kind of fitting exercise.
If you generate hundreds of “planetary constants” and then pick half a dozen at will in parameter fitting, it just like a quantised free parameter. There will always be something near enough to get a decent fit.
Jan “Curve fitting is a very common error to make. It is easy to believe that you have found a good formula for predicting the future when you in reality have just performed curve fitting on historical data. ”
I can think of much more widely known example : the models use for AR4.
It’s the same problem except that they can’t eve back cast properly.
lsvalgaard says:
January 22, 2014 at 6:36 am
“The microwave flux from the Sun is a good index of solar activity and the modern sunspot number is a good proxy for the flux: http://www.leif.org/research/SHINE-2010-Microwave-Flux.pdf
There are indications that over the past decade the official sunspot number has been a bit too low compared to the flux, but that is a second order effect.”
Thanks, Dr. S. Interesting presentation.
Dan/Greg, there are good journals, there are no so good journals, that was as true before the push for open access as it is now. Part of being a good researcher is knowing which journals to monitor for interesting papers and which journals are worth sending papers to. Science is actually in reasonably good shape, or at least it would be if the funding were a bit better, but we have had to put up with the financial downturn, just like everybody else, the money has to come from somewhere.
The real problem (IMHO) is researchers being assessed using metrics that favour quantity too strongly over quality. If there was no reward for publishing in a low quality predatory open access journal, there would be no low qualty predatory open access journals. The problem is that quality is much less easily assessed than quantity.