Sunspots and Sea Level

Sorry Paul Westerhagen.
I’ve just submitted my comment as you submitted yours.
However your’s tells it all.

vukcevic,
I speculate that what happens on the earth may be the result of solar activity, Radiation, solar wind, solar farts, etc… all that stuff, seen and unseen and yet-to-be-discovered.
Is there a good study of cloud cover (anywhere/everywhere) vs sunspot number (or a proxy)?
I suspect that there is a transfer function of some kind between solar activity and terrestrial variables, like sea level change etc. Maybe the TF is an aggregate of all of these. Why not?
PW

A high autocorrelation means two adjacent data points are almost the same [and so are not independent] so that when you think you have 1000 data points you may only have [say] 50 points. this affects the p-value making you believe that the correlation is more significant than it really is. A good example is the sunspot number: in a solar cycle there are 4000 days and thus 4000 daily values of the sunspot number, but since a high sunspot number on a given day is always followed by a high number the next day, the number of independent data points for a whole cycle is only about 25, not 4000.
Excellent summary. I wrote a whole Phys. Rev. paper on this once upon a time, elaborating (among other things) how one can actually determine the autocorrelation time (or number of samples per “independent” sample) by looking at the scaling of the variance of the data. It is easy to make egregious claims for the precision of an entirely spurious experimental result if one has thousands of samples but the scaling of the variance indicates that you really only have a hundred.
I was actually doing it the other way around — I was doing numerical simulations using (and comparing) several Markov Chain processes — Metropolis, Heat Bath, a cluster/metropolis method and a mixed cluster/heat bath method. Metropolis was cheap and generated enormous numbers of samples, but the samples had an enormous autocorrelation as one might expect. Heat bath did better — it isn’t accept/reject and so every spin in the model was changed in every step. Cluster methods alone did break up spatial correlations quickly but — surprise! They were accept-reject AND preserved large blocks of spin bonds in unchanged states per move, and actually slowed down the decay of energy autocorrelation (accept/reject methods more or less guarantee that SOME bonds in a lattice are not thermalized for many, many sweeps). Best of all was a mix of cluster steps to break up egregious spatial correlations at a variety of length scales and heat bath sweeps to facilitate the ergodic progression of the bond energies. All revealed by comparing how the standard deviation of the sample compared to what one would expect from a knowledge of the system variance and the number of samples in question.
This is one reason I’m very skeptical about climate science conclusions in general. There isn’t one autocorrelation time or important time scale in climate science, there are dozens (at least) — there is probably a continuum of them to where Laplace transforms are more appropriate to speak of than an single exponential decay process. And it isn’t clear that these times themselves are stationary; it may not even be that the usual concept of an “autocorrelation time” HOLDS for a climate system, even in an approximate sense.
That’s why I read papers on Hurst-Kolmogorov statistics with great interest. Many climate variables appear to exhibit an H-K pattern of autocorrelation — stretches of order decades of approximately uniform variation followed by a discrete jump to a new stretch of order decades. “The pause” is just such a stretch in the GASTA , where the 1997/1998 Super-ENSO is coincident with the preceding jump, and where quite a few of these intervals are visible e.g. here:
http://www.woodfortrees.org/plot/hadcrut4gl/from:1940/to:2014
or even more clearly here:
http://www.woodfortrees.org/plot/uah/from:1977/to:2014
or here:
http://www.woodfortrees.org/plot/rss/from:1977/to:2014
In the satellite era in the last two of these curves, note well that one does quite well imagining that the climate was locally stable from 1980 to 1998 give or take a year or two — call it twenty years. Then it jumped by around 0.3 C over a very short period — two or three years, with large oscillations attending the jump — and has been stable since for anywhere from 13 to 17 years depending on just where you want to put the jump and whether you want to discretize it or extend it over a 2-4 year period that includes the initial very sharp rise, the overcorrection, and the smaller scale bounces that could just as easily belong to the new equilibrium.
Similar jumps are clearly visible in SST graphs generated by Bob Tisdale. Similar jumps are clearly visible in rainfall patterns (one of the places where they were first observed and analyzed by hydrologists like Koutsoyiannis).
Guestimating (looking at longer timeseries and allowing for both poorer data and various thumbs on the scales) 15 to 25 years is one highly non-exponential autocorrelation time — the Earth seems to like to spend 1-2 decades going only slowly up or slowly down interspersed with intervals where it goes up or down comparatively rapidly. Much of even this cannot really be trusted — IMO global climate data itself has a clear trend of decreasing reliability as one goes back into the past (that no one ever gets to see as no one prints the curves with even guestimated error bars, but the estimated errors in things like HADCRUT4 now are pretty large, on the close order of the 30 year supposed warming “signal” of a few tenths of a degree).
If one reduced the last 36-odd years of satellite data — the most reliable data source out there for an unbiased global temperature anomaly — to the two independent “climate” samples it probably represents, and decorated each of those points with an error estimate of 0.1 to 0.2 C (based solely on the variance around the sample means of the observational data taken in two 18 year chunks) nobody would dare to say anything about a trend in the climate. One point at an anomaly of C centered on 1986, one point at an anomaly of C centered on 2005. The probable error (accounting for the scaled autocorrelation and hence sample independence on a smaller scale WITHIN these intervals pretty much completely overlaps and suggests a mean warming of perhaps 0.2/C over 18 years, or roughly 1.1 C/century if it made the slightest bit of sense to extrapolate two data points in any graph ever built!
The problem here is that we really do not know the various climate autocorrelation times (plural) and so we do not know how much “significance” to assign to any apparent linear trend in the highly correlated data with its own internal dynamics that is essentially noisy but stable for apparent decade-plus intervals.
To conclude, I hate to see Bayesian reasoning bad-mouthed, so I’ll just point out that the correct application of Bayesian reasoning in problems like this is along the lines of the stuff Willis was discussing, to automagically correct one’s initial prior estimates of probability until they are in asymptotic agreement with the observational data.
For example, one might examine as he did flipping a two-sided coin. You might begin with a strong bias, an experience-based belief that two-sided coins are likely to have a probability 0.5 for heads, 1 – 0.5 for tails, and 0 for landing perfectly on edge, making them perfect Bernoulli Trial objects with a nice binomial distribution of outcomes.
Bayesian analysis in principle gives one a way to systematically and nearly smoothly correct this belief on the basis of data as one starts to actually flip the coin. Initially one’s prior for heads might be 0.5, but after observing 78 flips in 100 flips, one would/should have a posterior probability that is much higher than 0.5. This is the flip side of ordinary p-values. The Binomial distribution might allow one to compute the probability of getting 78 heads out of 100 flips of an unbiased coin — the usual definition of the p-value — but it does not tell you what your best estimate for the probability of heads is given the data.
The problem with Bayesian reasoning is that it is often misapplied or used as a means of legitimizing an improbable estimate. By weighting your prior beliefs highly, you can essentially demand a lot of evidence — probably far too much evidence — before you start bending your posterior estimate much. There are right and wrong ways to do it, but in the hands of a clever person Bayesian analysis can conceal a kind of statistical lie that resolves to “the result of my analysis shows that the probability of the event is p, very near where I expected/wanted/needed it to be”. Taleb reviews a very similar case in The Black Swan where a Scientist refuses to alter his prior estimate for the probability of a tail in the face of the observational evidence of 100 straight heads because he knows that 2^100 events can happen (one in 10^30 tries, give or take a bit) where Joe the Cab Driver immediately recognizes this as “a mugs game” — the coin is very, very probably fixed so that p(heads) is nearly unity, at least in the hands of the clever person flipping the coin.
This is very similar to the problems with concluding either correlation or causality from the data. Suppose I flipped a truly unbiased coin several thousand times. Over several thousand flips, it is very likely to see sequences of heads and tails at least 8 or 9 long. If somebody wishing to demonstrate that the coin is biased were allowed to pick samples out of this data and concentrate their analysis on some subset, all they have to do is point to one of these long sequences (which have low probability of happening in any given sequence of coin flips even as they will occur quite reliably if one honestly samples the unbiased coin) and say — LOOK — the coin produced ten highly correlated flips in a raw! The coin must be biased! Our brains make us especially susceptible to this — all of the mundane flips sequences are boring; even though the probability of getting HTTHTHHTHH. is exactly equal to the probability of getting HHHHHHHHHH, the latter is exciting and we notice it while forgetting all of the equally unlikely but less strikingly patterned sequences. Picking particular sequences of data that favor some desired conclusion is called “cherrypicking” and is a cardinal sin of science. The tendency to discover favorable sequences and cherrypick them is called “confirmation bias” and is a even worse sin as the latter can happen by accident but confirmation bias involves deliberate action e.g. presenting the average of 9 tidal stations that we know appear to have some sort of correlation but deliberately concealing — often even from ourselves — the fact that 9 stations picked at random from the full set of stations would exhibit no such visible correlation or worse, that the unbiased average of all of the data exhibits no such correlation.
Climate science (on both sides) research in much of the humanities and soft sciences are rife with confirmation bias and cherrypicking and overreaching conclusions drawn from inadequate or incorrect statistical analysis. A lovely example of the latter is the infamous hockey stick. An example of the former is D’Arrigo’s infamous testimony: “if you want to make cherry pies, you have to be willing to pick cherries”. Whether or not the misuse of statistics is deliberate or accidental, much of published climate science is statistically incompetent. A properly cautious approach would simply refuse to draw conclusions about trends, causes or effects until far, far more reliable data is collected and analyzed by somebody other than a collection of world-saving zealots — on either side. Let me be clear — we do not have sufficient evidence to reject the hypothesis of catastrophic warming by the end of the century (human caused or otherwise) any more than we have sufficient evidence to accept it. At the moment, the evidence in favor of the hypothesis is weak, but because of the uncertainties in things like autocorrelation, the range of so-called “natural” variation, the effect of a variety of processes that we know have timescales on the order of several decades (e.g. the multidecadal oscillations, the observed multidecadal variability of the sun, multidecadal oceanic turnover processes) it is far from sufficient to positively assert that CO_2-linked global warming will not, in fact, eventually cause real catastrophic e.g. sea level rise. There just isn’t any evidence that it is doing so yet.
For fans of proper Bayesian analysis, there is a delightful article here:
http://marginalrevolution.com/marginalrevolution/2005/09/why_most_publis.html
that analyzes a surprising but plausible claim: Most published research findings are false! This is probably not true in all fields (I’d like to think it isn’t true in physics, for example, because we police the field at roughly the level where any further stringency would eliminate the fringe from which new revolutions rarely but significantly spring) but in fields like medicine it is probably (and tragically) true!
It is interesting to see where climate science falls in the schema given at the bottom of this article.
1) I’d have to say climate science is very high in what he calls background noise, the number of hypotheses tested. Examples of this are frequent on WUWT — when every biologist has to add the magic words “to look for evidence of the impact of global warming” to a grant proposal to study pinkfooted bungie-jumpers, when every population study, every medical study, every study of coral reefs or the migratory patterns of beetles invokes the phrase on the presumption that such evidence exists (that has to be presented sufficiently compellingly to help influence a grant officer into funding the work when otherwise there is nothing particularly special to look for sheer Bayesian analysis dictates that a significant fraction of such studies will find something to report at the level of marginal statistical significance.
2) Many results supporting a supposed coming catastrophe are reported for tiny sample populations, or highly constrained and limited environments. Last week we heard about the supposed extinction of a subpopulation of a butterfly in a part of a single meadow subject to uncontrolled and confounding environmental alterations in addition to any supposedly discriminable change in climate. Again, many of these studies of marginal subpopulations or specific locations can easily turn out to be at best statistical accidents — happening to test green jelly beans first and never bothering to report that jelly beans in general have a null result, and even green jelly beans have a null result unless they were tested on one particular Sunday.
3) Small effects are to be distrusted. Wow. Talk about defining an entire science! The entire predicted catastrophe is less than a one percent effect on the absolute temperature scale. All observed warming to date in the sixty-odd years where it could be attributed to human-produced CO_2 is on the order of two-tenths of one percent of the absolute temperature (some fraction of which is almost certainly natural) and there has been no warming at all for the last quarter of that era precisely where one would expect it to be the strongest.
4) Multiple types of evidence are desirable, but are not forthcoming. The GCMs that are the sole basis for predictions of warming fail to predict global warming, they fail to predict tropospheric warming, they fail to predict rainfall, they predict storm intensification that has not happened, they do not predict the correct patterns of what warming there has been, and they utterly failed to predict what has happened with e.g. Antarctic sea ice, the Greenland ice pack, sea level rise, sea surface temperatures, El Nino, the PDO, and well, pretty much anything. It isn’t clear that there is anything that GCM’s have gotten right!
The problem there is that when multiple biological studies find supposed evidence for negative sequellae attributable to GW that supposedly took place over the last 15 years, that is not evidence for CAGW when in fact no statistically discernible warming whatsoever took place over the last fifteen years. So a huge body of what people consider to be “evidence” is in fact evidence — against the hypothesis, and for the presence of an enormous degree of confirmation bias in the literature given that people are finding effects for a cause that in fact has not budged since many of the researchers were in middle school.
5) This also is a problem here. Much of the literature can be discounted when they find evidence of global warming in e.g. the biosphere when no discernible global warming occurred or is occurring since the 1997-1998 ENSO, and only 0.2 C of warming occurred then. Changing “global warming” to “climate change” is intended to hide this since “climate change” cannot be falsified given that the climate is always changing, but does not repair the science. Then there is the deification of Mann by the IPCC and the consequent corruption of the mainstream science. Post-Mann climate science has been all about a literature controlled by the authors of a few individual papers, some of which have been shown to be terrible examples of statistics. Briffa, Jones and others published many papers clearly showing the MWP before Mann “erased” it. Climategate clearly revealed the naked backstabbing, gatekeeping, and behind the scenes pressure to suppress the competing voices that ordinarily keep science honest. It has taken fifteen years plus where the sky stubbornly refused to fall to gradually free the editors and referees to consider papers that don’t conform to a narrowly politicized conclusion as evidenced by the presence of one or more catch phrases about global warming even where it ends up being irrelevant (the mirror image of the grantseeking process alluded to above). One shudders to think about how long it will take to de-politicize the granting agencies themselves.
6) There are damn few papers published that test other people’s theories in climate science, at least as far as I’ve seen. If there were, would not the GCMs largely have been rejected at this point? WUWT has indeed covered at least one paper that tested GCMs directly against one another for a toy problem, where they failed miserably, but only a handful of researchers dare to question the party line, often at the expense of being called names by their peers and being subjected to a brutal and withering peer review process in journals where the editors are routinely subjected to pressure from a small cadre of climate scientists. Indeed, a properly skeptical test of the fundamental theoretical basis for the prediction of CAGW would be most welcome, as those predictions are busy not coming true and everybody knows it but nobody will say so in the literature!
7) Don’t reject papers that fail to reject the null hypothesis. Boy, Climate Science in a nutshell, at least if the null hypothesis is that human CO_2 has had a negligible effect on the climate, one impossible to disentangle so far from natural climate variation and impossible to directly observe even with sophisticated instrumentation and hence discernible at best by dubious analyses that attempt to discriminate recent warming as being due to CO_2 from previous but identical warming that was supposedly natural, e.g. comparing the first and second half of the 20th century.
If a non-expert in the field cannot tell the difference between GASTA in the first half of the 20th century (when most climate scientists agree that CO_2 levels had not yet begun to change at a significant rate from pre-industrial levels) and the second half of the 20th century (where all or almost all of the warming is openly attributed to CO_2 in IPCC ARs) then one has little basis for rejecting the null hypothesis.
rgb

Greg Goodman says:
January 24, 2014 at 4:03 pm
“I have been tyring to point for some time that you cannot detect or refute an particular driver (like solar) by trivial single variable regression of correlation.”
Throw out all the other comments on this thread. That is the only one needed.

More ad hominems.
I see what I see…

Willis,
Regarding R^2 equals correlation.
R^2 isn’t very good at helping you find components of a signal. The correlation I (and others) are referring two is of the signal processing variety (Sum of the product, not sum of the square of the differences). This allows you to be wrong in phase, in scale, and still ‘detect’ a correlation. In addition there is another type of correlation known as synchronicity. R^2 is only valid for linear models. Consider Milankovitch cycles ‘correlation’ with ice-ages. If you were to do an R^2 you’d probably get a very a very low value. This is because the Earth is a complex non-linear feedback loop system and correlation assumes a linear model. Whatever Milankovitch cycles do and how they do it, it is probably more of a ‘trigger’ then a direct driver. Blindly using R^2 you’d probably get a value so low that you’d assume that Milankovitch cycles are irrelevant. However, synchronicity is amazing. No rational person can look at the synchronicity of Milankovitch cycles and ice ages and conclude “it’s just a coincidence”.
Regarding dismissing the graph:
However, you also linked to some other data that was far less flattering for sun spots (both synchronicity and statistical correlation). As I’ve said I’m not a ‘sun spot guy’. I objected to your use of R^2 meant for linear models and dismissal of the graph above when to the eye there appears an obvious synchronicity. Does the synchronicity hold for longer periods? Who knows? I thought you were too quick to dismiss it based on linear model metrics. However, If other data shows that this paper is essentially ‘cherry picking’, then that is a different story.

Hi Willis,
re failed Shaviv 2008 p=0.54 replication.
Shaviv used Lean 2000. Is your data identical ?
Shaviv removed secular trends.
Your zscore uses column D and not detrended values in E ?
=’C:\Applications\Microsoft Office 2011\Office\Add-Ins\myfunctions2.xla’!zscore(F305:F407,D305:D407)