The Button Collector or When does trend predict future values?

So, you know now who to call if YOU loose a bu...
How many buttons will he have on Friday? (Photo credit: Wikipedia)

Guest essay By Kip Hansen

INTRO: Statistical trends never determine future values in a data set. Trends do not and cannot predict future values. If these two statements make you yawn and say “Why would anyone even have to say that? It is self-evident.” then this essay is not for you, you may go do something useful for the next few minutes while others read this. If you had any other reaction, read on. For background, you might want to read this at Andrew Revkin’s NY Times Dot Earth blog.

­­­­­­I have an acquaintance that is a fanatical button collector. He collects buttons at every chance, stores them away, thinks about them every day, reads about buttons and button collecting, spends hours every day sorting his buttons into different little boxes and bins and worries about safeguarding his buttons. Let’s call him simply The Button Collector or BC, for short.

Of course, he doesn’t really collect buttons, he collects dollars, yen, lira, British pounds sterling, escudos, pesos…you get the idea. But he never puts them to any useful purpose, neither really helping himself or helping others, so they might as well just be buttons, so I call him: The Button Collector. BC has millions and millions of buttons – plus 102. For our ease today, we’ll consistently leave off the millions and millions and we’ll say he has just the 102.

On Monday night, at 6 PM, BC counts his buttons and finds he has 102 whole buttons (we will have no half buttons here please); Tuesday night, he counts again: 104 buttons; on Wednesday night, 106. With this information, we can do wonderful statistical-ish things. We can find the average number of buttons over three days (both mean and median). Precisely 104.

We can determine the statistical trend represented by this three-day data set. It is precisely +2 buttons/day. We have no doubts, no error bars, no probabilities (we have 100% certainty for each answer).

How many buttons will there be Friday night, two days later? 

If you have answered with any number or a range of numbers, or even let a number pass through your mind, you are absolutely wrong.

The only correct answer is: We have no idea how many buttons he will have Friday night because we cannot see into the future.

But, you might argue, the trend is precisely, perfectly, scientifically statistically +2 buttons/day and two days pass, therefore there will be 110 buttons. All but the final phrase is correct, the last — “therefore there will be 110 buttons” — is wrong.

We know only the numbers of buttons counted each of the three days – the actual measurements of number of buttons. Our little three point trend is just a graphic report about some measurements. We know also, importantly, the model for the taking the measurements – exactly how we measured — a simple count of whole buttons, as in 1, 2, 3, etc..

We know how the data was arrived at (counted), but we don’t know the process by which buttons appear in or disappear from BC’s collection.

If we want to be able to have any reliable idea about future button counts, we must have a correct and complete model of this particular process of button collecting. It is really little use to us to have a generalized model of button collecting processes because we want a specific prediction about this particular process.

Investigating, by our own observation and close interrogation of BC, we find that my eccentric acquaintance has the following apparent button collecting rules:

  • He collects only whole buttons – no fractional buttons.
  • Odd numbers seem to give him the heebie-jeebies, he only adds or subtracts even numbers of buttons so that he always has an even number in the collection.
  • He never changes the total by more than 10 buttons per day.

These are all fictional rules for our example; of course, the actual details could have been anything. We then work these into a tentative model representing the details of this process.

So now that we have a model of the process; how many buttons will there be when counted on Friday, two days from now?

Our new model still predicts 110, based on trend, but the actual number on Friday was 118.

The truth being: we still didn’t know and couldn’t have known.

What we could know on Wednesday about the value on Friday:

  • We could know the maximum number of buttons – 106 plus ten twice = 126
  • We could know the minimum – 106 minus ten twice = 86
  • We could know all the other possible numbers (all even, all between 86 and 126 somewhere). I won’t bother here, but you can see it is 106+0+0, 106+0+2, 106+0+4, etc..
  • We could know the probability of the answers, some answers being the result of more than one set of choices. (such as 106+0+2 and 106+2+0)
  • We could then go on to figure five day trends, means and medians for each of the possible answers, to a high degree of precision. (We would be hampered by the non-existence of fractional-buttons and the actual set only allowing even numbers, but the trends, means and medians would be statistically precisely correct.)

What we couldn’t know:

  • How many buttons there would actually be on Friday.

Why couldn’t we know this? We couldn’t know because our model – our button collecting model – contains no information whatever about causes. We have modeled the changes, the effects, and some of the rules we could discover. We don’t know why and under what circumstances and motivations the Button Collector adds or subtracts buttons – we don’t really understand the process – BC’s button collecting because we have no data about the causes of the effects we can observe or the rules we can deduce.

And, because we know nothing about causes in our process, our model of the process, being magnificently incomplete, can make no useful predictions whatever from existing measurements.

If we were able to discover the causes effective in the process, and their relative strengths, relationships and conditions, we could improve our model of the process.

Back we go to The Button Collector and under a little stronger persuasion he reveals that he has a secret formula for determining whether or not to add or subtract the numbers of buttons previously observed and a formula for determining this. Armed with this secret formula, which is precise and immutable, we can now adjust our model of this button collecting process.

Testing our new, improved, and finally adjusted model, we run it again, pretending it is Wednesday, and see if it predicts Friday’s value. BINGO! ONLY NOW does it give us an accurate prediction of 118 (the already known actual value) – a perfect prediction of a simple, basic, wholly deterministic (if tricky and secret) process by which my eccentric acquaintance adds and subtracts buttons from his collection.

What can and must we learn from this exercise?

1. No statistical trend, no matter how precisely calculated, regardless of its apparent precision or length, has any effect whatever on future values of a data set – never, never and never. Statistical trends, like the data of which they are created, are effects. They are not causes.

2. Models, not trends, can predict, project, or inform about possible futures, to some sort of accuracy. Models must include all of the causative agents involved which must be modeled correctly for relative effects. It takes a complete, correct and accurate model of a process to reliably predict real world outcomes of that process. Models can and should be tested by their abilities to correctly predict already known values within a data set of the process and then tested again against a real world future. Models also are not themselves causes.

3. Future values of a thing represented by a metric in data set output from a model are caused only by the underlying process being modeled–only the actual process itself is a causative agent and only the actual process determines future real world results.

PS: If you think that this was a silly exercise that didn’t need to be done, you haven’t read the comments section at my essay at Dot Earth. It never hurts to take a quick pass over the basics once in a while.

# # # # #

0 0 votes
Article Rating

Discover more from Watts Up With That?

Subscribe to get the latest posts sent to your email.

222 Comments
Inline Feedbacks
View all comments
John Whitman
October 18, 2013 3:26 pm

Kip Hansen on October 18, 2013 at 2:34 pm said,
Whitman: Do you have a link to “rgbatduke’s partial draft of his book project ‘Axioms’.”?

– – – – – – –
Kip Hansen,
I see rgbatduke has already responded to you with a link.
John

rgbatduke
October 18, 2013 3:30 pm

A better model for mean surface temperature anomaly could be constructed. It would include a process model for the 65 year dynamics observable here, and possibly the 21 year process as well. The 65 year process is key – it was the upswing of that dynamic which was mistaken for an anthropogenic effect in the latter decades of the 20th century, and it is that process which is currently driving temperatures back down again. All independently of egocentric humans who, like a flea on the elephant’s back, think they control the beast because he happened to turn when the flea thought “turn!”.
Dearest Bart,
We don’t always agree, but Damn! We agree on that. Indeed, as I’ve been posting elsewhere, HADCRUT4 can be modelled almost perfectly (well within the visible noise in the underlying process with a four parameter fit — a straight line with a roughly 0.2C sinusoid with period somewhere in the 55 to 70 year range (the data is too imprecise to do much better) Since HADCRUT4 is indeed an anomaly, the base of the linear part is irrelevant, so this is really a three parameter fit as one degree of freedom doesn’t count.
If one counts the degrees of freedom that are CLAIMED to be important in GCMs, there are at least ten — one can list things like TOA insolation, albedo, shortwave and longwave absorptivity, heat capacity of Earth and atmosphere, and that’s six parameters in a single slab model that averages over all rapid processes, in the simplest POSSIBLE model that actually illustrates a GHE. It is claimed that CO_2, that did not change in any relevant way until roughly 1950, is responsible for the sudden temperature upturn in the 1980-2000 era, when it could not have possibly been responsible for the almost identical upturn in the 1920-1940 era, when a single THREE parameter fit works across the entire interval with no change in functional form precisely where we have to expect one if CO_2 is indeed a proximate cause of global warming.
This requires a stunning coincidence between multiple nonlinear parameters in order to work numerically, and that coincidence is, in fact, rigorously statistically unlikely.
rgb

Editor
October 18, 2013 4:02 pm

rgbatduke: I suspect that you too are a Statistician or something close to it. You Statisticians do not own the world — despite your claim to God-like certainty or at least your probability of having God-like certainty of the probabilities. egads…you’d think I had been burning your sacred texts in public or wearing your Holy Statisticians Robes to a frat party.
Jesting aside, the problem here is the child-like simplicity of what I have to say. It is nothing more or less than actually stated. We do not speak the same language — you have your ingrained classical statistical definitions and seem to fail to grant anyone else a right to have a parallel correct understanding. You want to make this some great thesis on statistics and probability.
Have you done your homework, your part in trying to understand what I am saying? Did you read the Dot Earth essay, as suggested, and the comments there? Do you even realize that we are talking about different things? I simply state that trends are not themselves causative agents, they are effects.
You know perfectly well that drawing a line through three dots (numbers) on a piece of graph paper is a meaningless activity. If we know nothing but those three numbers thus plotted, they tell us nothing. Forget all the statiscianishness. Three points on a piece of graph paper, with or without a line, tell us nothing. The same three points, identified as button counts on three successive days, still tell us nothing more than that. No amount of statistics will make this as yet unknown system reveal any more secrets — certainly not the future.
Maybe you and Ted and Fred could get together and defend the sacred halls of StatsWorld by explaining the question I posed for Ted above http://wattsupwiththat.com/2013/10/17/the-button-collector-or-when-does-trend-predict-future-values/#comment-1452514. How can you use statistics to predict under conditions of total ignorance of system? I give you three numbers….make a scientifically defensible prediction based on them–you may make no assumptions.

Ted Carmichael
October 18, 2013 4:04 pm

Hi, Kip. Okay, I understand now. You think a three-day trend has absolutely no information. As you say, “Nothing else — nothing possible, nothing probable, nothing about tomorrow’s value at all.” Extrapolating from that, if – as you say – there is zero information in a three-day trend then, similarly, there must be zero information in a 30-day trend, or a 300-day trend.
Don’t you see that if your friend adds two buttons a day for, say, 342 days, then it’s a pretty good bet that he will add two more buttons on the 343rd day? It would have to be something quite exceptional, by the common-sense definition of the word, for this trend to end at exactly the point when you make a one-day prediction.
My brother has tried to argue something similar. He believes that all polls are useless; that it is impossible to survey 500 or 1000 people, and assume the results of that survey can give us any information about what the rest of the country thinks.
My brother is very smart, but he doesn’t understand statistics at all. Cheers.

Ted Carmichael
October 18, 2013 4:26 pm

One more quick note: you said, “I simply state that trends are not themselves causative agents.” Kip, thank you for clarifying this. I would like to assure you that no one believes a trend is a causal agent. Trends are correlated with causal agents, and therefore – even if we don’t know what the causal agents are – we can gain some information from a trend.
But I’m afraid “trends are causal agents” is the strawest of straw-man arguments. No one makes that mistake, even if (as with the dog-on-the-leash post) their wording is somewhat sloppy.

October 18, 2013 4:48 pm

@rgbatduke at 2:54 pm
I don’t know about the frog….but your story was enchanting.
The layers of logic were as rich as a pastry.
Especially that last: “All doors lead to certain death. Some perhaps sooner than others…”

Editor
October 18, 2013 5:00 pm

Ted — You must be a statistician. If you can still say this ” I would like to assure you that no one believes a trend is a causal agent. ” you have not done your homework. Read to comments at Dot Earth. And some here….You may not believe it….I certainly don’t. Since that is not an issue for you, and you agree with me 100% on that, why are you going on and on? My essay clearly states that the model is predictive. You somehow can’t get past the fact that the model VIEW you prefer is represented by a graph that may show some trend therefore the trend is …what? You are arguing about what I, as an internet programmer, would call a “view” — the database is back there, and I present various views of it with my web pages. My web pages are not the database, they are simply a single out of many views of it. It is not that there is zero data in a three day trend–there is zero data (above the three numbers) because none has been supplied.
(Remind me not to hire you as a fortune teller…..or to tell me whether to buy this gold mine stock based on three supplied ore sample numbers. )
Let me know when you are prepared to supply a prediction from three numbers alone — no system – no assumptions ………..and please, be prepared to defend it scientifically.

Editor
October 18, 2013 5:10 pm

Ted — Don’t worry, Briggs supplied me with a demonstration of the mean and probabilities of just a single number! You guys must have some wild parties!

John Andrews
October 18, 2013 10:05 pm

Tomorrow is going to be just like today, only a little different.

donald penman
October 18, 2013 10:43 pm

The rules that the button collector uses could change over time given future changes to his collection then even if we asked what the rules were going to be tomorrow he would not be able to tell us. In my opinion this is true of the model of how the Earths temperature changes over time and why we can’t know what the global temperature is going to be a hundred years in the future .

October 18, 2013 10:58 pm

Statisticians, including the usually amazing rgbatduke:
Not that you are wrong, but you are substituting belief for knowledge. I’m glad you believe in your statistician skills.
Yes! If you’ve analyzed every assumption and condition one can build a reasonably accurate trend.
Two buttons a day for an extended period does allow one to ‘reasonably confidently’ state one more day means two more buttons.
Only, reality is lurking with it’s own set of conditions. One possible condition is that two buttons has exhausted all button sources and instead one gets zero buttons added per day till new sources are located.
Yes, one can build companies that specialize in providing modeling services and you reap a financial benefit for being good at it. “…Not only do we build predictive models, we do so without any strong underlying assumptions concerning the nonlinear function that describes the joint probability distribution we are modelling. They are effectively almost purely empirical models, in other words, with only a few Bayesian assumptions that go into the structures…”
And every time the models are not up to snuff or even if something bothers you, they’re torn apart till you identify exactly what degraded the estimate.
The basic truth is; your confidence drives a ‘belief’ in your models. You’re d__n good, so your confidence is high. Does this mean you always hand your model report over to a customer without explanations, caveats, concerns? Of course not! Your team(s) analyze every run until you’re sure the bases are covered. No customer should ever get a model’s report until they understand what’s involved.
You are working out probabilities not absolutes. I’m glad you’re good and running a series of successful model runs; that doesn’t bring you close to omnipotent. Instead when you nail a forecast I’ll bet you’re jubilant, and you have every right to be. I’ll also bet you have a good idea when a model run needs work before handing off.
Sure, it’s a sucker bet to bet something will always fall when dropped, unless somehow something goes wrong with physics. That is not a trend, nor is it a forecast. The suckers who take the bet are deluded.
But, what if you’re dropping a down feather? A fluffy feather has enough surface area as compared to mass to make it vulnerable to other forces that are locally stronger, like wind or even one’s breath.
I’ll consider that bet now. Still willing to make it?
You don’t ‘win’ the bet by stating any model based on dense object trends; the bet is only won when the feather is ‘observed’ to fall down.
Sucker bets always pay off, reality bets are a risk. A risk that is mediated by high confidence model trends, but risks are never converted into confidence absolutes.
Gentlemen; you’re d__n good; both at your work and your comments here.
Kip:
I like your article. I do think you have a perspective that is different than some of the statisticians. I don’t think this makes you wrong, but be aware how perspectives affect our thoughts. The statisticians writing here are quibbling about some definitions. What’s missing is a lack of realization that when a statistician insists models can be accurate, they’re making that statement with full awareness of exactly what they consider accurate, within bounds unaffected by anomalous factors. They’re not magicians nor are they omnipotent; they’re well educated skilled humans.

karl
October 18, 2013 11:03 pm

Ted
You said:
“Extrapolating from that, if – as you say – there is zero information in a three-day trend then, similarly, there must be zero information in a 30-day trend, or a 300-day trend.”
And you are absolutely correct. A 30-day trend has absolutely zero predictive value for day 31!!!
Do we have to go back to flipping a (close to truly random) coin heads or tails to get the point that a trend is NOT predictive at all?
10, 100, 1000 heads in a row, does not change the fact that it is just as likely to be a tails as a heads, on the next flip, even though the trend screams heads.
I certainly hope that no one on this board would posit that the trend of 10000 heads in a row has even 1 infinitesimal bit of predictive information regarding the outcome the next flip.
If one argues that many heads in a row may indicate a trick coin, it may — but the law of large numbers supports 10,000 heads in a roll even for a random coin.
Either way — we would need knowledge of the fairness of the coin (underlying causal agent !!) to make the trend a useful predictor — it must be a trick coin for the trend to be predictive.

October 18, 2013 11:04 pm

“John Andrews says: October 18, 2013 at 10:05 pm
Tomorrow is going to be just like today, only a little different.”

Making fun of us huh? Good twist of humor of a forecast using a trend. 🙂

Janice Moore
October 18, 2013 11:50 pm

“… a pretty good bet… ” may turn out to be a winner, but the bottom line is
it is merely an educated
guess.
A sudden gust of wind can alter a trajectory.
An unexpected twist of the jet stream can alter local climate.
Trends are useful tools; they make poor masters.
We will be RULED by the junk trends of those worse-then-useless IPCC computer simulations if we lose the fight for truth about what drives climate (and what clearly does not).
Kip Hansen’s pointing out that we do not “know what a day may bring forth” is a POWERFUL argument against CAGW (regardless of the statistical nuances that he chose to overlook).
Let us not devour one another — we’re in this fight for truth to WIN.
At WUWT, we’re on the front lines of the battle.
“In essentials, unity.
In non-essentials, liberty.
In all things, charity.”
This motto will keep us strong.
While the other side shrieks at and backstabs each other — go for it Fantasy Science Guys! — we maintain a united front for the essentials and, thus,
TRUTH WILL WIN.

richardscourtney
October 19, 2013 3:04 am

Janice Moore:
Thankyou for the rallying cry to unity in your post at October 18, 2013 at 11:50 pm.
As you say, eventually,

TRUTH WILL WIN

and we seek ‘truth’ as a method to refute the false claims of alarmists.
It follows from this that we need to proclaim truth and to refute falsehood.
It is a falsehood that “trends cannot and do not predict the future”.
The truth is that
A trend can and does predict the future so long as the trend continues into the future. But trends change with time and, therefore, trends are imperfect predictors of the future.
And that truth is everybody’s common experience. Before people knew why day followed night they assumed it would because it always had: and that trend was a reasonable predictor that dawn would break tomorrow.
Proclaiming falsehoods which are not in agreement with everybody’s experience does not provide credibility when we proclaim truth.
Richard

mitigatedsceptic
October 19, 2013 3:11 am

Unfortunately, the alarmists do not see ‘truth’ in quite the way ‘realists’ do. Many ‘Realists’ believe, and some even claim to know, that ‘out there’ there exists an objective world – the ‘true’ world that is available for observation by seekers after truth (normal scientists). The contrary view is that our view of the truth is socially constructed and that science is just another vehicle for imposing one world view on others – this is rather quaintly called ‘post-normal science’. Heavily disguised as truth seeking, this is no more or less than a fraud contrived to achieve some political aim – such as the realisation of Agenda 21 – if that is strange to you, just Google it up for a big surprise. That is only one example of the host of ’causes’ riding the post-normal bandwagon. Indeed anyone who thinks that the Hand of Destiny is on his/her shoulder is a welcomed passenger. Even Paul Nurse, President of the Royal Society, contributes to this simple pragmatic view in the name of drumming up cash for research such as his microbiological research centre being established in the heart of London and incalculable cost to taxpayers (i.e. everyone).
I have suggested elsehwere that it is not reason that drives the emergence of ‘institutions’ from conversations but the comforts of togetherness that arise from participating on language games (yes old LW again), that these institutions colonise and consume human lifetime is the cause of their own survival and this goes for all belief systems however zany.
All this is quite ‘natural’ and simply a feature of humanity – but it really is perverse and if we think that ‘institutions’ serve human interest, we probably have fairies at the bottom of the garden too.

mitigatedsceptic
October 19, 2013 4:04 am

Before the moderator suggests it, I would like to withdraw the reference to Paul Nurse and offer the following amended version…
Unfortunately, the alarmists do not see ‘truth’ in quite the way ‘realists’ do. Many ‘Realists’ believe, and some even claim to know, that ‘out there’ there exists an objective world – the ‘true’ world that is available for observation by seekers after truth (normal scientists). The contrary view is that our view of the truth is socially constructed and that science is just another vehicle for imposing one world view on others – this is rather quaintly called ‘post-normal science’. Heavily disguised as truth seeking, this is no more or less than a fraud contrived to achieve some political aim – such as the realisation of Agenda 21 – if that is strange to you, just Google it up for a big surprise. That is only one example of the host of ’causes’ riding the post-normal bandwagon. Indeed anyone who thinks that the Hand of Destiny is on his/her shoulder is a welcomed passenger.
I have suggested elsehwere that it is not reason that drives the emergence of ‘institutions’ from conversations but the comforts of togetherness that arise from participating on language games (yes old LW again), that these institutions colonise and consume human lifetime is the cause of their own survival and this goes for all belief systems however zany.
All this is quite ‘natural’ and simply a feature of humanity – but it really is perverse and if we think that ‘institutions’ serve human interest, we probably have fairies at the bottom of the garden too.

acementhead
October 19, 2013 4:31 am

Jan Smit says: October 18, 2013 at 4:21 am


Point taken, sorry for being so sloppy! However, it was only meant as joke about how statistics can be used to assert any nonsense (hence the winking smiley). But I consider myself suitably chastised…

Oops, sorry I guess I overdid it. Must confess that I failed to see the smiley. I claim, in mitigation, very late hour and refusal to wear glasses even though eyes almost time expired.

Here’s a good link to support acementhead’s correction of my spurious nonsense:
http://www.prb.org/Publications/Articles/2002/HowManyPeopleHaveEverLivedonEarth.aspx
Do you agree though, acementhead, that statistics can and are used to support all manner of claptrap?

Yes I most certainly do agree with you on that.
Cheers

H.R.
October 19, 2013 5:13 am

Quite some time ago here on WUWT, regular commenter Anna V pointed out that “You can never cross the same river twice.” The earth, the solar system, the galaxy, the universe are on unique paths that started somewhere and will end somewhere else. So I really appreciated this comment;
John Andrews says:
October 18, 2013 at 10:05 pm
Tomorrow is going to be just like today, only a little different.
I would add that every once in a while, tomorrow turns out to be very different as, for example, that day where the earth was struck hard enough to blast the material for our moon out into earth orbit. Though I probably won’t be around to see it, one day the sun is not going to rise. That’s a good long term prediction. Short term I’m betting on old sol popping up tomorrow.

As Richard points out just above (richardscourtney says: October 19, 2013 at 3:04 am), the predictive value of a trend is only good for as long as the trend lasts.

Editor
October 19, 2013 6:52 am

A comment aimed at acementhead and Jan SmitThank you for being an example of how it is possible to work out misunderstandings and smooth over little offenses cause by “the heat of the moment” in a public forum. If only all of us would work so hard to work through and resolve these types of differences. Well done.

Editor
October 19, 2013 7:22 am

Reply to ATheoK (October 18, 2013 at 10:58 pm):
Thank you for your well-nuanced, well-mannered and thoughtful contribution. Of course, what you say is right. Of course, what our resident Statisticians say is right within their own frame of understanding and given their own very well-trained adherence to their specialized definitions. Unfortunately (or fortunately–opinions vary) we do not all speak that language nor do we all have the training to think in those specialized ways—we need to understand things in the context of our lives and experience.
It is seems obvious to me that when our Statisticians say “a trend” they actually refer to (what I call) the underlying model that produced the trend and conflate the two into a single entity – thus speak of the trend having predictive capabilities.
While, as a budding blog author (for years I have simply commented on topics that interest me) who has just recently stepped out onto the public stage, I have gained some experience with the comments section in which abject name-calling and intellectual bullying are part and parcel of the course. Plenty of that at Revkin’s Dot Earth. I was caught by surprise though being mobbed by a tag-team of (mostly) polite professional academic statisticians, including, arguably, one of the finest minds in the world. It has been a learning experience.
Thank you again.

Editor
October 19, 2013 7:51 am

Reply to karl (October 18, 2013 at 11:03 pm) :
Thank you for your supportive comment.
I think the points that I would take away from your discussion of truly random systems are these:
1. In order to deal with any trend on a time series, one absolutely must (with apologies to rgbatduke) know something of the system being represented—the system that produced—that is ‘caused’—the metric considered in the time series. Doesn’t matter if it is buttons-in-jar, GMST, blood serum levels of some biological indicator, or sunspots. Before using the apparent trend to make a prediction, once must know at least something about the system—the physical process itself—what we are talking about here!
If, as in your coin toss, the process is entirely perfectly random, then we must use a model of a random process to make our prediction. With a model for a two-sided fair coin, the prediction must be (real world here please!) 50% probability of heads and 50% probability of tails—which in a practical world sense, is no useful prediction at all.
2. Most of the physical processes that we encounter in our day-to-day and scientific lives are not perfectly random. Thus, the things we wish to be able to predict need a different model. Even processes that seem simple can require quite complex models to produce practically useful results. We must not incorrectly assume randomity. Examining past data (trends, spreads, anomalies, etc.) helps us to determine the rules (causative agents) in the process under consideration and enables us to produce a functional model that is adequately predictive.
3. In almost all of the examples offered of the use of trends successfully predicting, the “trend” offered is what I have tried to clearly delineate as the model—even if just an assumed, automatic mental model—of the underlying process.
Thanks for your contribution here.

Reply to  Kip Hansen
October 19, 2013 12:50 pm

For those who do not trust the use of statistics in data analysis, I suggest you read http://books.google.com/books?id=8C7pXhnqje4C&printsec=frontcover&source=gbs_ViewAPI#v=onepage&q&f=false. This is based on a paper I presented at an international corrosion symposium way back in 1971. It was published as a chapter in a handbook and has been revised and published twice since then. I think it is still valid. If you don’t understand how to use statistical techniques, you should talk to a statistician and at least learn from them. The probability that you will make mistakes in your research will be greater when you don’t use statistical techniques correctly. Much of the the IPPC research is a good example of missuse. Ask most any statistician.

Matt Skaggs
October 19, 2013 7:56 am

Kip Hansen wrote:
“Reply to Richard:
We’ll just have to see what others think about your view.”
Yeesh, this has to one of the most depressing and pointless threads on WUWT in a long time. Thanks (again) to Richard Courtney and Dr. Brown for injecting some sense. Hansen’s bilious argument can be demolished with the simplest of concepts: if you are attempting to predict the future, it is wise to do the best you can with what you have. If you have empirical information, that is a good place to start!

Theo Goodwin
October 19, 2013 9:01 am

Steve Obeda says:
October 17, 2013 at 7:06 pm
You overlook your own tacit knowledge. If you rigorously developed and maintained metadata on your “forecasts” then that metadata would emerge. It is still not prediction in the scientific sense. But there is information in your metadata that serves as practical reasons for your forecast.

Theo Goodwin
October 19, 2013 9:11 am

Jquip says:
October 17, 2013 at 8:38 pm
“Nor is there any necessity, or even sanity, in attempting at self-fulfilling logical positivist approaches to causality. Newton went commando with Hypothesis non Fingo. Kepler was a pure data fiddler. So was Galileo. Point of fact, any postulated cause has less expectation for correctness, in general, than a long chain of validated predictions arising from a non-causal model. And that’s putting aside any expected issues that arise from the field of physics and it’s distinct reliance on impossible objects taken to every infinite asymptote they can find.”
I would be happy to debate you on this matter if you would make substantial claims. Your claims here amount to nothing more than trolling. Are you really willing to claim that Kepler’s work and Newton’s work cannot serve as exemplars of scientific method?
Also, on the matter of prediction, you speak with the vulgar. You write:
“Point of fact, any postulated cause has less expectation for correctness, in general, than a long chain of validated predictions arising from a non-causal model.”
One cannot assume, as you do, that the term ‘prediction’ has a clear and defensible meaning in your sentence. Your job is to explicate the meaning of ‘prediction’. In other words, whether “predictions arising from a non-causal model” actually qualify as scientific predictions is the issue being debated. The loose way that you state matters requires that you explicate “non-causal model.”