Guest essay by Eric Worrall
Why pre-judging the value of potential stepping stones to a solution, stifles scientific innovation.
Kenneth Stanley, one of the world’s top artificial intelligence researchers, has produced a fascinating presentation, on why focusing on an objective can sabotage innovation.
Consider how you would teach a robot to solve a maze. The obvious thing to do, is to teach the robot to do its best to reach the end of the maze – to reward movements which bring the robot closer to the end of the maze, and punish movements which cause the robot to move away from its objective.
However, researchers quickly discovered that if you give these logical seeming instructions to a robot, something unfortunate happens.
The obsession with finding a solution, and on focusing resources on actions which seem to bring the robot closer to the solution, blinds the robot to the possible paths which actually lead to a solution.
How do you solve this dilemma? Ken’s solution to preventing this kind of hangup is incredibly simple – the way almost every ingenious breakthrough is simple. Instead of punishing the robot if it makes a move which takes it away from its goal, you reward the robot for innovating – for always doing something new. The only caveat is, the robot should try to avoid repeating itself.
Focussing only on trying something new avoids prejudging potential stepping stones to the solution. Until you know what the solution is, there is no reliable way of knowing how valuable a potential stepping stone might be. The only way to solve the maze in my example, is to take a path which leads away from the solution – a path which an entity obsessed with reaching the solution would find very difficult to take.
Or to put it another way, if 3 decades of attempts to build a model which works, has failed to deliver results, maybe its time to stop repeatedly butting your head into the same wall.
This approach to problem solving, focusing on novelty, does not prevent you from monitoring the process, to see whether a solution has been found – solving the problem is the ultimate goal. All novelty search means is that you keep your opinions to yourself – you let the robot do its thing, regardless of how silly any individual move might seem at the time, until it finds the solution on its own terms.
The following is Ken’s presentation “Novelty Search and the Myth of the Objective”. Ken’s presentation covers a lot more than my quick summary – including valuable insights into why scientific advance has stalled in a number of disciplines.
Update – Ken Stanley has published a book, available for pre-order on Amazon, which discusses his ideas in more depth.
Click here to hear an interview with Ken, about his new book.
Insanity: doing the same thing over and over again and expecting different results. – Albert Einstein
I think it is core part of the scientific method. Not the 95% approved scientific method, mind you
Errrr, Francisco – is that not – actually – 97% approved?
Might that sort your micro-problem?
Auto.
It is not uncommon to come across this quote from Einstein.
I wonder how many people here comprehend what it takes to make a scientific breakthrough, to intellectually reach the end of the road and then to formally account for territory previously out of reach to the scores of academic/scientist that could not advance any further. It is the strenuous and most complex endeavor I am aware of. It involves maniacal revisitation and intensive reanalysis of the same set of concerns over very, very extensive periods of time. I would argue that 99% of academics/scientists fail 100% of the time while the remaining 1% make progress 1% of the time.
I’d say that is probably about right.
There is absolutely no reward in science like holding on to your position until unambiguous data permits you to gloatingly look at your friends and colleagues, drop a manuscript on their desks and say, “I told you so.” This is true particularly in field sciences, where a pattern to one person is just noise in the data to another.
Well i have a counter example, the quest for novelty propels the Academy(and politcs) to make even crazy theories and get out of strings of Scientific Method.
The raise of “social” sciences and crazy theories are testimony of that.
The quest for unbridled novelty could also be a testament of unbridled narcissism.
An excellent point. Not all things “novel” are good. But there are tests for merit, and I think the real issue is tossing aside merit for the sole property of novelty.
You miss the point. Unanticipated solutions can only by found by emphasizing novelty, by definition. There are lots of wrong novel solutions, but an infinite number of repeats of the same wrong solutions possible. It’s only the few right novel solutions which are the payoffs.
I agree Alex, and it’s all dressed up as altruism.
Good point, and even the tests for merit are relaxed along the way as seen in IPCC models and prediction accountability.
So in the case of the maze above, just what happened that the ball interpreted as punishment or reward, that led to the solution ??
It was programmed not to repeat known errors, so it didn’t get “stuck”.
well in the first video, the ball never did try making a right turn instead of a U-turn.
Rewarding novelty may work with a the shown puzzle where the number of options is small but suppose instead it was a puzzle with say, 50 binary decision points. Rewarding novelty is in effect biasing toward a brute force search. With large search spaces, it is no more the answer either.
This.
Actually, DAV, rewarding novelty, where there is a large number of possibilities bias towards no solution in our lifetime, if you will. the solution might be as simple as the maze – 4 right turns and 3 lefts, but would taking 4 right turns in a row be “novel” or would it be repetitive? Would the robot choose to follow that first right turn with a second, or would it, out of novelty triy to turn left? then, confronted with no forward movement, would it have turned left a second time since that would be novel at that point and proceeded back out of the maze? Rewarding novelty might very well never solve the problem.
Ken covers this issue in his presentation. If you are attempting a truly deep search, you have to give up on achieving a pre-determined objective, and just keep an eye out for something “interesting”. Being open to any “interesting” discovery, rather than ignoring outcomes which don’t fit your pre-determined criteria, helps tilt the odds back in your favour.
Ken’s point is that for most real world searches, there is no set of criteria you can apply to determine the “best” decision at any particular point in the search, because you don’t have the information you need to make that determination – you don’t yet know how to reach the destination.
Ken doesn’t claim his technique is a cure all – there are some classes of problems where other algorithms work better. For example, if you wanted to solve the travelling salesman problem (shortest path linking a set of cities), you would apply an evolutionary algorithm. But in a real sense the travelling salesman problem has already been “solved” – you might need a new solution to a particular instance of the problem, but solving any instance of the problem is just a matter of applying a well known algorithm. The classes of problem Ken is talking about are problems where the solution is totally unknown, where you are taking a step out into the unknown, to try to find a totally new solution.
Well in patent law, novelty, while necessary, is not sufficient. It must not be obvious to a person of ORDINARY skill in the art.
In other words, you don’t get a patent for doing nothing more than applying the basic ordinary skills of your art, even if you never performed that exact task before. (nor if nobody else did it before).
On the other hand, in copyright law, you can repeat the same “one from column A, one from column B, …. ” process over and over again, and ALL of your resulting garbage, is copyrighted.
Joseph Haydn wrote 104 symphonies (at least), and most of them were junk when he wrote them, and still are. Likewise, Mozart wrote 41 (at least) and at least three of them are noteworthy.
But none of those ancients turned out garbage at the rate that modern “rock” bands do. And yes, the moderns do occasionally do something good.
I have never used the same design for something, ever again; always a new design; but hardly a patentable design.
So novelty is of limited value.
Real insight maybe.
Of course the solution to a maze is to follow the same wall, and if you find yourself back at a point you have previously visited, switch to the opposite wall and follow that. Note that the simple maze in the diagram can be solved by following the left wall, producing the illustrated solution – or the right one, which produces a shorter solution, avoiding the blind alleys. Now the correct algorithm is a real advance, and is rather more sophisticated than some instruction not to repeat mistakes. However, there is no way of knowing whether it will be quicker to start following the left or the right wall. Finding the optimal solution entails exploring both alternatives. That of itself is a paradigm that climate science would do well to adopt.
Until the maze has more than “one central island” … And if those multiple center islands interlock with one another (imagine several “C” shapes keyed into each other) so if you “switch walls” within a key or keyhole, you find yourself still “on” the same central island but on the mating wall of the other central island keyhole.
You’d need a spraypaint can to mark each wall as you “leave” a wall.
Could the quest for unbridled novelty also be a luxury? I’m busy viewing the video and obviously, dohh, by definition, novelty is the most important factor in groundbreaking research. (Slaps himself upside the head a few times.)
It can’t hurt to try something new and see what your morning coffee tastes like with just a drop of jalapeno oil. I don’t think everyone has the time and patience to try a new way every time you brush your teeth. But a good lover will combine experience of what works with a constant curiosity to know what else works too.
Once we find a solution that seems to be the best, like the optimum route to work and back every day, the search for novelty could become a luxury to keep us from getting bored.
That is not a “counter example.” It simply is the way things really work. The robot in the original article has to explore the “solution space” available to find an actual solution. Some of those are “crazy,” some are not. From above we can tell that. from the robot’s eye view, the information does not exist until the space has been explored. Once that space has been mapped, if the robot has adequate storage, it will not explore that space again.
The problem with the social sciences is that, like climate, ecology and economics (which is in truth a mere subset of ecology), the social sciences deal with extraordinarily complex phenomena and worse, most of those phenomena are “robots” composed of collections of individuals exploring the available “space” for solutions to problems. The solutions can be either evolutionary or adaptive and they may be optimal, barely work, or lead the social entity involved down the path to extinction. The problems may be real, perceived, or so absurd that almost anyone outside the community can see that the problem is a delusion.
An excellent example of a delusional solution could be the Shaker communities of the 19th century. Shakers did not reproduce. They relied on recruitment instead with what some would regard as predictable results. Whatever the “problem” was that the Shakers were attempting to solve, their solution was a non-violable one that lead to community extinction. Some Shakers remained in the community the community to death, while others left community to lead other kinds of lives. All we have left now of Shaker society are some fine furniture forms.
Well even today, we have communities that do not reproduce, but do recruit. And they are lauded as examples to the rest of us to revere.
go figure.
commieBob April 2, 2015 at 9:05 am
Robert, I have 36 years in the oil and gas industry. I’ll tell you that the principle of doing the same thing over and over again is deeply embedded in industry “leaders”. There is a social disincentive for trying something new. If you fail at what worked before or what the others are doing, you get sympathy: obviously there were other factors beyond your control that did you in, so here, have some more money. If you fail three times doing the same thing, well, you’re an idiot for not making a reasonable thing work. If you try something new and it fails, you were an idiot for trying. If it works, well, you clearly identified something new in the environment that others hadn’t picked up on yet – your approach isn’t new, the environment is. You are not smart, just luckily observant.
I recall struggling with new projects and budgets during the late 80’s, during one of the downturns in the industry, in which we had to bring forth ways to succeed in the “new” situation. All our new ideas were shot down. I concluded that the real directive was “to do exactly what we have always done, but faster, cheaper and more profitably”.
The try-try-again-without-change appears to be our modus operadi as a species. Speculative action is viewed highly suspiciously. Which is why skeptics are demonized. We all “know” what should be done, so those who think or act otherwise are bringing us to disaster.
Doug,
I think a lot of what you write about is to do with the types of personalities that percolate to the top of organisations and how they tend to act to not let a new generation show them up as being the ‘old guard’. A lot of the innovators just leave.
Just talking with one of my sons this evening he is in that boat with bosses who employed him expressly to develop a new portion of their business but then will not resource him to do so rather spending development money on the parts that they established before he came along. Go figure.
Not so much the novelty driver as the ‘my’ novelty, ‘I invented it’, ‘its my baby’, stay away from my funding mentality.
When I worked for other people, in the olden days, we ( the people who made the stuff actually happen) would shake our heads and have group hugs at the dopey management calls, the doing completely dud deals with their buddies etc. We called them ‘sparkling executive decisions’. The guy who invented that term was an Ulsterman with that wonderful accent and it drips with the sarcasm that only an Irishman can properly articulate.
Since you’ve been in the O&G business that long, I’m sure you’re well aware of how the entire industry treated George Mitchell as though he was completely insane for his belief that the oil shales could be incredibly productive, if he could just figure out how to frac them right.
Yep, that idea was just bug-eyed crazy, it was. Nobody wanted anything to do with it for 20 years while Mitchell worked on it, alone.
Doug, very interesting what you said. I was talking to a girl yesterday who has good job, but has been head-hunted by another organisation, who interviewed her, she doesn’t know whether to take the job (if it is offered to her), or not. When I asked her why rock the boat, if she is happy where she is, she told me, quite logically, in my view, that she can take risks now because she doesn’t have children or a mortgage and is still living with her parents who would support her if it didn’t work out. This sounds fairly obvious, but it wasn’t to a 59 year old in the tail end of a career, getting to my age and “rocking the boat” is not necessarily a good idea,
From your spelling Oi would guess you are American, in the UK we have endless rules and regulations which stifle initiative, which can lead to the end of a career. The robot analogy is a good one because the young (who have relatively,nothing to lose) will by nature, take more risks than I would, because I have a lot to lose if it goes belly up. This is why young people are more employable than the middle-aged, because we behave like frustrated, penalised, robots!
I think you are interjecting risk/reward into the discussion, as it needs to be. If the owner of the robot loses nothing each time it makes a wrong turn, then there is no problem letting it try until it succeeds. However, if there was a hefty price to pay for any wrong decision would the owner of the robot stand by and watch continued failure?
Tom, did you miss the bit where punishing failure failed?
re: Brian H, April 2, 2015 at 9:13 pm
Punishment of failure fails if the punishment means nothing to the entity being punished. That would include the entity being too stupid to understand how the punishment will effect it.
“The following is Ken’s presentation “Novelty Search and the Myth of the Objective”. Ken’s presentation covers a lot more than my quick summary – including valuable insights into why scientific advance has stalled in a number of disciplines……”
Such as in the area of nuclear power and nuclear technology thanks to all the anti-nuke propaganda being spewed out on a regular basis.
In routing Integrated Circuits and Printed Circuit boards design tools have struggled with this for a long time, they use a procedure called simulated annealing to find a better overall route by allowing a less than ideal path to be selected.
http://en.wikipedia.org/wiki/Simulated_annealing
Serendipitously, it turns out that butting up against that wall pays really well.
Pay wall?
In some fields of science, assumptions and subequent inferences are taught as facts, and anyone who points this out is treated with suspicion or contempt. Promising areas of research are forbidden, not because they mightn’t have value but because the facts they could reveal are not acceptable to the people who control the funding. They just don’t advance certain political agendas and so are forbidden.
Remember Eisenhower’s warning about the military-industrial complex? I wager you don’t know the second half of the warning in which he explains that the marriage of government and academia was what was largly responsible for the dangerous situation.
These people are obsessed with money because it’s power, so they’ve endeavored to control all of it, and the little bit of private money they cannot control galls them. Like Mr Potter in the movie, “It’s a Wonderful Life,” there are still a few people not under his control, and it galled him no end.
Well, Third Stooge, most people might not be familiar with Eisenhower’s second admonition concerning the marriage of science and government – but I’m sure most of the regulars here are.
Tomb,
You insulted me and then spoke on behalf of everyone else, while ignoring my premise.
I don’t know what better describes our current politics and governance. And here is the second warning which has also come to be.
It is interesting Larry, that while Eisenhower’s 1st warning is part of the cultural lexicon, his second warning is not and difficult to find. I only found it by going directly to the speech. Thanks for pointing it out!
Eisenhower’s warnings are more accurate than all of the models combined. That we have ignored his warnings is much more problematic than any various changes to the climate since one is an internal issue corrosive to the nations core and the other an external factor which humanity has been amazingly adept at addressing since it’s inception.
Reasonable clear minds believe in the scientific method only because it is proven to be the best method for discovering the truth or solving a problem. Better to only believe in something when there is evidence and strong argument. Today money and power have conflated the entity with the principle. Ideologues, self-righteous, elites, may proclaim their faith in “science” as and when it suits their ideological or financial needs, however religion is the only place for faith.
Larry, the point Eisenhower makes is not simply that a marriage of government and academia will be responsible for the “dangerous situation.” What he says is much more complex and nuanced:
The prospect of domination of the nation’s scholars by Federal employment, project allocations, and the power of money is ever present and is gravely to be regarded.
Yet, in holding scientific research and discovery in respect, as we should, we must also be alert to the equal and opposite danger that public policy could itself become the captive of a scientific-technological elite.
That is, he fears a positive feed back between government funding and academia. Where funding federal money dominates academia and in turn academia comes to influence policy far beyond what is wise. He also draws the conclusion that the hazard is a direct result of the establishment of the post-Korea military-industry-research cycle, caused by the growing Cold War.
“Or to put it another way, if 3 decades of attempts to build a model which works, has failed to deliver results, maybe its time to stop repeatedly butting your head into the same wall.”
Heck yes! Perhaps it is time to question the very basis of the current paradigm that James Hansen came up with involving CO2. Just maybe CO2 don’t warm the planet after all.
I invent for a living and I can tell you that Ken’s approach is sound principle. All of my break troughs have been mistakes or throwing out assumptions and repeating previous steps that did not work with the assumptions in place. It took me several years to change to a format similar to Ken’s example and quit making assumptions based on the objective.
His presentation points to the fact that I am likely to be retired in the near future. The current window is likely to be small to finish my current projects. I suspect we humans will obsolete ourselves possibly within my lifetime.
Or to put it another way, if 3 decades of attempts to build a model which works, has failed to deliver results, maybe its time to stop repeatedly butting your head into the same wall.
The trouble is they’re not looking for a model that ‘works’, they’re looking for one that proves their theory correct.
In a NUTSHELL
Well said!
Or, of course -whisper it – they seek a model that covers salary and, perhaps bonuses, and pension contributions, until retirement.
I do wonder (a little) at how little emphasis is based on paper authors’ proximity to retirement – versus the likely timeline for fairly assessing their magnum opus . . . . . .
Should I go back to watching the daily rise and fall of the price of marine plywood?
Auto
May I suggest an edit?
“The trouble is they’re not looking for a model that ‘works’, they’re looking for one that appears to prove their theory correct or is at least good enough for a headline.
😎
Nah, just one that keeps the alarmist position in place and the money rolling in. It’s tough having to move on to another field when it’s about all they have ever known. Big pay cuts are no fun.
I currently do not have a conclusion on what is likely driving our climates from a natural or an anthropogenic perspective. I do see many possible paths but nothing concrete.
Many here have made their conclusions and often interject them in the comment section of WUWT threads to remind us of their views and that we are wrong to not consider their conclusions as the basis for skepticism.
The weight of the evidence suggests it is not CO2-based. There are periods such as the Mediaeval, Roman and Minoan Warm Periods that had far less CO2 and were warmer than today. The Ordovician period had 11 times the CO2 and an ice age. It hasn’t warmed for 17 years despite rising CO2 and there is no tropospheric hotspot basically disproving the central positive feedback tenet of their hypothesis.
CO2 rise trails temperature by 800 years.
These basic points do not deny the GHG nature of CO2 and we do not even almost fully understand climate but CO2 is not a major climate driver, if climate sensitivity was high the feedback amplification of CO2 would create a runaway Greenhouse.
It never has.
Water is obviously the key. The radiative and convective actions of water in our atmosphere, in particular the cumulonimbic radiative subsidence at the emission level are significant counterbalancing negative feedbacks. The warmer it gets the more the atmospheric layers expand and expose heat to the emission level and radiate more heat away (one mechanism). Low level clouds also provide a cooling albedo effect.
Venus didn’t have this water mechanism as it lacks the liquid iron dynamo effect at it’s core like the Earth to give it a protective electromagnetic field, thus the hydrogen and oxygen was swept away or photo-disassociated by the solar wind while the heavier CO2 remained in the atmosphere leading to a runaway Greenhouse.
When made to think about it, Atmospheric Physicists tend to say that all gases in an atmosphere could be termed greenhouse gases. But the effect is caused by Mass, not internal radiation,
This looks like a repackaging of the classic “exploration” vs “exploitation” balance issue in machine learning. Just exploration reduces to random search, a futile effort in a big enough search space. Just exploitation gets stuck in local optima. The right balance is problem dependent and algorithm dependent.
Here is one novelty, intensely disliked and opposed by some who ‘presume to know’ all unknowns.
http://www.vukcevic.talktalk.net/Back1.gif
It was devised when only 3 previous ( 2 + and one – ) peaks where known. Since then one negative peak hit the target, soon we will know if the positive one is going to do it too. If it does it will be even more fervently opposed by the ‘settled’ science, despite it being based on precise and well defined orbital numbers.
I could speculate how it works, so could many readers of this blog how it could or it couldn’t, but to be truthful I don’t think anyone knows either way.
All of nature follows various rules one of which is the wave rule. When you make a wave, each wave is weaker than the previous one unless more energy is fed into the system.
The sun follows this very much. As it moves about the Milky Way galactic system, it changes its energy levels based on gravity situations relative to other star groups within the spiral arm (Perseus Arm in the present case).
Our local star is somewhat unstable.
The same kind of limited sample problem exists with cycles and peaks in the AMO. It could be modeled much like the solar case. The larger problem is side stepping the limitations of the AMO cycle count and variance of amplitude to diminish it in models and predictions. The cycle influence does not go away it is just over-generalized as an average cycle which it is not. The science and policy worlds lose in that case.
I wonder what it would look like in this cycle problem——-
http://www.climate4you.com/images/AMO%20GlobalAnnualIndexSince1856%20With11yearRunningAverage.gif
North Atlantic has two more cyclical events but running on a different time scale
http://www.vukcevic.talktalk.net/EAS.gif
Resourceguy
I just realise there is one more oscillation in the north Atlantic, beside the two shown in the graph above, and that is the geomagnetic Y (east) component, again running on yet another time scale.
See my post further below at April 2, 2015 at 12:09 pm.
North Atlantic is throbbing with all kind of oscillations.
You devised this formula in 2003? No tweaks? Honest?
Mr. Westhaver
Tweaks are: 152 required to set amplitude, year 1943.5 is required to set time reference, cos2pi/3 to set initial phase, numbers are from astronomy ( largest planet’s orbit 11.862 and synodic period 19.859, the names get me into trouble).
I hardly knew anything about sunspots. In 2003, my daughter (now Oxford postgraduate, working for well known and by far the largest world company in its field) was showing me her geography homework. I noticed sunspot cycles graph, my electronics background told me: that is a rectified amplitude modulated signal. I ‘de-rectified’ it and did simple assessment of cosA + cosB, and left it. Few days later I realised that approximate numbers 23.5 and 20 are actually 2 x 11.862 and 19.859.
See also my comment further down at April 2, 2015 at 12:09 pm.
Well I like it, but…. it may be right for the wrong reasons 🙂
I would like it even more than you do, if I can convincingly show why it happens to be so.
It is good to start with being right, the reasons can be disentangle at some future time
Any advance is strangled at birth by starting with being wrong
The graph you show is not based on the link you give to the source. Honesty has to be earned and you get a fail here.
I used to work for Analog Precision on computer board prototypes way back in the very early 1970’s.
Designing a good ‘electronic information path’ was a highly ARTISTIC skill requiring a sense of ‘this is a good thing’ based on knowledge of how Nature evolves various solutions to say, moving water through a root system.
These computer components evolved just like Nature but in reverse: from ‘higher level plants’ to ‘bacteria types’ to ‘viruses’. Smaller and more independently enclosed.
Most computer creators back then were all ‘antisocial/crazy/outside the box/hippie type/pot smokers’. Seriously. The old guard hated us.
The second example is the obvious one to me, but then I’m left-handed.
While I applaud the presentation, I just don’t find anything new in it.
Bloke down the pub April 2, 2015 at 9:58 am
The trouble is they’re not looking for a model that ‘works’, they’re looking for one that proves their theory correct.”
Exactly correct. And obvious.
Eric Worrall,
I enjoy your contributions to WUWT. I often read them. You use the term Novelty. I have also proposed a similar notion but I refer to it as “Wonder”
The Scientific Method is a great means to grind to the details. It is useless to the inspiration necessary to have the initial spark. You say novelty, I say wonder.
In any event I agree that novelty is essential. A foolish consistency is the hobgoblin of small minds.
Great article. Keep writing.
Thanks Paul 🙂
Here are two of my favorite science quotes:
“The essential point in science is not a complicated mathematical formalism or a ritualized experimentation. Rather the heart of science is a kind of shrewd honesty that springs from really wanting to know what the hell is going on!” Saul-Paul Sirag
“The world is full of magical things patiently waiting for our wits to grow sharper.” – Eden Phillpotts
Love it!
Encourage everyone to watch the video… can open your eyes to how it could be extrapolated to every scientific field.
Maybe someone(s) here can educate me –
How does one punish or reward a robot?
algorithmically
As Richard Russo wrote – the real competition in an English department
is for the role of “straight man”. Good one.
You punish or reward the owner of the robot.
For robots both the punishments and rewards are identical, load in more advanced program.
I confess to ignorance about mazes. But if I had to write a program to exit a maze my first approximation would be “choose right or left, then follow that wall to the exit.” For those who may have studied that problem, are there algorithms that in fact are more reliable and/or faster? In particular is there an algorithm using innovative (random?) trials that works better?
Yes, that would work. But the idea is to test AI – to test the ability of the robot to discover its own solution.
You need a small modification: if you find yourself retracing the same wall, switch to the other one. Obvious if you consider a central island in the maze, which you might follow round and round if you didn’t switch.
Eric,
I have several patents that were produced by assuming novelty inherent in system analysis. As an exercise I would often abandon conventions and take a reverse approaches or perpendicular postures to the problem at hand. I can’t tell you the number off times some old fart told me something to the effect that “the science is settled”. Some people are so limited in their mental wiring that their imaginations fail to consider a different meal time. Novelty causes a great deal of work and a huge number of failures but every so often a NOVEL solution is availed… and I make money! 🙂
Novelty precedes science. Science is the reliable and necessary boring part that explains everything in a cogent manner.
The benefits of novelty seems to cross into every field. The crucial key to novelty search seems to be to switch off the internal censor, to let ideas percolate without shooting them down before they have time to develop. In business it is call brainstorming, in writing and art it is called creativity, in science it is called innovation.
Well summarized. I love your stuff. Keep writing.
The best way to search for novelty in science through theoretical thought is to understand the nature of the critical assumptions made in any field of interest as well as understanding how key parameters interact to create outcomes.
Then you have to be prepared to question whether any of the assumptions may be invalid in certain situations and, if so, what the consequences of that would be.
However, in many cases, the drive for innovation is a totally unexpected experimental observation which begs an obvious question: why?
I was supervising an MD student years ago and we were trying to create novel transgenic mice targeting expressing of a particular protein to specific tissues. During the breeding, we started to notice that we simply weren’t generating male homozygotes, which of course is a prerequisite to create an effective breeding colony.
I wasn’t still around to be involved in working out what was going on, but we certainly weren’t starting out from the point of asking ‘what does this protein do in that tissue?’, since the relevant tissue was far away from where we were actually trying to target expression to.
Noticing unexpected things is a very useful skill to have if you want to innovate…….
Dr. Stanley’s description of the inadvertent collaborations on the picbreeder site reminds me of Matt Ridley’s talk “When Ideas Have Sex”:
http://www.ted.com/talks/matt_ridley_when_ideas_have_sex?language=en#t-100193
… or, when the foundation gets laid …
The problem with climate models is dogma. The scientists abandoned the systematic understanding of natural climate drivers decades ago to join the CO2 warming bandwagon. Greenhouse gas forcing became the major part of every model. Today, it is clear that such thinking is flawed, but unfortunately the understanding of natural climate drivers is stuck where it got to decades ago.
Not sure what “program” they used for the ‘bot, but it wasn’t “follow the right side”. (following the left side is just “wrong” 😉 )
Yup: https://www.daniweb.com/software-development/c/code/463066/right-hand-rule-maze-solver
End first, then we will show you how we got it!
Mr Worrall
Thanks for the video. It recommends what I have already practiced for some time.
You may have seen some of my dozen or more correlations, among many others you may not think much of the results, but all of them are product of the process described in the video.
I plot set of data and if I find it interesting I do spectral analysis, if that looks interesting take next step etc, etc. Here is an example:
Dr. S goes on about his SSN revision, I read his paper and find that he uses geomagnetic ‘Y – east component’ as one of the metrics. Now, I go to the geomagnetic data and plot annual values of Y component, expecting to see sunspot cycles. Not a thing, but than I realised it looks like the AMO, and here it is:
http://www.vukcevic.talktalk.net/GMEC-AMO.gif
Dr. S pronounced it as a ‘spurious’ correlation, adding ‘vuk is not stupid but …misleading people’.
I will watch the video’s second half a bit later. Very much recommended !
Perhaps it is true that the “y” component correlates well with the AMO and it is still not the driver of the AMO, but since the “y” component can be correlated to solar activity in general, then perhaps the “y” component is simply a good proxy for the solar energy reaching into the depths of the ocean!
It will soon become apparent who is doing the ‘misleading’. I have great respect for Dr. S and his work, but there are major issues with his and the IPCC’s 0.1% solar variation constraint that deserves re-evaluation.
I try to look at anyone who doesn’t understand that the Sun caused the ~ 0.8 degree warming (big deal!) during the Modern Maximum as being slow to recognize the accumulative effect of solar heating upon the ocean (and the reverse). The hard of learning is evident in those already set in their ways.
In that vein, the idea of ‘misleading’ can then be attributed to ‘not understanding’. We can all relate to that on some level, I think. Given that, it has been my personal focus to make that understanding easy and very evident to all, a work-in-progress at this stage.
Keep on truckin’ Vuk!
Hi Bob
Y component is the weakest one of all (in the N. Atlantic it is about 50% of X and less than 10% of Z. It is simply an indication (or partial proxy if you will) of some other ( I would say two other) far more energetic process/es.
Bob
There are two data bases for geomagnetic data : ETHZ and NOAA.
Here
http://www.vukcevic.talktalk.net/AMO-GMY.gif
I have plotted both sets of data, for a single location in the far end of N. Atlantic (sub-polar gyre area, recently discussed on WUWT here .
Since the AMO oscillation follows the geomagnetic Y component, I maintained the AMO time scale and moved GM Y graph to the right along the time scale.
Dr. S’s sunspot correction metric (geomagnetic east Y component) has more to it than Dr. S might like to credit it with. Read all about it here:
http://www.leif.org/research/Sunspot-Number-Corrections.pdf
That’s why I’ve been patient, but it’s a long wait until August… Thanks to both of you for barely putting up with each other, again (and the new links & data) 😉
Unfortunately, Vuk is spouting nonsense as usual. I try, as usual, to teach him how things work, but, as usual, he is hard of learning, but while he is a lost cause one may hope that the lurkers out there are not so unfortunate to share that same affliction.
vukcevic April 4, 2015 at 2:16 pm
Two papers (quoted above) contradict each other
No they do not. It is you who has not understood what they said [as usual].
vukcevic April 4, 2015 at 2:27 pm
Btw. Section of the Pedersen current transversing polar region (see the graph above) I was referring to, is not vertical but horizontal and induces telluric currents. Two vertical sections definitely do not do so. I am a bit sceptical about Hall currents claim.
As usual, you have no idea what you are talking about. Fukushima’s theorem does not require the Pedersen currents to be vertical. And what they induce is irrelevant for the Hall current in the ionosphere that produces rY. As for being skeptical: in order to be that you should know what is going on. Since you do not [and are unwilling to learn], you cannot be skeptical, only incredulous.
Again, you are economical with the truth. The variableto consider is not the Y-component, but the amplitude of the diurnal change of the Y-component. The amplitude of that oscillation is a VERY good representation of the solar cycle. In fact, the best one we have, see e.g. slides 31-38 of http://www.leif.org/research/Keynote-SCOSTEP-2014.pdf
I have a hard time believing that you are ignorant enough [although you might be; you tell me] so that you can’t see the difference between the Y component itself [which arises from deep within the core and has nothing to do with the Sun] and the [much smaller] regular variation of the Y-component that is caused by currents in the ionosphere created by solar EUV radiation.
To start, just to make it absolutely clear as it is stated before, Y component and its variability (diurnal or annual) is far too weak to cause any significant effect; it is simply an indication, or a partial proxy, of some other far more energetic process.
I am not certain of the mechanism linking two (day-light and annual variability), thus I have not suggested one. Equally, I am not prepared to accept your total rejection that daytime and annual (accumulation of daily) changes are totally unrelated.
On page 31 of the paper you quoted (I have looked at once or twice before) there is rather complex flow chart. Because of my limited knowledge of whole complexity of the events I am happy to accept it as a conventional representation of reality.
There are two paths between the yellow coloured boxes (EUV to SFMfC) with number of external inputs. Some of the events involved are just ‘day-light’ transitory, while others leave residual effect from one day to the next, thus propagating the cumulative effect further along the time line into the annual variability.
Just to mention briefly, I have looked into number of papers and presentations (Svalgaard et al) and I am of the view that there is serious omission which would cast doubt on accuracy of an exact SSN – rY relationship. It appears that authors wrongly assume that the geomagnetic storms disturbances never happen around 8-9 am, but more about that on some other occasion.
Equally, I am not prepared to accept your total rejection that daytime and annual (accumulation of daily) changes are totally unrelated.
Then learn that the annual changes are NOT the accumulation of the daily ones. The latter are due to currents in the E-layer that disappear at night, thus no accumulation. And are cyclic: In the morning, the excursion is to East [or West in the opposite hemisphere] as the Sun rises and in the afternoon the excursion is the same amount to the West [or East] as the Sun sets, before returning to the same level as before the morning excursion.
This daily oscillation is due to external sources [the ionosphere maintained by EUV]. The slow change of the level from day to day and thus accumulated from year to year is due to internal sources deep in the core of the Earth and has nothing to do with the Sun.
lsvalgaard April 3, 2015 at 7:11 pm
Again, you are economical with the truth. The variableto consider is not the Y-component, but the amplitude of the diurnal change of the Y-component.>/i>
No sir !
Economical with the truth would be to ‘conceal’ the multi-decadal annual variability of the Y component, as it has been the case in most of the papers devoted to the subject.
– Both graphs quoted from your paper do show title as you have it in the papers.
– Annual variability of the Y component is made up of the final aggregate of the daily measurements, it doesn’t just pop-out on the 1st January (or any other individual day) of each relevant year and then goes away until next year.
– If you care to look at any of your annual normalised graphs you will notice that the peak amplitude is 50-60nT which is line with peaks of the multi-decadal annual variability on my graph.
Just a coincidence ? Ehh, to many coincidences and spurious correlations to your liking ?
Have a happy Easter.
Just a coincidence ? Ehh, to many coincidences and spurious correlations to your liking ?
Here are some examples of ‘Vuk-science’ http://www.tylervigen.com/
Failed again to account for the science we are supposed to take for granted.
No sir, some of us do not take anything for granted
You are still avoiding to address residual changes that occur in just one branch of your paper’s flow chart (page 31) EUV …- Ion formation – Recombination – Ionospheric conductivity – Dynamo process. -…
There are further problems with your ‘sunspot like graph’, there is a magic cut off below ~34nT, there is now link to actual daily values for at least one station, even for one individual year, for purpose of verification for the data ‘vacuum’ below the ~34nT.
In another paper you have an illustration graph of a day-light change (reproduced in my comment April 2, 2015 at 12:09 pm, from Hobart, Tasmania, more than 100 years old. Why not something more recent?
This is what a more recent magnetograph from 1- 3 March 2015 looks like.
Blue line is horizontal component H, red line is declination angle D, from which East component Y can be calculated as Y = H Sin(D).
Above linked graph for any day for two middle quarters (6am – 6pm) in no way can reproduce the ‘Taz’ graph from your articles.
Now, you might claim that these are geomagnetic disturbances produced by geomagnetic storms.
Yes sir, that is the most likely case, but I could not find a single reference to geomagnetic storms in the crucial paper http://arxiv.org/pdf/1407.3231v1.pdf .
A simple search would show following:
the solar activity effect on the changes in the Y component intensity are:
Solar minimum ~ 22 nT,
Solar maximum ~ 36 nT,
Moderate Geomagnetic storm ~ 70 nT,
Severe Geomagnetic storm ~200 nT usually spread over couple of days
Here is graph showing one of the aspects of geomagnetic activity
http://www.vukcevic.talktalk.net/Ap40.gif
The graph shows that for the last 80 years the geomagnetic storms show sort of sunspot cycles pattern but out of phase (delayed), but more importantly that the ‘cycle’ maxima are not proportional to the amplitude of the corresponding sunspot cycle. Smoothed version of the above graph shows peaks more even cycle ‘train’, a bit reminiscent of your rY annual graph..
Using above umbers it can be calculated for a year of a Solar maximum, for an average cycle
365 days x 36nt = annual daily average = 36
20 days of moderate storms x 70nT = annual daily average = 3.84
3 severe storms x 200 = annual daily average = 3.29
sum of the geomagnetic storms gives annual daily average of 7.13 or 20% of the 35 for a year of Solar maximum
This is a significant contribution and since the stronger sunspot cycles (eg SC19) are geomagnetically proportionately weaker than lesser cycles it follows that:
Using daily rY component changes to estimate (or even calculate) sunspot cycle intensity is an unreliable metric.
Let me continue your education:
1) The ‘cutoff’ at 34 nT is because 34 nT is the lowest rY can go. It reaches that value at every solar minimum.
2) The data for Hobarton is data obtained at that station in 1848 when there was an observatory operating there. The observatory operated during 1841-1854 and not since. My rY graph is based on numerous [dozens] observatories during 1840 and 2014.
3) The rY variation is caused by solar EUV, not by the solar wind.
4) Geomagnetic storms cause intensification of the Ring Current [Van Allen Belts] and the Auroral Electrojects [seen e.g. at Tromsoe]. Those currents flow East-West, hence their magnetic effect is North-South. The rY [which is East-West] is thus insensitive to those currents and their effects average out to nothing. Which is why rY was chosen in the first place.
None of this is controversial. All of this is well-understood in quantitative detail. That you don’t understand anything is a defect of you, not of rY.
Using daily rY component changes to estimate (or even calculate) sunspot cycle intensity is an unreliable metric
Quite the contrary, as this simple plot shows:
http://www.leif.org/research/rY-F107-SSN.png
The rY is an excellent measure of EUV, F10.7, and SSN for the reasons I have explained.
To continue your education:
As I explained, geomagnetic storms have little effect on rY. In the plot below, I show the variation of the three components [H,D,a dn Z] for Kakioka [Japan] since 2015, March 1st. A strong magnetic storm occurred on 17th March. The effect on H of the intensification of the Ring Current is a severe depression of the Horizontal Component. On Z, the effect is a strong increase of the Vertical Component. It takes about a week for those effects to go relax away. But there is only a minor, short-lived disturbance on D [which is essentially the Y-component: Y = H sin(D)] which simply adds a bit of random noise to the values. That noise averages out over the many days that make up a year, so has no effect on the yearly average rY.
http://www.leif.org/research/Diurnal-Var-Kakioka-During-Storm.png
Using daily rY component changes to estimate (or even calculate) sunspot cycle intensity is an unreliable metric
It is generally accepted and well-understood that
“[20]1. Sq(Y) is strongly controlled by solar activity and reflects the characteristics of each solar cycle.
[21] 2. The solar activity dependence is caused almost completely by the solar activity control of the ionospheric conductivity…” from http://www.leif.org/EOS/jgra50386-Sq.pdf
So, Sq(Y) [or as I call it, rY] is an excellent indicator of solar activity.
Small variations of underground conductivity and of the slow [and varying from station to station] secular variation of the main field are minimized by computed the average rY from many stations across the globe.
“The advantage of the Y component is that Sq is the clearest in the component in middle latitudes, where most observatories that provide long-term data are located, most clear in the component, and the Y component is less sensitive to disturbances in the geomagnetic field” [thus no geomagnetic storm contamination.
Here are some examples of ‘Vuk-science’ http://www.tylervigen.com/
Sir, you swerved again away from discussing shortcomings of the Sunspot revision proposal.
And what shortcomings do you have in mind?
2.The data for Hobarton is data obtained at that station in 1848 when there was an observatory operating there. The observatory operated during 1841-1854 and not since. My rY graph is based on numerous [dozens] observatories during 1840 and 2014.
Why use the extinct Tasmanian ‘Taz’ from 160 years ago, why not a more recent data: London, Paris, Potsdam, Moscow etc
Just give us one weeks hourly, and one year daily data files links. I’m sure they must exist else you wouldn’t use the graphs without the data.
4. ) Geomagnetic storms cause intensification of the Ring Current [Van Allen Belts] and the Auroral Electrojects [seen e.g. at Tromsoe]. Those currents flow East-West, hence their magnetic effect is North-South.
Pedersen currents flow North-South and vice versa at different latitudes (partial neutralising effect is only in a very narrow band) .
http://www.vukcevic.talktalk.net/Image1.gif
Decades ago it was shown by Chapman that a cloud of charged solar particles approaching polar regions induces electric currents in the Earth’s surface layers causing globally sudden change in the Horizontal component of the geomagnetic field , a direct result of these currents, as it can be seen on any magneto-graph.
East component Y = H * Sin (D).
Since H changes during any storm and D 0 degrees, in severe storms H changes up to 5% peak to peak and D moves up to 2 degrees, it is plainly obvious that the
geomagnetic storms do cause daily, hourly, even down to seconds changes in the East Geomagnetic Component Y
By ignoring effect on geomagnetic storms you are failing to address shortcomings of the rY metric for revising SSN data.
The AMO – Y correlation I found (and you would prefer to not know about) it is most likely to the great extent due to the above.
You are not paying attention. Try reading this:
http://www.leif.org/research/Reconstruction-Solar-EUV-Flux.pdf
If I want to know what rY was in 1848 it makes sense to use data from 1848, wouldn’t you think.
On Slide 7 you can see that to measure the effect of EUV one has to stay away from high [and low] latitudes. Only stations within the two red boxes are to be used for this. The ring current and the polar regions show other effects not related to solar EUV. Do yourself a favor and try to learn something instead of posting irrelevant factoids [that you don’t even understand].
The Pedersen and field-aligned currents [orange] have no effect on the magnetic field at ground level. What we see is the effect of the [yellow] Hall currents.
Again, try not to pontificate on things you do not understand.
At mid-latitudes where H = 20000 nT, a severe storm [only a handful occurs in the most active solar cycles – none so far in cycle 24] has dH = 400 nT [the limit for Kp=9]; if D changes by a random 2 degrees the change in Y is a random 13 nT, which when averaged over a year is negligible. It is painfully obvious that you have not thought this through [or at all]. Again: the diurnal variation of Y is caused by solar EUV [not solar wind storms]. The secular [slow] changes from year to year are caused by convective movements deep in the core of the Earth, and have nothing to do with the Sun, or the climate. Gauss showed us how to tell from a magnetic variation if the source was external [rY] or internal [secular changes to Y].
D 0
D is more or less than zero
singes disappeared !
not much noise in this one
http://flux.phys.uit.no/cgi-bin/plotgeodata.cgi?Comps=dhz&tint=3hrs&block=0&day=2&mnt=03&year=2015&site=bou1a&update=+Update.gif
Look at the whole month:
http://www.leif.org/research/Boulder-HDZ-March.2015.png
and note that D is a bit noisy when we have a storm, but goes down and up such as to have no lasting effect.
Just as I showed for Kakioka
http://www.leif.org/research/Diurnal-Var-Kakioka-During-Storm.png
It is about time you learn something. Try to make an effort.
You constantly try wriggle out of the essence of the problem, when you find yourself in a deep ditch. First you say there is no east-west shift, then it is noise, then it cancels itself out, while your paper clearly states it is min-max for the day, or is it at 9am & 3pm, which one is it?
Noise or no noise, now you are migrating towards equator (my graph which you originally decide to attack is for 60N, top of the subpolar gyre), any excuse but no actual data!
Now you say “The Pedersen and field-aligned currents [orange] have no effect on the magnetic field at ground level”.
Absolutely wrong!
The interplanetary electric field affects the Arctic’s ionosphere, generating global circuit of Pedersen currents in the ionosphere, the effect propagates southwards. Resulting time-varying magnetic fields induce electric currents in the ground !
Then you say “severe storm [only a handful occurs in the most active solar cycles – none so far in cycle 24”
Wrong again !
There was one only 2 weeks ago
http://thewatchers.adorraeli.com/data/uploads/planetary-k-index%20march16-18-2015%20kp-8-g4-severe.gif
I shall leave it here, until you produce actual data.
Wiggle matching is not good enough to eliminate Grand Solar Maximum of 20th century. No data no contest.
Data for all observatories are here: http://www.wdc.bgs.ac.uk/catalog/master.html
The min and max must be taken at 8 am and 2 pm. rY is the average difference between the 2 pm and the 8 am Y-values. Those are the times when the effect from the Sq vortex are largest.
A truly severe storm [with an amplitude of 400 nT or more] has Kp = 9. None so far in this cycle.
The average value of Y anywhere on Earth has nothing to do with solar activity, climate, magnetic storms.
http://en.wikipedia.org/wiki/Naoshi_Fukushima
“He is best known for proving that, under certain conditions, the magnetic field from a field-aligned current and the magnetic field from the associated Pederson current in the ionosphere would exactly cancel at the surface of the Earth.”
Although the conditions are not exactly met, they are close enough that it is generally accepted that the effect we see at ground-level is due to Hall currents [East-West electrojets].
Figure 2 of http://www.swsc-journal.org/articles/swsc/pdf/2013/01/swsc130002.pdf shows the Hall current East-West electrojets [by scientist at Tromsoe…]
“The dynamics, sizes and shapes of the auroral ovals are determined by the interaction between the geomagnetic field and the solar wind. This interaction allows for currents flowing in and out of the upper atmosphere, and these currents are closed by horizontal currents in the ionosphere. Owing to channels of enhanced conductivity produced by particle precipitation, large Hall currents flow along the auroral ovals. These are termed the auroral electrojets. In the dusk sector in the northern hemisphere, the electrojet flows eastward, and in the dawn sector it flows westward.” Hence their magnetic effects are North-South.
Again, try to learn from this.
“severe storm [only a handful occurs in the most active solar cycles – none so far in cycle 24”
Another excuse, now has to be a ‘truly severe’, 400nT, funny that.
BGS and NOAA mark a sever storm at around 200nT and above.
I will look at BGS data and see what can be extracted out of it, but for now here is a BGS graph for change in D directly relating to the East geomagnetic component Y = H Sin (D)
http://www.geomag.bgs.ac.uk/images/dDtimeseries.jpg
more than a bit like AMO (or my top graph) but with shifted time scale.
Not Funny. You claimed ” in severe storms H changes up to 5% peak to peak”. 5% of 20,000 nT is 1000 nT. The limit for Kp = 9 is a mere 400 nT http://isgi.latmos.ipsl.fr/des_kp_ind.html None that strong [not of your 1000 nT] this cycle. The currents in the ionosphere that cause rY are Hall current as your link correctly says.
“Related ground-based magnetic (primarily north-south)” and “PSD time series of horizontal and vertical magnetic fields recorded at 7 monitoring stations in northern Chile exhibit a pronounced temporal variation of the east-west component with an amplitude of about 5 dB for periods between 100 and several thousand seconds. This pronounced variation of By is observed between solstice seasons and daytime and nighttime. Seasonal variations are also observed for the north-south and vertical magnetic components but with 3 orders of magnitude smaller power.” This is BTW for short period variations not the several tens of thousands of seconds of the daily variation. Bottom line: What we observe are East-West Hall currents as I have said. Those have no effect on rY. Try to avoid to pontificate about things you do not understand.
Naoshi Fukushima was wrong about Pedersen currents
See Space Physics 2012 paper
http://onlinelibrary.wiley.com/doi/10.1029/2012JA018008/full
Abstract: ……….”The ground-based measurements of magnetic and electric fields show statistically significant coherences with the interplanetary electric field derived from solar wind and interplanetary magnetic field data of the Advanced Composition Explorer satellite. The interplanetary electric field (IEF) penetrates the polar ionosphere from where it propagates toward equatorial latitudes by waveguide transmission, with ionosphere and solid Earth acting as conducting boundaries.
… We conclude that the IEF drives primarily a global circuit of Pedersen currents in the ionosphere. Resulting time-varying magnetic fields induce electric currents in the ground. Related ground-based magnetic (primarily north-south) and electric (primarily east-west) signals vary coherently at all local times and seasons. Conversely, magnetic signals caused by the IEF-driven Hall currents depend much on local time and season. We show for the first time that these ionospheric Hall currents cause no induction in the ground, but they generate magnetic signatures that are confined to the waveguide between ionosphere and Earth’s surface.”
So it is Hall currents (yellow on the graph) that do not induce telluric currents. You are confusing Pedersen and Hall currents. What might be excuse this time?
The situation is as follows: the S(Y) current vortex is a Hall current in the ionosphere flowing North or South in the Morning and Evening with resulting magnetic effect in the East-West direction [rY]. This is what is related to the EUV and the sunspot number. As shown in my diagram about causes, there is also currents induced in the ground [at depth] and in the oceans. The effect of those currents decrease the observed rY by some 25% depending on the location, hence the need for normalizing the record to a common station [Niemegk]. Special conditions apply to the polar caps and the equatorial regions which complicate matters, but are irrelevant for and avoided by the use of rY derived from midlatitude stations only. Hence the superb correspondence between rY and solar activity.
And BTW, Fukushima’s Theorem holds provided that the field-aligned currents are perpendicular to the ground which they are approximately in the polar regions [corresponding to the Figure you showed]. The situation near the equator is different as the field lines are nearly horizontal there. So Fukushima was not ‘wrong’. What is wrong is to believe [as you assert] that the theorem should hold where it is not applicable.
Uh, thanx for pointing out my being economical with words above. It’s the daily change in the “y” component. Haste makes waste – and I’ve seen you do it too.
Your new SSNs will enable me to tune my solar model, and thank you for being so meticulous and thorough with your reconstruction, so that we can therefore better understand how solar activity affected the Earth during the past three hundred years.
Certainly, without a correct measure of solar activity it is hopeless [and indeed stupid] to try to assess the influence of solar activity, if any. Unfortunately this truism is widely ignored.
Two papers (quoted above) contradict each other, and since they were published shortly one after the other two sets of authors may have not be aware of contradictions.
To be honest, I had more than enough for one day, but maybe tomorrow or in a day or two, I will show graph that combines solar toroidal magnetic oscillations with what as you said above “caused by convective movements deep in the core of the Earth” produces multi-decadal ‘oscillation’ coincident in time a form with the annual changes in the geomagnetic east component Y as shown in my second graph, and coincidently when delayed with the AMO.
You say that this ‘has nothing to do with climate’ .
Maybe or maybe not.
Btw. Section of the Pedersen current transversing polar region (see the graph above) I was referring to, is not vertical but horizontal and induces telluric currents. Two vertical sections definitely do not do so. I am a bit sceptical about Hall currents claim.
Reblogged this on Rebekah Slusher and commented:
This applies to so many areas and disciplines! Several social programs come to mind, where we keep doing the same things over-and-over, expecting different results.
So many scientific discoveries are made by accident. Can robots be trained to detect these?
It depends what you mean by a discovery. Training AI systems to detect shifts in financial markets is big business these days. AI systems are used extensively in structural chemistry, to tease out details of difficult structures from available clues. Google language translation is not quite the joke it used to be.
As computers become more powerful, and insights into how intelligence works becomes more advanced, it seems likely to me that AI will take an increasing role in fields such as pure research.
Novelty: Milky Way is corrugated and much larger than we thought
http://news.rpi.edu/content/2015/03/09/rippling-milky-way-may-be-much-larger-previously-estimated
Does this affect the amount of dark matter??
One step forward towards eliminating the ‘need’ for the three darks: dark matter, dark energy and dark (gravity) force
Something I see all the time is that people — even those with a firm understanding of the scientific method — have a model of some phenomenon burned in their mind, and this precludes them from entertaining input from other models. If they hear one sentence fragment that contradicts their pet model, they rise to quash any further discussion.
Personally I try to think in terms of a ranked list of models for a given phenomenon. Even if the second best model is totally lame relative to the (current) best model, I keep my eye on it. I try to understand it in toto. I give it a fair chance.
The ranked positions can change at any time with new input, but this process can get short-circuited by a closed mindset.
I suspect people get bullied out of this approach because if one entertains more, shall we say, colorful options, one quickly gets labelled as a pseudo-scientist or crank or whatever. There is unseen peer pressure, even if it comes from ourselves.
Frankly, I find there is little downside to keeping a mental ranked list of possible theories for some phenomenon. It’s not like mental storage space is limited. The upside is that it promotes an open mind (but not so open that one’s brain falls out).
Does anyone else think in these terms?
You have a correct perception Mr. Photon. It happens all the time everywhere.
Max, I agree with this assessment. This is sort of how I do it. There is just one nuance that I will state. I first noticed it in graduate math courses. Often in approaching a problem I get a sense of gravity or direction or a way to go or a feeling. The impetus is initiating and does not deter proper techniques once I am under way. But there is definitely a feeling or a sense that I am going in a certain pathway. I am often wrong or I can’t snag the root of the problem sometimes, but it has helped out numerous times. I don’t know what to call it. Novelty, wonder…it is real and it precedes disciplined scientific activity.
Call it a “gut feeling”. Intuition.
Friendly reminder:
Don’t forget to check out the lunar eclipse!
Oops … that’s not the song I wanted.
Yeah if the moon had been that big this is what it would have looked like (~3 billion years ago):
http://i.dailymail.co.uk/i/pix/2015/03/23/13/26EC666100000578-3007624-image-a-16_1427115769595.jpg
Here we go … (much better) …
This theme of novelty was and is popular for logical science type puzzles. It is great fun to explore, like Eric’s example of the maze.
Geoff
http://www.geoffstuff.com/martin.pdf
It IS great fun. So much for my productive evening though.
(I think the trick is to use circles 😉
This is an example of the reader getting set-up by the examples. The 2 examples are complex as a whole, so equally complex patterns of arrangement of the parts are needed. I tried a few patterns for arranging internal parts of the square, failing to find a solution.Then I realized as was looking for far too complex of a solution.The box is a simple shape, so I tried a simple pattern: Running 4 equally spaced parallel lines through the box, either vertically or horizontally, divides the box into five equal sized rectangles.
SR
I probably should have specified that the width of each rectangle should be 1/5 of the whole square so no one can find a way to misunderstand my solution.
SR
SR,
Congrats, you solved the problem by discarding the ?misleading info in the examples. It took me a couple of hours, many years ago, before I hit my forehead and went DUH!! Some really respected mathematicians laboured over it in Sci Am until their agony was relieved in the issue a month later.
But now, how would you work the logic if you wanted to write a computer program to address the puzzle as put?
Cheers Geoff.
Geoff,
Under 2 minutes, but then I’ve always been considered simple-minded ;o)
Wait, wait, wait.
The instructions say there are at least TWO solutions.
I took that to mean that the trivial solution is dividing the square by five rectangles (which I got in about five seconds), and set about to find a non-trivial solution.
Am I the only one reading the instructions this way? Did anyone find a non-trivial solution?
Okay, I’ll admit to being tricked by complexity.
Rotating the trivial solution 90 degrees is another solution. That’s “at least two.”
I thought of that in 10 seconds, but thought there was something more interesting to the problem … which I suppose is interesting.
” you reward the robot for innovating”
Robot reaches pulls self on top of walls, walks along walls until reaching its goal. Innovation.
I seriously thought of making the wall walking solution my second scenario – but it probably violates the “physics” of the simulation world… 🙂
Innovation. Think – outside of the box LoL
Science may have entered ‘post normal’ without any help from whacked out philosophers who nowadays don’t ‘philo’ the ‘sophus’. The frustration of physicists of the past (say) 75 years since the close of the golden age is palpable. All these desperate brains with nothing to do. It is not even a search for the novel. As long as they take as chiseled in stone what is ‘known’ about general relativity/gravity they, like the robot above, are destined to come up with dark matter, strings and other ridiculous things. Having discovered something wrong with gravity of big things like galaxies and perhaps small things like, electrons and protons, and quarks, etc for which they had to invent totally new sets of forces: strong forces and weak forces and intermediate forces all to deal with the same matter – this is what they do instead of looking for (an)other factor(s) or variable(s) that makes Newton’s and Einstein’s gravity, Van der Walls, and intra atomic forces into special cases of a new and more general theory.
What’s the next step after 75 years of scientific silence in the search for novelty? Why it’s to invent stuff and then beat, bend and wrought the data until it fits. Not only is it ‘novel’ its profitable. And when this is finally abandoned? The end game for science would seem to be to to evolve like music. The great composers seem to have used up all the notes and there is nothing left but to become a performer of the great works of others. Lab coats and horn-rimmed glasses become the tuxes and tails of the performer.
This is good insight for problem-solving, keep trying something new.
It’s how evolution works and it’s also how the immune system operates (g.o.d. = generator of diversity).
I assume most of readers have gone elsewhere by now, but
as promised in my comment on April 4, 2015 at 2:16 pm
here it is:
http://www.vukcevic.talktalk.net/SUN-and-AMO.htm
There are things worthy of debate. That is not one of them.
Regardless of your ignorance about the basic physics of conductivity, currents, induction, plasma, etc there is no doubt that the diurnal variation of the geomagnetic East component is an exquisite measure of solar activity, expressed both by the F10.7 cm microwave flux from the Sun and the International Sunspot Number, with correlation coefficients of 0.99. In addition, we know the physical processes that are involved so there is no question about correlation vs. causation. The data is public and readily available, so anybody can check the calculations. There is nothing new nor controversial in this, the effects being known for centuries:
http://www.leif.org/research/rY-F107-SSN.png
The present comment brings a fitting closure to the matter.
The present comment brings a fitting closure to the matter.
I strongly doubt it .
You still have to get it accepted, and no sign of that at the moment.
I looked at some of the BGS data (thanks for the link), for the H component there are solid numbers, however D changes from morning to the afternoon are tiny (order of 0.1 degree at Lerwick), calculating SinD differences at these values is prone to a relatively large uncertainty with ratios of 36 to 15,000, or approximately 1:400.
As you can see from my post, there is an ‘uncomfortable’ link of the main source of natural variability AMO to the matters solar. Eeven the NASA’s new climate supremo, btw: we both went to the same university, perhaps he is just being kind, has found something of interest, else it would have been dispatched to the BoreHole
http://www.realclimate.org/index.php/archives/2015/04/unforced-variations-april-2015/comment-page-1/#comment-628160
calculating SinD differences at these values is prone to a relatively large uncertainty
Nonsense, the Sine Function can be accurately calculated with any precision that you want, hundreds of decimal places if so desired. No uncertainties at all.
R^2 is a neasure of how much of the variation is the same for two series. The statistical significance of the regression for F10.7 on rY is of the order of 10^(-40) with the 68 degrees of freedom.
About ‘uncomfortable’ links: they only become so when significant, so as I said: no need for further discussion of your spurious correlation.
R^2 of 0.97 and 0.98.
Autocorrelation ?
Natural (unforced – as Gavin calls it) variability is the next big thing. Mann and co are now talking about subpolar gyre slowing down (cooling in another words), Gavin kept on his blog my ‘spurious’ correlation. Soon they will need a reason for downturn, no major volcanoes, no aerosols to blame, Chinese pumping CO2 and temps stalled. Only thing they can blame is sun, which conveniently is going to take a rest. You have no chance of selling your revision, it came 10-15 years too late.
Irony of it all is that you may well be right, and that there is not much to it above 0.1C, but that will not matter.
Re. sun-earth oscillation phase differences; recently I came across rather strange phenomenon: the Earth has two kinds of year: astronomic and geomagnetic, and they are of different length !
I hope you had a good holiday break.
See you elsewhere.
The Revision of the Solar Activity is ongoing, and need not be ‘sold’ as it is firm. It will likely be adopted by IAU in August. It is obvious that there is a 0.1C solar cycle effect. All the rest is spurious and in some cases plain garbage. Such does not deter some people in their search for said garbage.
You must remember that the rY connection is not my idea. It has been known a long time and is widely accepted and its physics is known. All I have done is to use all available data [some Gigabytes] to construct a complete time series. You could benefit from reading http://www.leif.org/research/Radio-Ionosphere-Magnetism-and-Sunspots.pdf
posted comment in a wrong place.
see further below: April 6, 2015 at 3:40 pm
I once programmed a game sometimes called “Othello” or “Reversi”. Since I programmed it, I knew its algorithm and could easily beat it, no matter how intelligent I tried to make it. Then I added some uncertainty, random weighting of various moves. Sometimes its move is stupid, as are some of mine, and sometimes its moves seem brilliant although of course there’s no such thing; it is a mixture of algorithm and random. I let it look for the “long shot” several moves in advance but whether it decides to try for the long shot is somewhat random.
Wow , I never expected to encounter an example I was going to mention .
I find this article interesting , but I’m not overly impressed . Perhaps my greatest professor , and one of my strongest supporters was Don Campbell whose Blind variation and selective retention , 1960 , and Evolutionary Epistemology make very similar arguments .
The actual images and their algorithms remind me a lot of Brad Nelson’s Forth Haiku . See his very impressive presentations at the last couple of Silicon Valley Forth Interest Group Hangouts , particularly February’s meeting .
Getting back to Othello , Martin Gardner had a column about it in his Mathematical Games column in Scientific American back around 1963 when I first met a computer , an IBM 1620 . Essentially its a game where the object is to find a path from one side of a board to another . As Michael2 says , programming the winning strategy was pretty easy . However , having the computer figure out whether it had a connected path from one side to the other was not . Essentially it had to run a maze and know when it got all the way to the other side .
I worked quite a while on it in the available FORTRAN and kept running into the need to keep tables of state I had not anticipated . Only somewhat later , when I learned Algol , which has recursion built in , did I succeed . ( In an APL , I later implemented it in about 5 very short definitions . )
In a psychology course on the theory of learning , I submitted a paper titled Maze Running Behavior in the CDC 3400 emphasizing the necessity of recursion to provide path memory . The significance of this insight was , I feel , under-appreciated and I only got a B .
At any node , it simply tried “left” , then “forward” , then “right” , or backed up and tried the next direction at the last not exhausted node . It had no metric sense of distance from it’s goal , just success if it got thru . So , in the present terms , the only thing the “program” required was the requirement of novelty – not repeat a previous path .
Yea, thanks for the link. I looked at it not long ago, will do again.
I find it rather odd that the flow chart (here on page 32) I mention before doesn’t leave any longer term residuals, and that annual change is approximately same as the daylight max-min difference.
How they calculate annual, just look at two dates a year apart?
However, nothing in it about mesosphere, the one I forgot it existed, and you concluded (!wrongly!) ‘ vuk is not stupid ..etc’.
Talking about ‘stupid’, when I got the latest correlation, I couldn’t believe how good it was, but I am not that stupid to think that it was or will be there for ever, possible but highly unlikely to be coincidence, even 150 years is far more than could be expected.
16 years is the second strongest component after 21 in the Jackson data, but I have no clue where it comes from.
Is there a point of asking: have you ever come across 16 yr period ?
I searched for it and found it only in the Himalayan monsoons, after vague reference by a NZ scientist. I found data, did spectrum and it is there.
A long shot idea: Monsoons take millions of tones of water (in a short time, takes years for it to get back) from equatorial region to the higher latitudes, and in doing so changing rate of rotation (angular moment and so on).
Oh well, never mind ….
I have no particular conviction about any of this stuff , but it it is lot of fun and keeps me out of pubs; if it wasn’t for my daughter’s homework you wouldn’t ever heard of me.
good night
Here it is
http://www.vukcevic.talktalk.net/HM.gif
(data 1870-2000) with elNino, 16 years and Miyahara (not only at Maunder min GCR) 28 years.
Many things that look odd are quite natural, e.g. that the Moon has the same angular size as the Sun.
The annual change can be calculated in many ways: e.g. by computing the average for each year, or by a fancy low-pass filter. The diurnal variation over a year is always positive [greater than 34 nT], while the annual change is negative for some locations and positive for some locations. Averaged over longitudes at a given latitude the secular change is close to zero. The mesosphere is not mentioned because the electric currents that cause the diurnal change does not flow in the mesosphere. Occasionally a large solar flare will produce enough X-rays to ionize the D-layer giving rise to a Sudden Ionospheric Disturbance [SID http://solar-center.stanford.edu/SID/activities/activity.html ] for something like an hour. These are not cumulative and have negligible effect on rY. About 16 years [and other cycles]: if you look for cycles you will find cycles, most of them will mean nothing, as this one:
http://jonova.s3.amazonaws.com/graphs/temp-co2/us_post_causes_global_warming_lrg.jpg
Yours fall in the same category.
The Aa index is very similar to the Ap index, but not exactly the same since it is calculated from only the two stations’ data Canberra (Australia) and Hartland (UK).
Because of only two stations being used for Aa, Aa is inferior to Ap which is based on about a dozen stations. So Aa is a poorer index than Ap and less representative of true geomagnetic activity, hence more likely to be show spurious effects.
even 150 years is far more than could be expected.
No, if you have 60-100 years cycles then in 150 years you only have about two degrees of freedom and can easily get a couple of cycles to match, just like getting two heads in a row when tossing a coin.
Correlation I had in mind is not directly related to Y, it is this
http://www.vukcevic.talktalk.net/Sun-Earth.gif
Significant 16 years (above noise !) only appears in monsoon, Jackson geomagnetic, Aa but not Ap index and only Arctic temps data.
Thanks anuway.
Since Aa and Ap are just two expressions of the same thing, this is a sure sign of a spurious, non-significant signal.
The Aa index is very similar to the Ap index, but not exactly the same since it is calculated from only the two stations’ data Canberra (Australia) and Hartland (UK)
vukcevic April 7, 2015 at 1:49 am
The Aa index is very similar to the Ap index…
Because of only two stations being used for Aa, Aa is inferior to Ap which is based on about a dozen stations. So Aa is a poorer index than Ap and less representative of true geomagnetic activity, hence more likely to be show spurious effects.
I will say that VUK’S correlations are never straight forward and there is to much of this adjustment then that adjustment to make it come to some sort of fit. To much adjusting and accounting for my taste.
I am simple when it comes to this stuff maybe because my knowledge is not as great but I like correlations that are straight forward and either they work and hold up or they do not.
Vuk’s conclusions on the AMO versus solar activity , have been shown by others to be in direct opposition to what he shows. Some have made the point that during weak solar periods the AMO is positive and vice versa.