Estimating Cloud Feedback From Observations

“Everything must be made as simple as possible. But not simpler.” ~Albert Einstein

Looks like a random line drawn through an amorphous cloud of points.
This is the same problem I have Dressler, Lindzen & Choi, and Spenser & Braswell. There might be something there, but it sure is well hidden, if it is there. At the end of the day I am less certain of any relationship after all the smoothing and statistical gymnastics to tease something of value from the cloud, than I am with just the scatter plot cloud.

Willis,
An interesting analysis that makes sense. One nit. OK two nits.
“That is to say, to warm the surface by 1°C, the amount of incoming energy has to increase by about 5 W/m2. This, of course, means that if there were no feedbacks, a doubling of CO2 (+3.7 watts per square metre per the IPCC) would only cause about 3.7/5 or about three-quarters of a degree of warming.”
These numbers confused the day lights out of me because SB Law shows a bit over 5 watts/m2 to raise the average surface temperature (15 degC) by one degree, but the IPCC meme is that CO2 doubling = 3.71 watts/m2 = + 1 degree. I had to go back to AR3 to figure out how they came up with it. The answer is that they calculate +1 degree against the “effective black body temperature” of the earth, which is about -20 degC. Given SB Law is P=5.67*10^-8*T^4 with P in watts/m2 and T in degrees K, the math comes out correctly. 3.71 watts/m2 to go from -20 degC to -19 degC and a bit over 5 watts/m2 to go from plus 15 to plus 16. Or, as I prefer to put it, CO2 doubling = 3.71 watts/m2 = +1 degree as seen from space = ~ 0.6 degC “on average”. There being no such thing as “average” when it comes to surface temperature, the change at -40 in winter at high latitudes is larger than 1 degree, and the change in temperature at +40 in the tropics is not only less than 1 degree its far less than the “average” of 0.6 too, more like 0.2. Of course that’s assuming that the 3.71 watt/m2 is uniform. I’ve never checked for myself, but logic suggests not.
The other nit is time constant. I haven’t really thought this through all the way. If the 5 time constants exceeds one month, then some of the cloud feedback shows up in the next month or perhaps the month after that…given that you’ve gone all the way around the calendar however doing a month over month analysis, I’d think the average for the year would be representative.

Don’t know what the r^2 is, but it looks like it sucks…

[snip – way, way off topic – w.]

Which program is the code?

Easy for you to say … seriously, I need two things for that, a land/water mask, and 28 hours in a day …
KNMI lets you download monthly temps by land only and sea only for hadcrut. they also show a cloud cover for land only from CRU and a sea only from COADS. Would that help instead of building a land/sea mask?
On the 28 hours in a day, I can help there too, as I came across this problem some time ago. My solution was to begin also working nights.

Look at the “outliers”.
100W/m^2 with NO TEMP change… I’d like to explain that!… How can you have -100 W/m^2 and no temp change?

Maybe I am missing something here, but I have difficulty seeing how a month on month temperature change can cause a month on month change in cloud forcing.
I’d expect the cloud forcing to be a direct function of temperature. Warmer = more clouds = increase in the cloud forcing. What matters is the temperature, not whether the temperature has increased/decreased.
So I’d expect to see a similar (or larger) effect if cloud forcing were plotted against monthly temperature.

I’m so old I remember being mocked at Climate Audit when I suggested that we knew neither the magnitude nor the sign of the water vapor feedback.
===============

Willisl
Anomalies only, not absolute temperatures, AFAIK. Also no data on the net cloud forcing, just albedo>>>
Select monthly observations, there are several sections under “Land”. The last one (just before the Tmin and Tmax data sets) is CRU Land TS 3.1 1901 to 2009 in absolute temps. I thought there was an equivelant for SST, but I was wrong. There is however MOHMAT which has marine air temps from 1856 to 2007 in absolute temps. The need to correlate to net forcing slipped my mind, sorry. They do have Tmin and Tmax however…would it be of value to correlate cloud cover to those? I’d think that Tmax in particular would lead increase in cloud cover (tropics in particular) but one the cloud cover reached a certain point, Tmax increases would start dropping off? Would be better still to do that same cut at the data but break it up by night versus day.

Not that I believe much in models, but couldn’t you model an earth with clouds of different kinds, in 100% fashion with some of the IPCC assumptions? That is, a model of the earth with 100% “A” – “D” type clouds, to see what the effect is?
Yes, a lot of work, but in the end you would start to get a handle on how clouds effect earth. These could give you the areas to look at: which clouds are most important to temperature change. You could then run different cloud assumptions through the IPCC models of the earth, and see how these different cloud assumptions change earth’s temperature. If there are reasonable assumptions that lead to negative feedback, then it’s up to the modelers to prove their assumptions.

@Willis
[SNIP – off topic. I don’t censor criticism. I censor off-topic posts, as I said I would. – w.]
Censoring criticism is not very scientific so if you don’t want an angry Marine dogging your ass about it for the foreseeable future I suggest you not resort to censoring your critics.

Very cool if it holds up Willis. Not only does there appear to be negative feedback when it is warm, but positive feedback when it is cold.

I see R^2 value has been added to the chart now. .11 is not a good correlation at all…

Looks like about 1% of the points are contributing to this correlation, so this may be a small area of the surface, and therefore the forcing should be divided by a corresponding amount. Can the correlation be applied as if it is global?

Elegantly done, with no obvious holes. I would like to suggest a different dot coloring scheme from the land/water one others above have suggested, and much easier to code. Can you color the dots by latitude, say from red at the equator to blue at the Arctic circle and green at the Antarctic circle? I expect this scheme to illuminate your tropical cloud thermostat theory, and could also contrast the mostly water southern temperate latitudes with the northern temperate latitudes that have a much larger proportion of land.

I have the notion that the feedback varies with latitude, as the cloud types vary somewhat with latitude. I agree with Doug in Seattle, where he says (October 8, 2011 at 4:41 pm):

Looks like a random line drawn through an amorphous cloud of points.
This is the same problem I have Dressler, Lindzen & Choi, and Spenser & Braswell. There might be something there, but it sure is well hidden, if it is there.

That is the first thing I saw, how much it looked like Dressler. And the r^2 value is awfully low.
But I am not so sure that the amorphous cloud will exist if is broken down by latitude ranges. I am thinking sorting it out will show more relationship with latitude as a factor.

Willis,
I watched about the first third of a talk by Samuel Stechmann this morning, “Multiscale Models for Atmospheric Convection and Waves”. I’ll get to the rest tomorrow morning (much better than watching the news). Anyway, the lecture covered the MJO and gravity waves (I did not know such things existed). One of the motivations of the work was that the GCMs do a poor job at emulating these oscillations. Your post stirred a thought related to these oscillations, specifically the MJO. By their very nature, such oscillations have to be self propagating or there would be no oscillation.
I have not dug deeply into either Spencer & Braswell or Lindzen & Choi, but a quick reading gives me the impression that their models, as well as your analysis here deal with feedback over time but the positions are stationary. Perhaps, the correlations would be tighter if the moving waves set the reference frame, rather than using geographically fixed positions. It would make the math a little hairy. Just a thought.
One of the problems with the GCMs is that many of the processes that influence the heat flux is that they occur on scales much smaller than the grid size in the simulation. The only way to deal with this is to parameteizer the process so that the output is as close a possible to observations averaged over the grid cell. Since thunderstorms and their influence on atmospheric processes necessary for wave propagation are clearly sub-grid phenomena, I am not surprised that GCMs perform poorly in capturing the MJO and other wave phenomena.
http://online.kitp.ucsb.edu/online/turbulence11/stechmann/
The following talk by Joao Teixeira from CalTech discusses low level clouds in the northeastern Pacific and how parameters might be set. It’s yet another significant problem for the GCMs, because if the proper amount of clouds is improperly emulated, then the improper amount of heat enters or leaves the ocean with significant affects on ocean and air temperatures. These errors are then carried forward throughout the simulation. Mmmm.
Turbulence, Clouds and Climate Models http://online.kitp.ucsb.edu/online/turbulence11/teixeira/
Home page of KITP: http://www.kitp.ucsb.edu/

Now what could be causing that vertical spread of southern hemisphere data? That is really interesting. Thanks for the color coding. Now what happens when each hemisphere is looked at separately?

From the standpoint of feedback considerations, it might be important to distinguish between two cloud types, which are fundamentally different. The first type I will call an atmospheric advection cloud. These are created by the interaction of two different air masses moving against each other. Thus they have little thermal contact with the surface except via LWIR. Surface conditions have little to do with the formation of these clouds, so I would expect minimal direct ground feedback in this case.
The second type I will call a surface convection cloud. As the formation of these clouds do depend on surface temperature, I would expect a strong, probably negative, feedback associated with them. So, in studies of this kind, I think the word ‘cloud’ by itself may be incomplete if we do not make a distinction between surface convection clouds and atmospheric advection clouds.

@Willis Eschenbach
You make a point about cloud and DLR:
“Net cloud forcing is the balance of how much solar energy the clouds reflect away from the earth on the one hand, and on the other, how much the same clouds increase the “greenhouse” downwelling longwave radiation (DLR).”
Then a big generalized leap:
“That is to say, to warm the surface by 1°C, the amount of incoming energy has to increase by about 5 W/m2. ”
This generalization does not hunt. The type of energy and type of surface makes a difference. For instance if the surface is ice and the energy is shortwave it will raise the temperature very little. If the surface is water and the energy is shortwave it will raise the temperature very much.
More to the point, if the energy is longwave and the surface is water it will not raise the temperature very much. This is predicted by the physics and experimentally confirmed.
http://tallbloke.wordpress.com/2011/08/25/konrad-empirical-test-of-ocean-cooling-and-back-radiation-theory/
This is extremely relevant to your hypothesis, Willis. Snipping challenges to underlying assumptions in your thesis is not very scientific.
I’m putting this comment in a macro and will repeat it ad infinitum in every thread of yours until you acknowledge it. If that doesn’t work I’ll start placing it in threads you don’t control.
Are we clear on what I think about censoring critics and how I intend to respond to you now?
[UPDATE: Dave, you apparently have mistaken me for someone who gives a flying flock about your ideas. If you wish to make a fool of yourself by screaming, thinking that repeating your claims more loudly or more frequently will somehow make them true or will convince people to pay attention to you, I fear I can’t help you.
So, this time I’ll leave your comment here, in the hopes that it will convince you to be polite and go away. You remember “polite”? That’s where, when someone says “Please discuss that claim on another thread, we’re discussing something else on this thread”, you say “OK”, and you discuss it elsewhere.
If you want to prove that you are a jerk, Dave, go ahead and keep repeating your claims, and persevere in trying to butt into a conversation where those kinds of claims are not being discussed. However, people here won’t get to read them, I’ll snip them all day long. WE ARE DISCUSSING ANOTHER MATTER HERE, how hard is that to understand?
So if you wish to pound yet again on your one-note drum, have the common courtesy and simple decency to do it somewhere else. I will snip it here, but not because it is scientific nonsense. It is nonsense, but that’s not the reason. I will snip it because we are talking about something else on this thread, and your contribution is neither appropriate nor wanted.
PS: Threats don’t work with me. You say that you are “putting this comment in a macro and will repeat it ad infinitum in every thread of yours until you acknowledge it.” Yeah, I’m shaking in my boots with fear now, that’s a brilliant plan, everyone appreciates being spammed, the lurkers will love you, and you’ll never get banned for that kind of childishness … you don’t seem to get it. REPEATING YOUR CLAIM DOESN’T MAKE IT TRUE. So go ahead, spam your message on every site you can find. A site discussing cross-stitching methods for needlepoint? Perfect, you’ve proven you don’t care in the slightest what others want to talk about, you are an equal-opportunity bore and you insist on your right to talk “radiation can’t heat the ocean” no matter what the thread is about. So go ahead post your claims on the cross-stitching site and every other one you can find, that’ll show the world you’re not a man to be messed with … – w.]

@Willis Eschenbach
You missed the memo from NASA.
[SNIP – you missed the memo about me snipping off-topic comments. w.]

hi willis,
it might be useful to run a series of fits sliced by latitude. this would also remove the need for area weighting. it would also give u some sense of the stability of the effect at least on a spatial basis.
d

In my view snipping comments because the thread originator decides they are off topic is not a good direction for this site . . . . . on the grounds of abuse, stalking etc it definitely makes sense. It leaves wattsupwiththat open to charges of censorship. . . probably the last thing it needs in the climate debate. . . And yeah this comment is off topic . . but it would be pretty pointless to snip it though . . . if there’s widespread disagreement with it then fair enough you’ll have snipping policy support for the future . . . . .

Willis, both graphs 3 & 4 have the same note about coloring of dots, description for #4 cannot be right. Interesting how tight the SH is compared to NH. One might assume the difference is land mass ocean ratio, and/or a difference in cloud cover related to that ratio. I would think on land there are a lot of differences in localized thermal gradients, whereas the ocean would be more homogenous in albedo and topography. The topography of the land causes changes in water vapor, for example, air moving over mountains loses water content. From the first graph, it is interesting that solid water 0°C cools as clouds.

Willis,
I’m out of my league here – poor in math and knowing nothing of climate science. So let me just jump in with both feet firmly in my mouth:
From Wikipedia, approximately 53% of incoming radiation from the sun is infrared.
From NSIDC(.org) approximately 30% of incoming solar shortwave radiation is reflected back into space. The remaining 70% strikes the earth and is re-emitted as longwave (infrared) radiation.
———————————–
The above stuff I got from the Internet, so if I’m wrong already, forgive me.
———————————–
Assume 1,000 Watts of radiation strike the earth from the sun.
530 Watts is longwave and will have a positive forcing (warms the earth).
470 Watts is shortwave of which 30% is reflected back, leaving 70% or 329 Watts to strike the earth and be re-emitted as longwave (infrared).
Regarding the clouds, do they prevent the incoming shortwave (the 530 W) from reaching the earth’s surface to the same extent that they will prevent the outgoing shortwave (the 329 W) from leaving the earth?
Seems to me that clouds would always have a negative forcing (cooling effect) when incoming infrared is greater than outgoing, at least to a first order approximation.
So what is wrong with my logic here?

Does the difference in slopes between the last two graphs show a real world effect? Could this be part of a real description of the planetary climate engine?

Willis, thanks for adding the latitude and hemisphere data. As expected, it clearly shows the temperature moderating effect of the Southern oceans, relative to the large continental land masses in the North. I find the robustness of the R=-0.27 R2=0.08, regardless of how the data is sliced to be remarkable, and would want to double and triple check it. We know from Dr. Spencer’s post above that cloud feedback, regardless of parameter, can only explain a small fraction of the net radiative forcing, and 8 percent seems within the realm of possibility. We know from Svensmark that cosmic ray modulation of clouds can supply a radiative forcing. If clouds are the thermostat, then cosmic rays, volcanoes, ocean currents, industrial aerosols, etc. all have a finger on the dial.

Willis I like your approach – let the data speak for itself and see how it matches the science.
A few points:
1. My concern is not that largish grids introduce smoothing but that using monthly data does. The effects of cloud cover are, more-or-less, instantaneous and on different days of the month feedback could go either way.
2. There is also implicit smoothing – you assumed the effect is the same for all latitudes for all times of the year which could mask seasonal trends. You have enough data to look at 5 sub-sets: northern hemisphere summer, northern hemisphere winter, tropics, southern hemisphere summer, and southern hemisphere winter. For summer you could take April to September (as close to the period between the equinoxes as is possible with monthly data).
My hunch is that your conclusion is right but the closer you can get to showing the ‘fuzzy’ plot of negative feedback is the balance of more clearly defined positives and negatives the stronger your argument will be.

Dave Springer says:
October 9, 2011 at 5:01 am
“the downwelling shortwave radiation that is the ocean’s only source of heating. Downwelling longwave radiation has no effect on ocean temperature.”
———————————————–
Dave, how is it possible that shortwave radiation is the only source of ocean heating?
Dave, how is it possible that longwave radiation from the sun has no effect on ocean temperature?
Thanks.
[UPDATE – See, this is the problem. Dave starts off with his claims, and right away someone (reasonably) says “Hey, that makes no sense at all”, and then we’re off the rails. I’ll leave this here to illustrate the point, but I will snip further responses along the same line. If you want to dispute the DLR/ocean issue with Dave, please do so elsewhere. w.]

Willis,
I made an error in my earlier post. Of the radiation reaching the earth, only about 1% is longwave (> 4 um). The other 99% is shortwave. Sorry to display my ignorance but that’s how I learn.
———————-
So virtually all incoming solar radiation is shortwave. That explains Dave’s comments.
Which leads me to a new question: Are clouds better at blocking shortwave radiation (negative) or trapping outgoing longwave radiation (positive)?

it also seems particularly strained to insist on explaining conduction, phase change and convective transport in terms of radiation.
ghg-man got bitten by a radiative spider in this comic series so long ago that now it only appears quaint and superstitious.

Willis this looks very interesting , it’s going to take time to digest but one big problem jumps right out at me. OLS regression on this kind of data will give BADLY wrong estimates of any linear relationship that is there.
Linear regression ASSUMES there is negligible error in the independent variable. If this is not the case the mathematical derivation of OLS simple does not apply. The net result is a REDUCTION in the estimated slope. One may imagine there is a grey area centred around the fitted line but in fact it can be shown to always reduce the estimated slope. This is called regression attenuation.
In short , it works well for nice clean lab experiments where you can control the quantity plotted on x axis. If you have noise or other effects, like correlated or uncorrelated cyclic phenomena it will reduce the ols estimator. Neither is this just a minor error that can be ignored in a rough estimate, it may be a factor or 3 ro 4 with the sort of cloud you are looking at.
This can be corrected for to some extent , but this requires knowledge of that nature and relative magnitude of the noise/error quantities that is often difficult or impossible. It is often better to try to avoid this happening that try to correct it.
Ignoring this problem is ubiquitous in climate science. This is principally why Dessler, Forster & Gregory etc get a positive feedback , their maths is bad and they under estimate the slope, leading to a false conclusion of +ve feedback.
Fitting straight lines to scatter plots basically and fundamentally wrong.
I hope this does not undo all your hard work , but it does need to be accounted for. Now to digest the rest.
best regards.

PS, one way to test what I say about regression attenuation is to invert the axes and redo the ols fit. If the result is good one slope will be the inverse of the other. I think you will get a shock how different they are. Clearly the result should not depend on which way you plot the data … unless I’m right about OLS.

Willis you are a liar and coward and you make me sick to stomach, GO FUCK YOURSELF!
REPLY: I’m allowing this post, which will be your last, to show what kind of person you are in your own words. And this is why you are banned, permanently – Anthony

Willis, could you redo the analysis by latitude band? Splitting by hemisphere showed a fairly large difference. Perhaps the feedback strength varies by the type of clouds most prevalent in tropics v. temperate latitudes.

Willis Eschenbach says:
October 9, 2011 at 11:31 am
Steve from Rockwood says:
October 9, 2011 at 10:19 am (Edit)
Willis,
I made an error in my earlier post. Of the radiation reaching the earth, only about 1% is longwave (> 4 um). The other 99% is shortwave. Sorry to display my ignorance but that’s how I learn.
———————-
So virtually all incoming solar radiation is shortwave. That explains Dave’s comments.
Which leads me to a new question: Are clouds better at blocking shortwave radiation (negative) or trapping outgoing longwave radiation (positive)?
Your question is the great unknown in climate science, and is what I’m attempting to answer in this thread.
w.
How much would this change what happens to the clouds?

http://journals.ametsoc.org/doi/abs/10.1175/2010JAS3282.1
Abstract
This paper focuses on two shortcomings of radiative transfer codes commonly used in climate models. The first aspect concerns the partitioning of solar versus infrared spectral energy. In most climate models, the solar spectrum comprises wavelengths less than 4 μm with all incoming solar energy deposited in that range. In reality, however, the solar spectrum extends into the infrared, with about 12 W m−2 in the 4–1000-μm range. In this paper a simple method is proposed wherein the longwave radiative transfer equation with solar energy input is solved. In comparison with the traditional method, the new solution results in more solar energy absorbed in the atmosphere and less at the surface. Li, J., C. L. Curry, Z. Sun, F. Zhang, 2010: Overlap of Solar and Infrared Spectra and the Shortwave Radiative Effect of Methane.

Richard G says:
October 9, 2011 at 12:24 pm
“I would like to point you to a musing from the Chiefio. It is about clouds.”
“Gives Us This Day Our Daily Enthalpy”
http://chiefio.wordpress.com/2011/09/27/gives-us-this-day-our-daily-enthalpy/
Excellent. Thank you. E.M. Smith is one sharp cookie…

Steve from Rockwood says:
October 9, 2011 at 9:37 am

Dave Springer says:
October 9, 2011 at 5:01 am
“the downwelling shortwave radiation that is the ocean’s only source of heating. Downwelling longwave radiation has no effect on ocean temperature.”
———————————————–
Dave, how is it possible that shortwave radiation is the only source of ocean heating?

Because the sun is its only significant source of energy and for all practical purposes there is zero longwave infrared energy in sunlight. But I see you already discovered for yourself that I was correct.
Now you need to discriminate between the ways by which the ocean absorbs and gives up energy. Radiation from the sun is the only thing that penetrates it. That is it’s only source of energy. Longwave infrared is totally blocked by a surface layer that is only a few microns thick. This skin doesn’t mix downwards. Viscosity overwhelms all other forces in skin layer. The skin layer evaporates. Evaporation is the primary way that the ocean gives up absorbed solar energy – 70% of it. 10% is given up by conduction and the remaining 20% is via radiation. This is RADICALLY different from the way land and ice gain and lose energy. There is no comparison between the two. Greenhouse gases are quite effective over land because their mode of operation is via radiation and radiation IS the primary way that land surfaces give up solar heat. Radiative emission from the ocean on the other hand is not the primary means of giving up energy so greenhouse gases play commensurately smaller role there.
Eschenbach refuses to acknowledge the difference between land and water and how they are quite different in the way they absorb and release solar energy. He’s decided that bringing up actual, well established physics that are counter to his anedotally acquired beliefs is “thread hijacking” and has promised to deprive you of this information so that he may pursue his ridiculous wild geese flying through the clouds.
It’s really kind of amusing. An ostrich with his head stuck in the sand chasing after wild geese…
I’ll clip this comment and post it to tallbloke’s website. Tallbloke has forgotten more science than Willis ever knew.
http://tallbloke.wordpress.com/2011/08/25/konrad-empirical-test-of-ocean-cooling-and-back-radiation-theory/

Dave, how is it possible that shortwave radiation is the only source of ocean heating?
Dave, how is it possible that longwave radiation from the sun has no effect on ocean temperature?
Thanks.
[UPDATE – See, this is the problem. Dave starts off with his claims, and right away someone (reasonably) says “Hey, that makes no sense at all”, and then we’re off the rails. I’ll leave this here to illustrate the point, but I will snip further responses along the same line. If you want to dispute the DLR/ocean issue with Dave, please do so elsewhere. w.]
UPDATE TO UPDATE No Willis, if you had the first clue about the physics you’d know that the response was not reasonable. There is practically no longwave infrared energy in sunlight. The responder apologized and corrected himself later in this thread for his error. ow if only Willis had the means, motive, and self-esteem to correct himself we could make some progress and people could actually learn some science from one of Willis’ misadventures into statistical analysis which, by all indications, is another anecdotal skill he picked up via osmosis while either surfing, sailing, fishing, trying to get a dishonarble mental discharge from the Army, or whatever his source of knowledge which decidedly does not include going to school or studying any science textbooks.
[Reply: You are invited to submit an article of your own if you wish. You might be surprised at the comments. ~dbs, mod.]

Willis says: “These are gridcell monthly averages, about twenty thousand of them. Which among them are you proposing to connect with dots, and on what basis? Please think this stuff through before tossing out un-finished ideas.”
First of all Willis, don’t get pissy. I am trying to make useful and constructive criticism. I’ve already done this kind of analysis and more and can probably point out some errors and make useful suggestions. If anyone is ” tossing out un-finished ideas” it’s you , as you readily admit in your post.
You specifically asked for people to check over what you’d done so don’t get offended when they do.

Willis says: These are gridcell monthly averages, about twenty thousand of them. Which among them are you proposing to connect with dots, and on what basis? Please think this stuff through before tossing out un-finished ideas.
Why don’t you think it through before dismissing it? Part of your problem is dumping that much incompatible data it one bucket. Read my posts , you may learn something. Don’t let it go to your head that Anthony let you post on his blog.
Please re-read what I have done. ∆T is the change in temperature with time, which is the discrete form of the derivative you are using, dT(t)/dt. No way to put this politely. You’re suggesting that I do what I just did, you’re not paying attention.
Yeah, but you are plotting against dR/dt , so once again you have not even though about what I said before rejecting it.
I don’t see your objection to the cosine weighting . ERBS data is W/m2 so you must relate it to an area related term.
Nick Stokes above already pointed out the proper way to deal with the weighting. Read the thread.
This thread was already >110 posts by the time I got to read it. Most of that is the usual garbage , bitching and inane comments. Life’s too short. I did scan the whole thread for anything you’d posted and picked up on help Nick had provided you with.
Your post said you were still unhappy about the idea of weighting, I was saying it was correct to do it and why. Again I see no reason for your off-hand response.
As a means of reducing the 2.5 gridded data I’d suggest a 1:2:1 binomially weighted average in lat , then the same in longitude. Alternatively interpolate the 5 degree data with a Catmull-romm spline (quite easy actually)
Lots of possible ways to do it, but since Nick solved the problem already, not necessary … plus the fact that there’s no way I’d do a smooth as you suggest, that’s just losing information.
If you want to move from a 2.5 grid to a 5 deg grid, how do you imagine you can do that without “just losing information”? The fact that you later throw away the whole time dependency of BOTH datasets makes you comment about losing information rather comical.
Don’t get your back and get narky, just read the posts, you may learn something. If you are unable to learn anything I’m sorry, I tried let you benefit from some of the things I’d learnt in doing a similar analysis, I don’t intend to spend my days arguing about it.
best.

To spell this out: Dessler (and others) do a similar scatter plot method and get an incorrect slope for similar reasons. It is a mathematical result that the ols estimator WILL be less than the true “slope”. How much less depends on the amount of noise. The data has more noise than signal. The regression attenuation will be very significant.
Since the ols value is less than the neutral 3.3 W/m2/K of the plank response he comes to the erroneous conclusion that he has demonstrated positive feedback.

Interesting and informative post George. However you seem to have misunderstood what ‘positive’ is all about.
“Water can only lower the solar energy that reaches the earth surface; it can never increase it.”
I’m not aware of anyone saying that is not the case. That part is negative. However, when you look at the similar effect of outgoing IR a similar argument would lead to saying this can only be positive. The big question is the overall, net effect on ERB.
Unfortunately, for the reasons I’ve stated above, this cannot be resolved by doing invalid OLS regression on scatter plots.
That Willis makes this error in a blog post is excusable. That the likes of Dr. Dessler do this in the published literature is rather less admirable.

Willis wrote: These are gridcell monthly averages, about twenty thousand of them. Which among them are you proposing to connect with dots, and on what basis? Please think this stuff through before tossing out un-finished ideas.
OK, so you don’t have time. Let it go at that instead of the pseudo-intellectual dismissal.

“”””” P. Solar says:
October 9, 2011 at 10:54 pm
Interesting and informative post George. However you seem to have misunderstood what ‘positive’ is all about.
“Water can only lower the solar energy that reaches the earth surface; it can never increase it.”
I’m not aware of anyone saying that is not the case. That part is negative. However, when you look at the similar effect of outgoing IR a similar argument would lead to saying this can only be positive. The big question is the overall, net effect on ERB. “””””
“”””” I’m not aware of anyone saying that is not the case. “””””
Well P, here at WUWT, we are simply inundated with people constantly posting that that IS the case. I can count on the fingers of one hand, the number of times anyone ELSE has ever posted that.
Essentially 100% of the input energy driving the earth climate system comes from one source; the sun. And over 75% of that input solar energy that reaches the surface, ends up getting stored in the deep oceans; as deep as 700 metres. That heat store ultimately drives the weather and the climate. To the extent that the captured solar energy decreases (due to more water in the atmosphere), the earth must cool down.
It may be months for variations in the stored ocean energy to show up in the climate. On the other hand, variations in the escape of LWIR from earth, as a result of GHGs or clouds; or anything else, takes place in time scales many orders of magnitude faster than the transport of stored ocean energy. Those effects cannot stop the escape of that energy; only delay it. The loss of that input solar energy is however permanent; it is NOT recoverable.
The point is that very short term changes in the atmosphere condition as it relates to GHGs or clouds, is completely swamped by the permanent change in the energy captured by the earth from the sun, as a result of those transient events.
The LWIR emission from clouds, is isotropic, and it is trivially obvious that at least for a small isothermal cloud, the result is that 50% of that emission is lost to space, and only 50% of it returns to earth., and most of that (3/4) simply results in prompt evaporation from the ocean surface, which transports huge amounts of latent “heat” of evaporation back into the atmosphere where convection processes quickly move iot to the upper atmospher for loss to space. Yes for larger clouds the tops could be cooler than the bottoms; but even that is not assured, since the cloud tops are also directly heated by the sun. Unless the escape delay of LWIR surface emissions RAISES the amount of incoming solar energy that the earth captures, that CANNOT be considered to be a positive feedback. The input driving signal to this sytem is ONLY the sun.

Willis,
there will be some climate-related presentations at the Joint Statistical Meetings in San Diego, July 28 – Aug2, 2010. I am working to extend your analyses of the TAO data. There will be about 5,000 statisticians present. TAO/TRITON is in the tltle. The program will be posted online next May or thereabouts. Meet me and we can discuss our work.
Maybe by then you’ll have figured out how to connect the dots.

[Reply: You are invited to submit an article of your own if you wish. You might be surprised at the comments. ~dbs, mod.]
I refuse to join any club that would have me for a member. :-p

Willis: Now you say I can’t do it with linear regression.
Sorry, the claim that you couldn’t estimate the slope with linear regression was stated first. I carried it forward implicitly, without restating it.
WHY is this particular dataset not viable for linear regression, and what mathematical methods did you use to establish that fact?
The X and Y variables are both random variables. Your computation of the statistical significance assumes that they are Normally distributed random variables. That gives you a bivariate normal time series. Regression methods assume that the X variables are fixed; or, that you are conditioning on the observed values of the X variables. Either way, the estimate of the regression coefficient of Y on X is biased. I missed the part where you derived the correctness of the formula for computing the observed significance level. Paraphrasing you, you did not put in the mathematics to establish that fact. How to estimate the slope of the regression of a bivariate normal time series is taken up in, for example, in “New Introduction to Multiple Time Series Analysis”, by Helmut Lutkepohl, Springer, 2006, corrected second printing 2007.
PS – I think you must mean “2012″ above. good catch, thanks

There’s an inconsistency between the order of the longitudes in the ERBE file and the HadCRUT file. The longitudes in the ERBE file go from 1.25 degrees to 358.5 degrees. The longitudes in the HadCRUT run from -177.5 degrees to 177.5 degrees. Your program incorrectly assumes that the order for the two files is the same. The simplest fix is the interchange the first 36 longitudes in the HadCRUT file with the last 36. By my calculation doing so increases the slope for the unweighted calculation from -1.9 to -1.4, and for the weighted calculation from -1.7 to -1.2.

Shrug, this is to you, not trying to “sneak” anything in… I suppose it’s time I accepted that most climate ‘scientists’, I use the word loosely because the general interest of this attracts scientists and others from different disciplines, see a completely different world to that which traditional physics teaches. How then do you account for the phenomenon of ‘the sun burns up the mist and clouds’ as we wait for it do so? Rhetorical. I’ve given up taking these alternate universe view seriously.

“Data colors indicate the location of the gridcell, with the Northern hemisphere starting with red at the far north, slowly changing to yellow and to red at the equator. From there, purple is southern tropic, through pink to green for the farthest south latitudes.”
Red in the north AND red at the equator? And what about blue? I guess it is blue at the equator, and not red?
Besides this: Really cool work! 😉
[REPLY: Thanks, fixed. -w.]

I’ve had a chance to review the data and I’ve come to the realization that your analysis is missing a major confounding factor, namely the incoming solar radiation. The quantity being analyzed is the net cloud forcing, namely the total outgoing radiation attributable to clouds. This does not measure the efficiency of clouds in dealing with a fixed level of incoming radiation, the relevant quantity if we’re interested in feedback from CO2 related changes. Stated more simply, the more sunlight coming in, the more sunlight is reflected off of clouds. Furthermore, increasing sunlight causes the air to heat up giving rise to the potential for a spurious correlation.
With this in mind I redid the regression analysis adding incoming solar radiation as one of the explanatory variables. In the following printout, dncf is the change in the net cloud forcing, dtem is the change in the temperature and dtsi is the change in the solar forcing. The tsi is determined from the ERBE file as the sum of the outgoing long wave radiation, the outgoing shortwave radiation and the net radiation. After a review of the scatter plot of dncf vs dtsi, I concluded that there is an obvious non-linear relationship between the two variables, which is reasonably approximated by a cubic spline of three degrees of freedom. So with that introduction, here’s the output of the analysis.
[sourcecode]
Call:
lm(formula = dncf ~ ns(dtsi, 3) + dtem, data = df2, weights = df2$area)
Residuals:
Min 1Q Median 3Q Max
-75.538 -4.112 0.089 4.138 53.575
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 42.46878 0.45574 93.19 <2e-16 ***
ns(dtsi, 3)1 -23.68547 0.31780 -74.53 <2e-16 ***
ns(dtsi, 3)2 -104.67009 1.03840 -100.80 <2e-16 ***
ns(dtsi, 3)3 -61.00918 0.46796 -130.37 <2e-16 ***
dtem 1.38452 0.03608 38.37 <2e-16 ***
—
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 7.575 on 18439 degrees of freedom
(4020 observations deleted due to missingness)
Multiple R-squared: 0.554, Adjusted R-squared: 0.5539
F-statistic: 5726 on 4 and 18439 DF, p-value: < 2.2e-16
[/sourcecode]
The R squared is now 55%, with about 51% attributable to the incoming radiation. The sensitivity to the temperature has now flipped signs. Given this model the uncertainty in the coefficients is nil, but that does not preclude the possibility of other confounding factors. Furthermore, as Roy Spencer so frequently notes, it’s not clear whether changes in clouds drive changes in temperature or vice versa.