By Bob Irvine
Conclusions based on large complex datasets can sometimes be simplified and checked by comparing expected results with the actual numbers generated. This is the case with the 58 temperature stations that cover the period 1910 to 2019 in the Australian BOM record.
Andy May was right to be suspicious when the USA warming as recorded of 0.25C per century
is changed using corrections and a gridding algorithm to 1.5C per century. It is, however, nearly impossible to prove non-climatic warming due to data manipulation by normal means. The fact that an algorithm increases warming may be legitimate, “time of observation” (TOB) for example.
A similar thing has happened in Australia.
“ACORN is the Australian Climate Observation Reference Network of 112 weather stations across Australia. The rewritten ACORN 2 dataset of 2018 updated the ACORN 1 dataset released in 2011. It immediately increased Australia’s per decade rate of mean temperature warming by 23%…” (Chris Gilham)
Again, it is right to be suspicious but difficult to prove by simply looking at the overall result. The changes could be legitimate, despite there being virtually no TOB issue in Australia.
EVIDENCE THAT NON-CLIMATIC WARMING HAS BEEN ADDED BY THE ACORNSAT CORRECTIONS
The term “corrections” here refers to all changes to the raw data for any reason. Some of these changes will be legitimate and some will distort the data in an artificial way. What follows is evidence that the artificial distortion of the data is significant and the “correction algorithms” should be revisited.
SOME BACKGROUND
I have used the following dataset for all calculations and acknowledge Chris Gilham’s effort in compiling it.
http://www.waclimate.net/acorn2/index.html
My term “magnitude of the corrections” for each station’s max, min and mean is derived as follows. From the Albany maximum data at the above link.
Average change per decade: ACORN 2.1 0.16C / raw 0.08C
The “magnitude of the correction” is the difference between “Acorn Sat 2.1” and “raw” for each station (max, min, mean). i.e. 0.08 (0.16 – 0.08) in this case.
The “Acorn 2.1 anomaly” in this case is 0.16C.
The Acorn 2.1 corrections are designed to correct the data for station moves, equipment changes etc. and the final result is expected to represent the real temperature anomalies at a particular station more realistically than the raw data.
While it is expected that the corrections will either warm or cool the data at any particular station, it can also be expected that the “magnitude of the correction” bear no relationship to the final temperature anomaly. The final temperature anomaly as represented by Acorn 2.1 should depend entirely on the real temperature anomalies at the particular station and, therefore, be completely unrelated to the “magnitude of the corrections”.
For example, if two hypothetical stations were a few hundred meters apart but showed a large difference in temperature anomaly due to differing vegetation or equipment etc., then Acorn 2.1 would hypothetically adjust one say by 1.0C per decade and the other by say 0.03C per decade to finish with the same temperature anomaly for each station as is expected. They are next to each other after all.
The “magnitude of the corrections”, in this case 1.0C/Dec. and 0.03C/Dec., should not correlate with the final temperature as recorded by Acorn 2.1. i.e., the final temperature anomalies in this hypothetical case will finish up being approximately the same while the “magnitude of the two corrections” are very different. I hope this is clear.
If we then take the 58 Australian stations that cover the period 1910 to 2019, we should find that the “magnitude of their corrections” and their final “Acorn 2.1 anomaly” have approximately zero correlation if the corrections are not artificially affecting the final Acorn 2.1.
To check this, I have calculated the Pearson Correlation Coefficient (PCC) for these two data sets. (“Magnitude of the 58 corrections” compared to the “Acorn 2.1 anomalies” for the 58 stations).
If this PCC is;
- Negative – then the corrections to the raw data are adding artificial or spurious cooling to the raw data.
- Approximately zero – then the corrections are not adding artificial cooling or warming and are doing their job as intended.
- Positive – then the corrections are adding artificial or spurious warming to the raw data.
JUSTIFICATION FOR THE METHOD USED
The BOM comparison station selection method has a number of reference points and two variables per reference point that they are trying to correlate.
My data is exactly the same and I use the same method as they do to correlate my variables.
The BOM’s reference points are the 12 months.
My reference points are the 58 temperature stations.
The BOM’s two variables are the Temp. anomalies at station 1 and station 2 for each particular month.
My two variables are the “Magnitude of the Correction” and the “Acorn-Sat homogenised Temp. anomaly” at each particular station”.
A Pearson Correlation Coefficient (PCC) is specifically designed for this application.
The BOM statisticians obviously thought this the best way to approach this issue, as I do.
See also Appendix A for a stress test of the method used.
LEGEND
A PCC only indicates correlation and is not designed to indicate certainty or put a number on probability etc. The significance of a PCC value depends entirely on the intrinsic nature of the datasets being compared.
In their Pairwise Homogenisation Algorithm, AcornSat extensively use nearby comparison stations to deal with incontinuities in any given station. For a comparison station to be considered useable they compare the monthly temperatures for a 5-year period either side of the incontinuity. If the PCC for these two monthly datasets is greater than 0.5 then that comparison station is considered accurate enough to one tenth of a degree Celsius.
I have used the AcornSat logic and PCC figure as a guide when comparing the “magnitude of the corrections” with the “Acorn 2.1 anomaly” for the 58 stations.
My legend;
- If the PCC is greater than 0.5 then – It is almost certain that the corrections are adding significant non-climatic warming to the raw data.
- If the PCC is greater than 0.3 and less than 0.5 – then it is very-likely that the corrections are adding significant non-climatic warming to the raw data.
- If the PCC is greater than 0.1 and less than 0.3 – then it is likely that the corrections are adding some non-climatic warming to the raw data.
- If the PCC is between -0.1 and 0.1 then it is likely that the corrections are not adding non-climatic cooling or warming to the raw data. The corrections are doing their job.
- If the PCC is less than -0.1 – then it is likely that the corrections are adding some non-climatic cooling to the raw data.
- Etc.
RESULTS
Figure 1 graphs the 58 Australian stations that cover the period 1910 to 2019. The general slope of the data points up to the right of the graph is solid evidence that the Acorn 2.1 corrections are adding non-climatic warming to the record.
Table 1 below shows clearly how the final Acorn 2.1 warming anomaly increases with the magnitude of the corrections. The implication is that the corrections are causing some of that apparent temperature increase.
Table 2, below indicates the Pearson Correlation Coefficient (PCC) (Magnitude of Corrections v Acorn temp. anomaly) for the total dataset. The red line has a lower PCC and implies that when less correction is required the final Acorn 2.1 temperatures have less non-climatic warming.
CONCLUSION
According to the precedent set by AcornSat for the use of the Pearson correlation coefficient (PCC) when applied to data of this type, a PCC of 0.56 (magnitude of corrections verses Acorn 2.1 temp. anomaly) as found here indicates that “It is almost certain that the corrections are adding significant non-climatic warming to the raw data”.
For some of the stations in the dataset, AcornSat decided that the raw data was relatively accurate and consequently did not need significant homogenisation or correction. These stations are likely to give a more accurate temperature reading and a better idea of Australia’s temperature history over the period covered.
As a check I calculated the same PCC for the 14 stations that only needed minimal correction (Red data in Table 1 and Table 2). A lower PCC of 0.25 still indicates some artificial warming but not nearly as much as for the whole dataset.
The average temperature anomaly per decade for the 14 stations that needed minimal correction is about 0.1°C/Decade (Table 1). As calculated, this will include some artificial warming so the actual average warming for these more accurate stations is likely to fall in the range, 0.08°C to 0.10°C per decade.
This method could possibly be used to check the adjustments to the USA datasets. These adjustments are fiercely contested so it would be good to have some objective analysis. Let the cards fall where they will.
APPENDIX “A”
This appendix seeks to stress test the method used above. It does this by applying the method to two extreme homogenisation algorithms. One that we know adds zero artificial warming and one that we know adds large artificial warming in proportion to the correction. In the first instance (“C” in table 1.) we expect the Pearson Correlation Coefficient (PCC) to be zero. In the second (“E” in Table 1.) we expect the PCC to be 1.0. If this is true in both cases, then the method used, and the conclusions drawn in the main paper above are likely to be correct.
THE TEST
We use accurate satellite technology to determine the temperature anomalies for all parts of an imaginary small island.
This imaginary accurate satellite data has been available for the last 100 years and tells us that the temperature of all parts of this island have increased at the same rate. This rate being 0.1C per decade over the last century (“B” in Table 1).
We the readers are the only ones aware of the true and accurate temperature rise for all parts of the island over the last 100 years as described.
The inhabitants of the island are unaware of this accurate satellite data set. In an attempt to determine the temperature rise, over the 100 years on the island, these inhabitants set up 10 temperature stations 100 years ago, all equally spaced around the island.
Three of these stations (Station # 1 to 3) were well sited with good equipment that has not needed to be changed or relocated for any reason for the 100 years. They each showed a correct temperature anomaly of 0.1C per decade.
The other 7 stations (Station # 3 to 10) were subject to various changes related to equipment, vegetation, station moves etc. The raw data from all these 7 stations showed anomalies less than the correct 0.1C per decade. To correct for these inconsistencies the inhabitants developed two homogenisation algorithms (“C” and “E”) that they then applied to these 7 variant stations.
To test which of these two algorithms was suitable, the inhabitants applied the test described in the main paper above.
| STATION # | A | B | C | D | E | F |
| 1 | 0.1 | 0.1 | 0.1 | 0 | 0.1 | 0 |
| 2 | 0.1 | 0.1 | 0.1 | 0 | 0.1 | 0 |
| 3 | 0.1 | 0.1 | 0.1 | 0 | 0.1 | 0 |
| 4 | 0.09 | 0.1 | 0.1 | 0.01 | 0.11 | 0.02 |
| 5 | 0.08 | 0.1 | 0.1 | 0.02 | 0.12 | 0.04 |
| 6 | 0.07 | 0.1 | 0.1 | 0.03 | 0.13 | 0.06 |
| 7 | 0.06 | 0.1 | 0.1 | 0.04 | 0.14 | 0.08 |
| 8 | 0.05 | 0.1 | 0.1 | 0.05 | 0.15 | 0.1 |
| 9 | 0.04 | 0.1 | 0.1 | 0.06 | 0.16 | 0.12 |
| 10 | 0.03 | 0.1 | 0.1 | 0.07 | 0.17 | 0.14 |
| Mean | 0.072 | 0.1 | 0.1 | 0.028 | 0.128 | 0.056 |
Legend;
A – Raw Temperature data for each station. (°C per decade).
B – Correct Satellite Data. (°C per decade).
C – Final temperature data after first accurate homogenisation algorithm is applied. It is correct and matches the satellite data. (°C per decade).
D – “Magnitude of the Corrections” as applied to “C”. Derived by subtracting Raw data from “C”. (°C per decade).
E – Final temperature data after using a homogenisation algorithm that adds artificial warming to the record. (°C per decade).
F – “Magnitude of the Corrections” as applied to “E”. Derive by subtracting raw data from “E”. (°C per decade).
Table 1. Compares the 10 stations with their raw temperature, correct satellite temperature, and two possible final temperatures after the two homogenisation processes are complete. The two “magnitude of the corrections” (“D” and “F”) are also shown and are derived by subtracting the raw temp. from both final homogenised temp. sets (“C” and “E”).
RESULT
The Pearson Correlation Coefficient (PCC) between “C” and “D” is zero as expected.
The Pearson Correlation Coefficient (PCC) between “E” and “F” is 1.0, also as expected.
CONCLUSION
On the imaginary island in this appendix, a homogenisation algorithm that was known to be accurate (“C”) gave a PCC when the “magnitude of its corrections” was compared to its final temperatures, of zero.
A homogenisation algorithm that was known to add artificial warming to the record in proportion to the corrections (“E”) had a similarly calculated PCC of 1.0.
Both these results are consistent and would be expected if the method and results of the main paper above were accurate.
The case for saying that the Australian BOM homogenisation algorithm adds artificial warming to the record is strong and possibly proven.
Haven’t we known this for like 20 years? Same with US data?
More evidence is always better.
the claims have existed for 2 decades. Most of us who pay attention and challenge the narrative ever doubted the intent of the warming adjustments. This paper should go mainstream but probably won’t. Really a sham and a shame. As long as NOAA gets away with falsifying data, it will get warmer on the record while we live in a cooler world.
NOAA has nothing to do with ACORN. And if you look at the raw data you’ll see that the global warming trend is actually higher relative to the adjusted data [1].
What you will in fact see is that the highly inconvenient period of 1880-1945, during which time CO2 increases were trivial, and in particular the very sharp increase understood to have occurred naturally from around 1910-1945, have all been erased.
The fudging away of the pre-anthropogenic warming period has the opposite effect of what you stated above (as per the document you linked). It reduces, rather than increases, the total global warming trend over the entire period. HOWEVER, the adjustments make it look as if the supposedly anthropogenic portion of the warming (1950- present) is markedly different than the pre-anthropogenic portion of the data, when in fact it was not. Lots of contemporaneous written history, together with measurements recorded during the earlier period, make the warming of that period clear.
Erasing or rewriting historical data seems to be an important foundation of climate science. Unfortunately, it is not an important foundation of actual science. In general, if you have to cook up excuses for massively changing data in order to have it fit your predetermined conclusion, then this is a pretty clear sign that your predetermined conclusion is wrong.
There is nothing inconvenient about the 1880-1945 period. As you can see the warming that occurred from 1910-1945 is very real and has not been erased.
The graph I posted above does not in any way provide information about why the global temperature was changing. It is only showing the change. The unadjusted line (blue) shows what the global temperature + biases were doing. The adjusted line (red) shows only what the global temperature was doing sans the biases. That’s all the graph is telling you.
Nobody is erasing or rewriting historical data. You can download the ACORN data here and the GHCN data here in its as-reported form. And the opposite of erasing is occurring. These repositories are growing in size even for years long past as new observations and old observations are being digitized and uploaded.
Firstly, what I and everyone else can see is that the warming from 1910-1945 has been greatly reduced by the adjustments made in your graph to warm the past. My eyes can plainly see, in fact, that the amount of warming has been reduced by approximately 0.3C.
Secondly, neither ACORN data nor GHCN data have anything whatsoever to do with ocean temperatures, and the warming adjustments in question are almost entirely due to adjustments made to sea water readings. This is explained in the article by Zeke Hausfather that you yourself referenced above, but that you do not seem actually to have read.
Thirdly, the revisions in question do not give you some kind of pristine truth about temperatures “sans the biases.” What they do, in fact, is to very heavily introduce the biases of Hausfather and Karl, who cooked up a facile reason for adjusting data that is inconvenient for their pet hypothesis. The nature of these adjustments has already been discussed at great length on this website, and I don’t intend to rehash the whole thing. However, if you are genuinely interested in the question of how ocean temperatures were recorded and calculated, and how easy it may or may not be to eliminate biases, you will find a great deal of information and discussion on this topic simply by searching “Karl et al” in this site’s search bar.
Can you post a link to the dataset that is free biases that you feel can be used to answer the question of which line in the graph (blue or red) above is more correct?
There is no dataset that is free of biases, because the underlying data is garbage. Seventy percent of the planet’s surface is water. Of that 70%, a very small fraction was traversed by vessels in commercial shipping lanes. A number of those commercial ships were tasked with dropping buckets over the side, hauling up water, sticking a thermometer into the bucket, and recording temperatures.
This task was presumably entrusted to the lowliest, most junior sailor on board. Did that person drop his bucket at exactly the same time every day? Who knows. Were the buckets dropped in consistent places, allowing for reasonable comparison? Almost certainly not. Did the commercial sea captains give a rat’s ass whether the task was actually done at all, or if the sailor responsible simply sat down for 5 minutes before the voyage and made up all the values for the following month? Again, no one knows.
As for the measurements, even assuming they were done as per SOP, which itself a serious stretch, how deeply did the bucket drop before it was hauled onto deck? 1 foot? 2 feet? 5 feet? The temperature of the water changes significantly depending on depth. How long was it before the thermometer was placed inside the bucket? How long was the thermometer allowed to stay inside the bucket before it was read? How accurate were the eyeball readings the sailor made?
And it gets even better. Was the bucket made of wood or of canvas? Was it an insulated bucket? All of these things, and many more, change the readings too. In most cases, there is no good meta data explaining how any of this was done, so no one actually has any idea how they should properly adjust the recorded temperatures, even if they wanted to. There is no known right answer, so you can happily come to any answer you choose.
There are also, incidentally, lots of issues with variance from readings taken in ship intake engines and also readings taken from buoys. It’s all garbage and realistically the data should have error bars as wide as the ocean itself.
Since the ocean data is garbage (the land data prior to satellite era is also garbage for numerous reasons), anyone can make any adjustment he or she likes. There is always a facile reason one can cook up to adjust temperatures either up or down. If you are motivated to come to a particular conclusion, then you can cook up reasons to get there. When you are hoping to find a signal in a dog’s breakfast of noise, then the data turns into an ouija board.
That being said, there was general agreement among people who attempt to reconstruct these temperatures that the best guess was something similar to the blue line above. Karl and Hausfather’s adjustments were an outlier. The facts that Hausfather, at the very least, is heavily and publicly invested in global warming dogma, that he and Karl made and published those adjusted numbers without going through standard peer review, and that they conveniently cranked that garbage out immediately before Obama was set to go to a global warming meeting, thereby providing him with a nice, convenient narrative to take with him, all suggest that Karl and Hausfather introduced bias into the temperature series rather than eliminated it.
Incidentally, the fact that Hausfather’s article shamelessly dissembles by proudly claiming that the adjustments reduce the amount of total warming, when the actual issue at hand is the convenient periods in which that warming was magically reduced, should also tell you something about the honesty of ideologues like Hausfather, as well as how foolish and credulous they apparently believe their audience to be (a not unjustified assumption).
Finally, there are numerous detailed criticisms of Karl et al’s data, if you are interested. Many of them can be found easily on this site. I’m sure you can find more criticisms on other sites as well. I think there may have been a few back and forths on Curry’s website several years ago. It is a complex issue, and you can educate yourself on it if you are truly interested.
Proponents of global warming dogma are constantly trying to rewrite history to make it less inconvenient to their narrative (goodbye Medieval Warm Period, Roman Warm Period, Little Ice Age, pre-1950 sharp warming, etc.). These re-writings of history are always based on spurious and facile analyses. In an age in which people were not hoping to find exactly those convenient results, the failings of the the analyses would be easily apparent. In our current dogmatic, tribal, and neo-religious age, however, any result that accords with the tribe’s dogma seems to be accepted immediately and uncritically, with all accepted previous truths instantly fading away.
If you think the existing datasets are garbage and dismiss the idea that any unbiased dataset exists then how do you know the global average temperature increased from 1910-1945 that isn’t already depicted in the graph above?
Surface warming and obvious climatic changes were well documented during that period by the people that lived through them. There was very obviously a significant amount of warming, freakish weather, changes to the Arctic, etc., even if the specific amount cannot be known to the hundredth of a degree, which is the false impression one would get from looking at the composite temperature reconstructions.
What is easiest to reject are the transparent attempts by climate activists to stare at the most inconvenient sections of the temperature reconstructions and cook up facile excuses for changing them- without exception in a way that appears to make post 1950 warming look more pronounced than prior periods, or to make post 1950 cooling look less significant than global warming theory would like it to be. This has been going on for quite a while. The surface temperature highs of the 1930s somehow keep getting reduced by “reanalysis,” the amount of cooling from ~1945-1975 keeps being reduced in the same manner, and in the case we have discussed, the total warming from 1910-1945 is reduced by cooking up a reason to pretend that the oceans were significantly warmer than measured, and then using that assumption to reduce the total estimated warming for the period.
Climatic changes in 1910-1945, including Arctic ice loss, global heat waves, freakish weather, massive droughts, etc. were well documented and discussed widely in newspapers at the time. There is no reasonable doubt that they happened, and there is no reason to think that people today have lived through changes of a greater magnitude. In so far as people want data on global temperature, a concept that doesn’t even make much sense, then it is best to try to get it from people who are not obviously trying to get the data to comport with their dogma. The data products that obtained before Karl et al were highly imperfect, but they were almost certainly closer to the truth than the garbage Karl cranked out that adjusted away all of the most inconvenient periods of weather history, including pre-WWII warming, subsequent multidecadal cooling, and the temperature pause that began in the late 1990s.
Is there a dataset the compiled this documentation from 1910 to 1945 to produce a global average temperature on ether a monthly or annual period that shows the warming you are talking? Would you mind linking to it so that I can review it?
Ask Tom Abbott about the Tmax records. Do you really want us to believe you’ve seen none of his posts on this?
That’s for the US only. The US is only 2% of the area of Earth. It’s also the unadjusted data which is contaminated with biases caused by station moves, time-of-observation changes, instrument changes, etc. How do you take USHCN unadjusted data and form a global average temperature with it that is immune from these biases?
“That’s for the US only.”
So what? Jeesh you are thick! You can do the same thing for the entire globe.
YOU WILL STILL WIND UP WITH A MULTIMODAL DISTRIBUTION! You’ll wind up with warm NH temps mixed with cold SH temps. You’ll wind up with continental US coastal temps mixed with interior temps. The exact same problem you get using mid-range values from around the globe. It’s why a global average is meaningless and that holds true even for anomalies since winter temp ranges are different than summer temp ranges!
My parents and grandparents lived through these times. I remember the stories.
Do they have the global average temperature written down on a monthly or annual basis from 1910-1945? How were they able to measure the global average temperature?
It’s the Tmax and Tmin temps that stand out, not the mid-range values. Why is that so hard to understand. No one cared about the “global average temp” during the Dust Bowl!
Then how do you know 1) there was warming from 1910-1945 that isn’t already accounted for and 2) that there was warming at all on a global scale?
There’s a clue here in this climate gate email from 2009.
“From: Tom Wigley <wigley@ucar.edu>
To: Phil Jones <p.jones@uea.ac.uk>
Date: Sun, 27 Sep 2009 23:25:38 -0600
<x-flowed>
Phil,
Here are some speculations on correcting SSTs to partly explain the 1940s warming blip.
If you look at the attached plot you will see that the land also shows the 1940s blip (as I’m sure you know).
So if we could reduce the ocean blip by, say, 0.15 degC, then this would be significant for the global mean — but we’d still have to explain the land blip.
I’ve chosen 0.15 here deliberately. This still leaves an ocean blip, and I think one needs to have some form of ocean blip to explain the land blip (via either some common forcing, or ocean forcing land, or vice versa, or all of these). When you look at other blips, the land blips are 1.5 to 2 times (roughly) the ocean blips — higher sensitivity plus thermal inertia effects. My 0.15 adjustment leaves things consistent with this, so you can see where I’m coming from.
Removing ENSO does not effect this.
It would be good to remove at least part of the 1940s blip, but we are still be left with “why the blip”.”
di2.nu/foia/1254108338.txt
Accurate to a tenth of a degree of course.
Let me clarify my questions. What dataset that is free from any biases caused by station moves, time-of-observation changes, instrument changes, bucket-engine-buoy differences, etc. and which you feel truly represents the global average temperature shows that there was warming from 1910-1945 that isn’t already depicted in the graph above?
“Then how do you know 1) there was warming from 1910-1945 “
Because “warming” is usually associated with the mid-range value in the AGW alarmists assumptions and conclusions. Mid-range temps don’t identify whether the warming is due to increasing Tmax or increasing Tmin. You’ve been given the problems with averaging temps many, many times before.
There are Tmax graphs for almost all regions on the earth, if I can find my copies I’ll post some other samples showing that the warming of Tmax from 1920-1945 was global.
BTW, is there warming on a global scale today? How do you know that from averaged mid-range values made up from multi-modal data sets?
They can’t measure the global average surface temperature. That’s the whole point. The only way that humans have come even remotely close to doing that is by using satellites.
The data that exist for the period in question come from a tiny fraction of the earth’s surface, and you can always come up with a reason to contest that data for one reason or another. As you can see from Bob Irvine’s post below, however, the people who are in charge of compiling supposed “global temperature” seem to be starting with a pre-formed conclusion of what they would like the right answer to be and then working backwards to figure out what adjustments can get them there while still sounding superficially plausible. This accounts for why the adjustments made to global temperature products unfailingly seem to make recent global warming seem worse. No one has seemingly ever discovered an “error” that makes pre-anthropogenic warming seem worse, or that makes supposedly anthropogenic warming seem more trivial.
However, beyond thermometer readings and adjustments, it seems clear that people in the first 4 decades of the 20th century experienced warming, extreme weather, and climatic changes all over the world. People noted and wrote about these changes. Just to take a few simple examples, here are instances of global climate changes from just the 1920s that would be taken in today’s primitive age as clear signs of climate crisis / environmental doom / Gaia’s wrath, but that our ancestors, in their ignorance, simply lived through and viewed as remarkable and unfortunate weather patterns.
https://timesmachine.nytimes.com/timesmachine/1923/02/25/105849503.html?pageNumber=62
https://www.loc.gov/resource/sn83045774/1921-09-04/ed-1/?sp=61&q=death+for+millions&r=-1.011,-0.106,3.022,1.485,0
https://www.nytimes.com/1921/10/03/archives/freak-weatherlaid-to-earthly-mumps-old-spheroid-convalescing-say.html?searchResultPosition=1
https://www.nytimes.com/1929/08/25/archives/strange-weather-has-swept-over-the-world-the-terror-of-the-skies.html?searchResultPosition=1
If its not possible for you to measure the global average surface temperature then how do you know the global average surface temperature increased more from 1910-1945 than is shown in the graph above?
What you know is that the thermometer readings went up sharply in all the places that had thermometers, and that significant warming, including Arctic melting, massive heatwaves, freakish weather, and droughts on every continent were observed and recorded by humans in just about all the places they lived and explored. That’s what you know. You also know, should you bother to read, that the observed climatic effects in that period were as great or greater than anything you have witnessed or experienced in your lifetime. That seems to be all you really need to know.
What do you believe you actually know when you see any line purporting to be a reconstruction of “global temperature” for that period? Do you believe that a handful of thermometers, spaced hundreds or many thousands of miles apart, almost completely non-existent on the continents of Asia, Africa, South America, the Arctic, and Antarctica, the data for all of which has been adjusted and readjusted subjectively, which is then mushed up into a black box that outputs one line of numbers is actually telling you something useful about the correct average temperature of the entire globe? Does the concept of average temperature even make any sense at all? If the surface of the Pacific ocean hypothetically warmed by 5C while at the exact same time the surface of the Atlantic cooled by 10C, such that the net global warming averaged out to zero, what significance do you think that data point would hold?
Moreover, if the values of the “global warming” graph change, and then change again, and again, and again, do all of those changes make you more confident or less confident that you are now in possession of unadulterated truth?
Why is it that the Climate Faithful seem to have no problem believing that they can estimate the entire average temperature of the Northern Hemisphere from a handful of tree rings harvested from a 1000sqm copse, and that this data is sufficient reason to believe that e.g. the Little Ice Age never actually happened?
Sloppy data is sloppy data, and if that’s all you have, then you don’t have very much. What we do have, however, are weather and temperature phenomena witnessed by billions of people around the world, that put together with raw temperatures recorded by at the same time, paint a fairly clear picture of rapid global warming that was at least of the same magnitude, and quite possibly of a larger magnitude than any we have witnessed in our lifetimes. If you choose to believe that Zeke Hausfather magically cooked up the exact adjustment that turns crappy bucket data and weather station data into an exact temperature for planet earth, and that this data overrides everything that the people of planet earth seemed to know about their own environment at the time, then you are free to do so. But to anyone who is not besotted by ideology or dogma, it is an obvious steaming pile of BS.
All I’m asking about here is the global average temperature. I’m not asking about Arctic ice. I’m not asking about tree rings. I’m not asking about heat waves, droughts, freakish weather, or anything other the global average temperature. If you can’t present evidence supporting the hypothesis that there was more global warming from 1910-1945 than is already depicted in the graph above then I don’t have any choice but to dismiss it.
I’ll try to make a good faith attempt at answering your questions.
Reacher51 asked: “What do you believe you actually know when you see any line purporting to be a reconstruction of “global temperature” for that period?”
The global average temperature.
Reacher51 asked: “Do you believe that a handful of thermometers, spaced hundreds or many thousands of miles apart, almost completely non-existent on the continents of Asia, Africa, South America, the Arctic, and Antarctica, the data for all of which has been adjusted and readjusted subjectively, which is then mushed up into a black box that outputs one line of numbers is actually telling you something useful about the correct average temperature of the entire globe?”
Yes. But I take issue with the wording of the question since there are several misleading or inaccurate statements embedded in it. Understand that I’m answering yes to the question of whether it can be done. I’m not agreeing to the misleading or inaccurate statements contained within.
Reacher51 asked: “Does the concept of average temperature even make any sense at all?”
Yes. The concept of an average is a useful tool in all kinds of analysis. That’s why we all learn about it in middle school.
Reacher51 asked: “If the surface of the Pacific ocean hypothetically warmed by 5C while at the exact same time the surface of the Atlantic cooled by 10C, such that the net global warming averaged out to zero, what significance do you think that data point would hold?”
I would conclude that there was a process that moved heat from the Atlantic region to the Pacific region. You can model your hypothetical scenario mathematically as 0 = ΔP + ΔA + ΔE where P is the Pacific heat, A is the Atlantic heat, and E is the external heat. If the average netted to zero then we know that ΔE = 0 which means the negative ΔA and positive ΔP indicates an internal transfer of heat.
Reacher51 asked: “Moreover, if the values of the “global warming” graph change, and then change again, and again, and again, do all of those changes make you more confident or less confident that you are now in possession of unadulterated truth?”
No. However it does make me more confident that I’m in possession of something closer to the truth. Remember, digitizing and uploading historic observations into the repository, addressing biases, fixing bugs/errors in the analysis, etc. are good things. We want that. If you encounter a dataset that is not improving then you should be more skeptical of it than datasets that are improving.
I’ve already given you a graph for the US showing Tmax during 1910-1945. How about one from china?
How many more can you just dismiss? I’ll give you one from India in my next reply. And one from Norway in my third reply.
What a crock! This is the argumentative fallacy of Argument by Dismissal taken to an extreme. You don’t identify a SINGLE MISLEADING OR INACCURATE statement, not one! You just dismiss the assertions out of hand.
Why would you conclude that? You want evidence that 191-1945 was warmer than today and then put forth an assertion while providing absolutely no evidence at all! The Drake Passage supposedly blocks thermocline flow between the Atlantic and the Pacific. Where else do you suppose heat transfer could occur? As usual you just totally ignore physical reality in your 0 = ΔP + ΔA + ΔE. There is no ΔP + ΔA.
Your bias is showing. It is just as likely that you are further from the truth because of the subjective changes. You are always so adamant that temperature measurement errors are all random and therefore cancel except, seemingly, in this case. How do you know that widespread changes in ocean bucket temps should be corrected? If the errors in that method are all random, some high and some low, then they should cancel and their average should be pretty accurate. Therefore changing the actual data would be introducing an unjustified bias.
See what I mean? If temp measurements include random errors then addressing “biases” is adding subjective changes to the basic data. Are you now going to admit that uncertainty in temp measurements of different things do *NOT* cancel?
here’s the second graph, India.
The concept of an average may be a useful tool in all kinds of analysis. But it is a useless tool in a far greater number of analyses and it is in this one in particular.
The entire reason that we purportedly need to care about global warming is the supposed effects that it will have on the weather and environment of a multitude of specific places. Heat waves, droughts, and changes to the Arctic that supposedly will imminently turn wherever you live into Atlantis are the only reason that this issue is taken as more than academic blather. If that were not the case, then we would not hear bleating on a daily basis about every storm, flood, drought, heat wave, and cold wave experienced anywhere on earth, each one of which the Faithful and a cast of supposed scientists hold up as clear evidence of their pet hypothesis.
Furthermore, the example I provided above would tell you nothing about heat transfer from one ocean to another. It would simply tell you about changes to the surface temperature of two segments of the earth. It would tell you nothing about temperature changes in in the near entirety of the oceans that lie below the surface, nor in the near entirety of the atmosphere above the surface, and it would not tell you anything about where and how heat is being transferred. If the metric you use to understand the world is global average surface temperature, then it would tell you nothing at all, in fact, since that would show no change whatsoever.
Given the coverage of monitoring stations, to believe that anyone can model the surface temperature of the earth in 1910 with any precision is to believe in many cases that a thermometer in one place can accurately tell you the temperature in another place up to 1200km away. It is also to believe that measurements of ocean temperature- about which people know almost nothing, including location, time, depth, specific instrument, or whether the data was simply made up- can precisely give you the temperature of 70% of the earth’s surface. To believe that Hausfather et al have somehow made those measurements “closer to the truth” is further to believe that Hausfather has uniquely obtained all of that missing information, along with additional information about the data’s highly imperfect comparators, and has magically found precisely the formula for harmonizing the two.
If you can’t present evidence supporting the hypothesis that there was more global warming from 1910-1945 than is already depicted in the graph above then I don’t have any choice but to dismiss it.
Indeed. How could one have any other choice? Upon reflection, it occurs to me similarly that no one has ever really presented me with evidence supporting the hypothesis that Santa Claus does not really exist. Yes, people have pointed out that my parents seem to move around late at night on Christmas eve and that after I leave cookies for Santa, crumbs from the same type of cookies end up on my parents’ bed. In fact, there are numerous criticisms of my Santa hypothesis (none of which I have actually bothered to read), but no one can present actual evidence of his non-existence. Therefore, I don’t have any choice but to reject the non-Santa hypothesis. Santa lives!
Not only that, but it has also occured to me that no one has presented me with any firm supporting evidence to prove the hypothesis that Gal Gadot is not secretly in love with me. Based on my new understanding of Climate Logic and the apparently improved scientific method, I can now conclusively reject that hypothesis too. I have no choice! I can’t wait until Gal confesses her love to me, and she and I can enjoy Santa’s bounty together.
Since I have now seen the light and am ready to be indoctrinated in the Climate Faith, would it be possible for you to teach me your secret handshake? I know that you have one because I can’t provide concret evidence that you don’t. Thanks.
The mathematicians among us, like bdgwx, believes that combining Northern Hemisphere temps (like the heights of Watusi’s) with Southern Hemisphere temps (like the heights of pygmies) and finding a average from that data can tell you if the globe is warming or cooling. In actuality it tells you nothing because both (NH vs SH and Watusi’s vs pygmies) are multi-modal data distributions and the average tells you absolutely nothing about reality. Even worse they try to use anomalies in the NH and SH when the temperature range is usually higher in winter, say in the SH, than in summer, say in the NH. You lose this data when you take anomalies of mid-range temperatures.
When the anomalies are built on long range, mid-range, station-by-station temperature averages, say in just the continental US, will those anomalies tell you that Washington state is generally colder than Florida? Will it even tell you anything about “average” temperature across the US let alone across the globe? Do those anomalies tell you that coastal plains have far different temperatures than in the high plains states? Or that the climate on Pikes Peak is far different than the climate in Denver?
Mid-range averages, especially of global mid-range anomalies, are just easier for the climate scientists to use. They hide their inability to tell us what Tmax and Tmin temps are going to be. They hide their inability to actually forecast climate on a regional basis let alone a local basis. Can they use the global average temp anomaly to tell us what the climate in Kansas is going to be in 20 years? Or in Brazil or Australia or Dubai? How about 100 years from now? Since the actual solution to climate change may be different for each of these locales then isn’t the local climate projection what is really needed? The AGW alarmists just find it easier to say that one-size-fits-all for a solution – kind of like what the US federal government usually comes up with!
I agree, except in so far as I find it strange to imagine bdgwx as a mathematician. If he is a mathematician, then he is apparently one that can only conceive of two dimensions. The conflation of energy found at the surface of the ocean with the total amount of energy stored in the depths of the ocean, for example, is rather strange. Perhaps, given his easy acceptance of Hausfather’s magical single number that supposedly represents the exact average temperature of all the earth’s ocean surface in any given year, his mathematical interest lies solely in the field of imaginary numbers.
To your point, the only things that really matter to people are their local climates and extremes of temperature and weather. These happen to be exactly the things that, from innumerable recorded observations all over the world, fluctuated in just as extreme a manner in the pre-anthropogenic era as in the supposedly post-anthropogenic era. That would seem to be all one really needs to know before they can happily get back on with their life.
Beyond the completely absurdity of believing that a single data point on a graph will give you the exact mean temperature of 139 million square miles of ocean surface, and the even more bonkers assumption that an average change tells you anything useful about what may have happened specifically at any time in any one part of any ocean, it seems unclear why one should even care about it. The few fish that live at the ocean’s surface presumably didn’t notice a theoretical average increase of 0.01C per year for the roughly three years that they were alive. How has our society come to a place where people find this meaningful? It’s like the Russian Old Believers arguing with the New Orthodox about whether the sign of a cross should be made with two fingers or with two fingers and a thumb.
Reacher51 said: “If he is a mathematician,”
I’m not a mathematician.
Reacher51 said: “The conflation of energy found at the surface of the ocean with the total amount of energy stored in the depths of the ocean, for example, is rather strange.”
I’ve made no such conflation. In fact, I want you to review Schuckmann et al. 2020 for a deep dive into the breakdown of the change in energy content of the various reservoirs in the climate system including the different layers of the ocean.
Reacher51 said: “Perhaps, given his easy acceptance of Hausfather’s magical single number that supposedly represents the exact average temperature of all the earth’s ocean surface in any given year, his mathematical interest lies solely in the field of imaginary numbers.”
It’s not Hausfather’s number nor is it magical nor is it exact nor is it only for the ocean’s surface. It comes from NOAA, it is an average temperature on a monthly basis, has a documented uncertainty, and is for the whole Earth including both land and ocean.
Reacher51 said: “To your point, the only things that really matter to people are their local climates and extremes of temperature and weather.”
It is important for sure. But that just that is important to some people doesn’t preclude the global average temperature from being important as well. It can be used to test scientific hypothesis regarding the global average temperature.
Reacher51 said: “Beyond the completely absurdity of believing that a single data point on a graph will give you the exact mean temperature of 139 million square miles of ocean surface”
It would be absurd to think we can measure anything exactly. The average temperature of the atmosphere is no exception. But nobody has ever claimed we know the average temperature exactly. Hausfather openly discusses this.
Reacher51 said: “and the even more bonkers assumption that an average change tells you anything useful about what may have happened specifically at any time in any one part of any ocean,”
The 1LOT disagrees with you. If you know the average temperature of system a system increased then via ΔT = ΔE/(c*m) you know how much energy it must taken up. Or vice versa if you know how much energy something is taking up then you know much it will warm. This is undisputable fact since energy is a conserved quantity.
Reacher51 said: “The few fish that live at the ocean’s surface presumably didn’t notice a theoretical average increase of 0.01C per year for the roughly three years that they were alive. How has our society come to a place where people find this meaningful?”
Let me give you an example. Using the Takahashi 1993 formula d[ln(C)]/dT = 0.042 we can rearrange it to C/Co = exp(0.042 * dT) and conclude that since exp(0.042 * 0.01) = 1.0004 and 400 ppm * 1.0004 = 0.17 ppm that the ocean warming cannot explain the 2.5 ppm/yr increase in the atmospheric CO2 concentration.
It’s hard to know where to start with this. So, in no particular order, a few points:
You pasted a graph several days ago representing global surface temperatures with a large adjustment made to the years 1880 – 1945. The numbers in question were describing surface temperature, which you can tell simply by looking at the text below the graph that describes it as, “Global mean adjusted and raw surface temperature”.
Your basis for this chart, which derives from NOAA, seems to have been an article by Hausfather, in which he clearly explains that, “The adjustments that have a big impact on the surface temperature record all occur before 1950…The large adjustments before 1950 are due almost entirely to changes in the way ships measured temperatures.” So it is these adjustments to pre-1950 ocean surface temperatures that were the issue in question.
Although the numbers originated from NOAA, they were posted in the Hausfather article, and Hausfather seems to have written a paper supporting those adjustments, so I have described these numbers as being Hausfather’s. Hausfather, NOAA… makes no real difference.
The NOAA information you referenced is a similar or identical heavily adjusted data product blending estimates of ocean surface and land surface temperatures. It does not depict in any way an average temperature for the whole Earth. It depicts an average temperature estimate for the surface of the Earth, which is a different concept.
The initial question was whether this data product produces a usefully accurate picture of average surface temperature. It does not. The data that underlies the average is sparse and unreliable. Since the weighted average of garbage does not magically transform into useful scientific knowledge, we need to accept what we have, which is very little.
It is of course possible to take crappy data, adjust it any way you choose, weight it any way you choose, and then to calculate a weighted average of those crappy numbers to a precision of as many decimal points as you choose. It is also possible to run statistical analysis on those same numbers and to produce something that you can technically describe as “uncertainty.” That is all possible, and some people seem to have based their careers on doing so. The problem is one of actual epistemological uncertainty, as opposed to the statistical construct of uncertainty, which arises in this case from the fact that the raw data and adjustments made to them are simply not fit for purpose.
Short of a time machine, there is no way to turn the sea surface temperature data we have from that period into anything that genuinely describes historical nature within a useful margin of error- not referring to a statistical margin that could be calculated equally well with made up numbers, but instead referring to a genuinely descriptive number that could be used to tell whether, e.g. the world was really 0.3C warmer than you thought previously.
You can pretend that random spots on sea lanes accurately represent all 139 million square miles of ocean surface, which they obviously don’t. You can further pretend that the temperature readings were all done at consistent times of day, that the buckets used were all identical, that the depth each bucket sank into the ocean was completely consistent, that the buckets rested on deck a consistent amount of time in a consistent amount of sunshine every day, that the thermometers were placed into the middle of each identical bucket and held for an equal amount of time, and that the sailors were meticulous in reading and recording data, absolutely none of which could possibly be true. You can then further imagine that you have a great reason for making additional adjustments to all of those numbers in order supposedly to make them consistent with readings from other imperfect instruments that occurred later. You can crank out numbers to your heart’s content, get published, and get enthusiastic pats on the back from people who want to use your adjusted numbers to confirm their belief in a tribal dogma that is now an integral part of their very identity. All of those things are possible. But truly, genuinely, knowing something accurate about sea surface temperatures in the year 1900 to a fraction of a degree is completely impossible, whether for you, for me, for Hausfather, or for any other human being either inside and outside of NOAA.
We do have some useful data about ocean temperatures from the Argo monitors, which date back to 2006- so about 15 years of data. That is good. We do not, however, have much useful data for what is going on in the ocean layers that lie below those sampled by Argo. So once again, to the question of energy in the system, there is still quite a lot that is unobserved. On top of that, and perhaps even more relevantly, we have absolutely no idea what natural temperature patterns obtained in those locations on a daily basis for the prior 4.6 billion years preceding the Argo probes. If we now have some understanding of daily ocean temperature in the top 2000m of the ocean for the last 15 years, how do those temperatures and changes then compare with previous periods? Are they greater, lesser, faster, slower, more or less stable? Were the comparable daily thermometer readings in those locations warmer, cooler, or completely identical in the years 1910-1945? No one has any idea whatsoever, since they didn’t exist.
If upper level ocean temperatures have fluctuated in the past, which is a certainty, then what natural forces were driving those changes, and what portion of the energy we see today arises from those natural forces? Again, no one actually knows. People make assumptions and guesses (and if you don’t know, you can’t conclusively say that they are wrong, which then forces you to believe they’re right, as I now understand it), but, in fact, they do not know.
People make loads of assumptions about prior states and put those assumptions into their models, the output of which they then use as a basis for comparison against real empirical data (which essentially describes a large chunk of supporting evidence presented in Shuckmann 2020). However, using SWAG as model inputs, then treating the model outputs as a valid basis on which to draw conclusions, tells you nothing about empirical reality. It tells you a lot about the appeal of circular reasoning, but nothing very much about nature.
This is but one general problem with modeling the Earth’s climate and making a relatively simplistic energy budget for it (simplistic relative to the overwhelming complexity of the actual climate system, not relative to the enormous black box of code). We have no good understanding of natural variability within the climate system. Modeling that variability based on assumptions, e.g. assuming that CO2 is a driving force in whatever you’re modeling, and then using the output of your CO2 driven model to show that whatever force you’re modeling must have been less in the past due to the fact that CO2 was lower, is sort of approach that passes muster in climate science, but is entirely useless in actually understanding reality.
The Earth’s energy budget is a theoretical construct, the details of which are not actually known, and the Shuckmann paper you linked to incorporates more assertions and assumptions than anyone could usefully address in one comment. This website has numerous comments and articles that address just about every topic brought up in Shuckmann, if not some of the papers themselves, and I suggest you use it as a reference if you would like to get more perspective on whether we understand each of the climate’s subsystems and how they interrelate over time.
The climate is not a simple system. It is an impossibly complex, dynamic, and possibly chaotic system, which we are unable to model correctly in detail. The specific amounts of energy in that system, how the energy flows and on what timescales, numerous emergent feedback effects, and the dynamic interrelation of all the above are unknown, except in gross terms. Powerpoint slides of energy flows are as useful in understanding climate response to a small forcing as a slide of human anatomy is to understanding how a few micrograms of a new molecule will impact a human being. The plethora of climate models that exist- incorporating an enormous range of inputs, outputs, and starting assumptions, tuned and re-tuned continuously to match the past and almost entirely inaccurate in forecasting the future until they are re-tuned yet again- strongly suggest that we have no real understanding of how climate will change and develop over time, whether a small theoretical positive forcing from CO2 has any noticeable impact on the entire system, or whether any current changes we observe are arising from more than natural variability.
third, Norway
And here is what Berkeley Earth shows for the global average temperature. Notice that it is consistent with the graph I posted above.
So which is it? Is Tmin going down around the globe? Or has the 1910-1945 mid-range anomaly been *adjusted*?
You can look at Tmax records from around the globe and see if it was warmer. You don’t need an “average” temp. An “average” temp loses the very data you need to decide what was warmer. Just saying the average was warmer is nonsense, it is not a measurement, it is a calculated metric that actually describes no real thing. Where would *YOU* go to measure the “average” temperature?
The crew doing the “corrections” should have the same reputation as Sir Cyril Burt.who corrected the data on intelligence testing.
Some years ago I used the Surface Station project ‘1’ (best) stations to show that while homogenization did remove some UHI from urban stations, it added spurious warming to all but one of the suburban/rural stations. Guest posted here. No reason to expect the BOM Acorn results to be any different.
Thank you for at least pointing out that there are legitimate reasons for making corrections to raw data. However, given the nature of the climate change business, it’s easy to imagine biases creeping in and gradually gaining acceptance. Apparently there needs to be a better process for refereeing data correction methodology.
Still the hubris.
Once you correct data, the default position is that you have probably added an artificial trend – not that the sun shines out of your proverbial and only a Big Oil still would disagree.
So you would not allow for a correction for TOB, for instance?
A TOB may be necessary, but it should only be done once not multiple times.
You assume that you did it wrong even if you got the answer that you wanted and check. I was under the impression that the trend of the corrected data is bigger than either of the trends of data separated into groups of times of observation. Maybe not proof of wrong doing but good enough reason to doubt the conclusion.
And if you compare this with what has happened in
Australia with a change to equipment clearly adding half a degree to average maximum temperatures and probably up to 2 degrees on extremely hot days. I suspect a correction would still increase the warming trend, but they refuse admit that any correction is needed.
The point is to stop a record when no correction process has been done such as a year long parallel run between the old and new to obtain a station by station correction factor.
Distinct records should be stopped and new ones started. It is a shame that that will add uncertainty, but it is proper science.
If you don’t know which station running in parallel is most accurate then how do you know if the correction should be plus or minus? Just assuming the newest station is the most accurate is not a justified assumption.
TOB as far as I know has never been physically tested by running thermometers in parallel to obtain a per station record. There is only mathematical justifications.
I’ll give you my perspective. Fronts moving through can cause more change than Time of Observation possibly could.
The averaging of Tmax and Tmin is a median temperature anyway and has not been integrated to find the true avg temperature during a day. We don’t have the DATA to do that.
TOBS is not any different than UHI adjustments. UHI does cause real increased temperature. Why do we adjust for that? So scientists can say, “Look we are reducing artificial temperature growth”. Notice that no scientist will say that UHI temperatures aren’t real! They just don’t want to have to explain that UHI is caused by humans, not CO2.
The *REAL* correction for TOB is to totally change how temps are recorded. If you want to know the true “global” temp then each station should record the temp a 0000UTC and at 1200UTC. You’ll get two snapshots in time of the actual global condition.
Of course you’ll have to start a brand new data set. So what?
The method used in the post, if it is valid, should indicate whether the TOB adjustments are adding artificial warming or not.
Are you trying to determine if pairwise homogenization creates a timeseries with a higher warming trend than the unadjusted data? I’m trying to figure out what you mean by “artificial”.
No. This is the misconception that I particularly wanted to guard against.
I expect adjustments to warm or cool the unadjusted data. I don’t mind if all the adjustments always add warming. That may also be legitimate.
What I’m concerned with is that the magnitude of the adjustment is correlated with the final adjusted temperature. This clearly should not be the case.
The final Acorn adjusted temperature should correlate only to the real temperature at the site (hopefully) and have no relation to the amount of adjustment.
The fact that there is a strong co-relationship between these two factors, to my mind, can only mean that there is non-climatic warming added by the homogenization process.
By artificial I mean non-climatic.
Thanks for the question.
One more clarification.
This does not mean that all the warming due to the adjustments is non-climatic.
If that were the case then the correlation coefficient would be 1.0.
It is actually 0.56 indicating significant non-climatic warming due only to the adjustments.
If the correlation coefficient were zero or close to it then the homogenization process would be doing its job properly without adding any non-climatic warming.
Yes. Data are data, observations. No correction.
Even TOB. Why not write the models in a way that they use the observed values and their times, instead of “correcting” the data?
I know that this would be a huge lot of work (I had to do some limited work of this kind). But would produce less a huge lot of BS.
It starts with disclosure, documentation and making all raw data and its correction algorithms available for scrutiny. You would think that this should be a standard scientific practice, especially for controversial science used to support trillions of dollars in economic damage.
Agreed, most certainly.
This makes no difference. A respected organisation fiddles the data and sends out press statements that today was the hottest ever.
A diligent individual assesses the changes and determines them to be speculative at best. The individual posts his findings where?
Australia has had technical reviews of the BoM. They have found homogenisation adds warming. A few recommendations get made and that is where it ends. Nothing changes.
Homegenisation is simply data fraud and accepting it is as valid is where the rot sets in.
“Homegenisation is simply data fraud and accepting it is as valid is where the rot sets in.”
Yes, it is. The data fraudsters should be put in jail considering the consequences of their temperature lies.
Not to mention especially because they work for you and attempting to fool the boss is a big no-no.
There is no legitimate reason to “adjust” recorded data. If data is incorrect, it should be rejected and new “created” data should not be substituted.
The basic reason for correcting this data is to maintain the illusion of long records. This is not a valid reason for creating new, and I won’t call it data because it is not, information.
I have seen the excuse that computers can be unbiased when doing this! ROTFLMAO. It really doesn’t matter if you do it manually with pen or pencil or program an algorithm. It is making stuff up.
Does anyone here think an engineer gets to “correct” measurement data after something is built? Do certified labs get to “correct” past measurements? If data is not fit for purpose it should be rejected immediately, not fiddled with.
Try and get a legitimate reason for “correcting” other than to maintain long records, you won’t get one.
Some things are useful to calculate averages for.
Fuel used on journeys for example for a particular vehicle.
But averaging temperature records from different stations is just a “mucking about with numbers” exercise.
It’s anomalies, not raw temperatures being averaged.
Is that the case? I thought temperatures were compared with nearby stations and obvious mistakes adjusted/averaged/homogenized out. Thereafter the anomalies were determined.
Or am I not following the thread here?
If you adjust a station you automatically adjust the anomaly.
Look at a temperature map on your evening news. Which ones are correct and which ones should be “corrected”?
How are the anomalies calculated? If corrections affect the absolute temps then they also affect the anomalies.
Temperatures are AVERAGED for a day, AVERAGED for a month, AVERAGED for a year. Station anomalies around the world are AVERAGED. Baselines are AVERAGED for 30 years. AVERAGES of AVERAGES. And meanwhile falsely claiming that averaging can add precision to physical measurements.
How anomalies basicly are calculated ? 😀
Anomaly starts when people repalce data by any other thing, call it “anomalies” or whatever word you like.
Well done, Mr. Irvine!
Re: The case for saying that the Australian BOM homogenisation algorithm adds artificial warming to the record is strong …
Strong enough that you have soundly shown that the burden of proof is STILL on the data fiddlers to prove their adjustments are based in reality.
The burden of proof was always on the AGWers. They have yet to make a prima facie case (having yet to produce even a scintilla (heh) of data which makes their conjecture about human CO2 more likely than not). Your powerful analysis simply confirms this fact.
**********************
btw: LOVE 🙂 Italian food and also the name of the Italian restaurant, “Criniti’s Woolloomooloo” 😄 in the photo, but….. kinda wondering how it illustrates your point….. care to give a hint?🤔
there is a row of gas fired outdoor heaters. So, maybe not so hot as everyone says? Or illustration of local nature of UHI?
“Added Non-Climatic Warming”
Thanks for the CLUE, Mr. Rotter.
Thank you, Mr. Gillard. Your surmising was logical and intelligent. The significance of that photo was pretty obscure. Clever, but, obscure….
It is a fact that changing the time of observation can make a difference to the average temperature computed for a given station . What it can not do is affect the record high or low for a week or a month or a year. So when the large number of US stations record record highs in the thirties you may be sure that somewhere at that station the noted record temperature was observed. True there may be station moves or equipment changes to consider, and of course n a well maintained system these will have been recorded. But a TObs correction can not affect a record temperature high or low
“But a TObs correction can not affect a record temperature high or low”
We should use Tmax, not the average when looking at Global Warming. If you want to highlight the warm years, use Tmax.
Tmax shows we are not experiencing unprecedented warming today because it was just as warm in the recent past, according to Tmax.
If data adjustment to pre 1980 data is the sole cause of a warming trend, then it should become apparent pretty soon that earlier data should not be given too much weight in policy making. That’s probably too much to hope for, but…
There’s also data adjustment after 1980, especially in the last twenty. In 2011 this was reported in the BoM’s annual climate report.
‘The Australian area-averaged mean temperature in 2011 was 0.14 °Cbelow the 1961 to 1990 average of 21.81 °C. This was the first time since 2001′.
Not any more – both have disappeared in the ‘adjusted’ Time series graph.
http://www.bom.gov.au/climate/change/#tabs=Tracker&tracker=timeseries
Many of the past 20 years have been adjusted upward by 0.1C – 0.2C.
Trying to figure out if you have a game here. And what it is
The data prior to 1980 in many cases was adjusted DOWN to cool previous warm periods the purpose being to make the past look static and flat and stable and suddenly increase, Steve Case has shown graphs of this many times.
Eliminating all data before 1980 means only looking at the start of the most recent warming from the recent coldest time and is therefor something a closet climate alarmist would promote.
We need to use all the real data, which for the USA at least would show the 1930s as warmer than the past 40 years.
Adjustments actually reduce the warming trend relative the raw data. If we don’t apply corrections to address the biases caused by station moves, instrument changes, time of observation changes, differences in ocean observations (bucket, engine, buoy, etc.) then we would be forced to conclude erroneously that the warmed more than it actually did [1].
This just looks like adjusted data vs more adjusted data.
The blue line is unadjusted or raw data.
The red line is contains adjustments for station moves, instrument changes, TOB changes, bucket vs engine vs buoy differences, etc.
The gray area is the difference of adjusted minus unadjusted.
I am shocked, shocked I tell you.
Are you dealing with a mean or median? If it is the mid-range of the max and min, you don’t know whether the temperature distribution over 24 hours is normal or skewed. Therefore, it would be more appropriate to refer to it as a median, or preferably, the mid-range measure. Calling it a “mean” or even an “average” is not the kind of precision necessary for rigorous statistical analysis.
PCC is a very roundabout way of dealing with trends. You can just look at the distribution of “magnitude of the correction”, ie difference of adjusted and unadjusted trends, as I did many years ago here. That would collapse Fig 1 into a histogram.
“It is almost certain that the corrections are adding significant non-climatic warming to the raw data”
No, the corrections are removing non-climatic cooling from the raw data. Mainly, the transition to better shelters which excluded spurious radiative heating.
ACORN-SAT only covers the post-1910 period, when all shelters were Stevenson, with only a very few exceptions. The BoM should be ridiculed for failing to deal with pre-1910 data, a feat that has been achieved by Berkeley Earth, and by Linden Ashcroft for SE Australia in her thesis work, … and by me.
“The BoM should be ridiculed for failing to deal with pre-1910 data”
Of course BoM deals with pre-1910 data. Here, for example, is the BoM record for Cape Otway since 1864.
They used 1910 as the start for their ACORN adjusted set. There is a simple reason for that – Federation. The BoM came into existence in 1908, following Federation in 1901. Records before about 1910 were kept, in various ways, by the different colonies (NSW,SA etc).
C’mon Nick, you need to up your game, and avoid these irrelevant statements.
If this is so, and you may be right, there must be an audit trail. Is there?
I mean, there must have been studies which showed the magnitude of the cooling effect due to improved housing, and the use of the adjustment for each station must be documented to have started when the housing was improved.
It doesn’t sound from the original post that this kind of method has been employed, it sounds like the method used was purely statistical.
But if you’re right, and the correct method was used station by station, is there a reference somewhere which describes calculation and implementation?
Yes. Here is a BoM summary page of the various source publications.
Nick, it doesn’t work for me. It redirects to http://www.bom.gov.au, and then I see nothing about stations or housing.
michel,
sounds like a security settings issue. It’s a http: page. It works for my settings. You might find it works in a different browser.
Yes, found it, and scanned several of the papers (mostly their abstracts since many are behind a paywall).
I have not yet found anywhere which does what I am looking for, which would be:
1) quantification of the new housing effect when compared to samples of the old housing in the same location.
2) a list of stations, with the dates when new housing was implemented in them, and a statement that from that date there was an adjustment, and of how much, to that particular station.
I can imagine that such a program could be carried out successfully. You would have to run each station in parallel for some period with the two measuring points near each other, calculate the difference introduced by the new housing, and then adjust the series for that. `it would be a bit tedious but quite straightforward and indisputable once done.
Then you could remove the old measuring point and be reasonably sure that you had not introduced a discontinuity.
Its the method that ought to be used when making any changes to a station hardware, including the temperature measuring technology.
Is this what they in fact did?
In fact, if you did it like this, I don’t understand why any statistical homogenization would be needed. You’d have a valid series which you had shown to have been controlled for the changes to the instrumentation or its housing. It would be a bit like changing fever thermometers for a ward. First calibrate the two instruments, then adjust.
Something similar is done with PSA tests when prescribing Finasteride. Its usual to halve the PSA level considered to be indicative of possible cancer if a patient is on Finasteride. The danger level is, say, 5 without Finasteride and 2.5 is the equivalent danger level when the patient is on it.
So I can see that this would be a valid method, my question is, where’s the documentation showing that this is what was done?
How do you field calibrate two stations and tell which is the most accurate? Calibration, especially of thermistor sensors, is done in a controlled environment, e.g. humidity, airflow, as well as temperature. How do you do this in the field, especially to a +/- .005C uncertainty which is required to differentiate temps in the hundredths digit?
1) I posted that down below.
2) You can get the station adjustments here.
The method you describe is essentially what PHA is doing.
“2) You can get station adjustments here.”
Is less than useless for my local town of Albany WA (an ACORN-SAT station) bdgwx. Deceptive or dishonest would more accurately describe the “adjustments”.
They claim “Data available from 1910“, yet the Airport (where its measured) wasn’t built until 1942. They state “Observations commenced at the airport site… in April 1965. There were no overlapping observations [with Town] at the time“. No overlapping observations!
Although only 11km inland from the coastal town, the climate is completely different. Its often 8 – 9C warmer in summer, cooler in winter, with less winter rain and more summer storms. The arrival time and strength of the cooling sea breeze (known as The Albany Doctor to those inland) can make a huge difference on any day. The Airport is more extreme, and no local would claim its weather, or climate, is similar to Town’s!
So, “No overlapping observations” with the very different Albany Town climate, yet some “genius” at BOM knows exactly how to convert one place to the other, one climate to another, for the half century prior to 1965?. And in so doing, “create” an ACORN-SAT data record for “Albany”, wherever that is now. What?!
And the “Professionals” at BOM have the gall to call it data. Those first 55 years isn’t “data”, its delusion! Or fraud.
You get the same thing when comparing Boston weather to Somerville weather. Or when comparing Denver weather with Pikes Peak weather. I’ve seen papers that use stations 1200km away to determine the temp at a second location. Huh?
Regarding No. 1: Look up “Reeximination of instrument change effects in the US Historical Climatology Network” by Hubbard and Lin, 2006.
“It is clear that future attempts to remove bias should tackle this adjustment station by station”.
Microclimate differences between stations introduces bias into any other method.
You should also look at “Sensor and Electronic Biases/Errors in Air Temperature Measurements in Common Weather Station Networks”, Lin and Hubbard, 2004.
“The interchangeability of the MMTS thermistors in from +/- 0.2C from temperatures -40 to +40C and +/- .45C elsewhere”
“The RSS errors associated with using the CR23X datalogger and the USCRN PRT sensor were from 0.2 to 0.34C.
Thus even in the newest measurement station, the uncertainty is such as to preclude the typical differentiation of +/- 0.01C typically used to declare the “hottest year evah!”.
How do you know which station, the old or the new, is the most accurate? Running them in parallel won’t tell you anything if you don’t know which one is the most accurate. Is your adjustment plus or minus? Just assuming the newest station is the most accurate is not justified. Its calibration may have suffered during transport and installation!
TG said: “Thus even in the newest measurement station, the uncertainty is such as to preclude the typical differentiation of +/- 0.01C typically used to declare the “hottest year evah!”.”
michel and the lurkers, It is important to understand that this is a misleading statement at best. The uncertainty figures cited in Lin & Hubbard 2004 are for a single measurement at a single station. This uncertainty will be different than the uncertainty of a global average measurement. And the claim of ±0.01 C uncertainty for a global average is patently false. See Christy et al. 2003, Mears et al. 2009, Rhode et al. 2009, Lenssen et al. 2019, and Haung et al. 2020 for references on how the uncertainty propagates into a global average and what that uncertainty actually is.
You didn’t even bother to read the title let alone the actual study did you?
From just the abstract: “The biases of four commonly used air temperature sensors are examined and detailed. Each temperature transducer consists of three components: temperature sensing elements, signal conditioning circuitry, and corresponding analog-to-digital conversion devices or dataloggers. An error analysis of these components was performed to determine the major sources of error in common climate networks.”
E.g.: “where the R13, R14, R15, and R16 are fixed resistors with 0.1% tolerance”
E.g.: Any fixed resistors in the signal conditioning circuitry can produce the error of temperature measurement due to their tolerances and temperature coefficients …”
The title of the paper is “Sensor and Electronic Biases/Errors in Air Temperature Measurements in Common Weather Station Networks.”
The uncertainties quoted are *NOT* specific to any single measurement station. They are uncertainties inherent in the weather stations themselves.
As you and I have argued over and over – the uncertainties quoted in all your links assume the standard deviation of the sample means is the uncertainty associated with the mean. It is not. The uncertainty of the mean, i.e. the average is the propagated uncertainties of the individual elements, just as Lin and Hubbard did in their analysis.
I read the whole thing. That’s how know their conclusions apply only to single measurements at a single stations. In other words, they do not apply to anomaly averages over time and space. And I’m not saying that their conclusions are specific to any particular measurement or station. That should be plainly obvious since they only provide general conclusions that apply broadly to the station network. And the abstract you posted doesn’t even mention anomaly averages. You are conflating the uncertainty and bias of spatial anomaly averages with the uncertainty and bias of individual measurements. Those are different concepts with different values for their uncertainties and biases.
You didn’t read the entire thing. You didn’t even read the excerpts I provided from the paper!
Tell us exactly how the tolerance of the fixed resistors in a type of measurement station has anything to do with “single measurements at single stations”? The uncertainty in *ALL* of the stations using those type of resistors is the same.
Anomalies are subject to very same uncertainties as the absolute temps. If one absolute temp is 70F +/- 0.5F and the other is 71F +/- 0.5F then the anomaly calculated from those temps can be anywhere between 0F (i.e. 70F + 0.5F and 71F – 0.5F) to 1F (i.e. 70F – 0.5F and 71F + 0.5F). Anomalies simply don’t cancel out uncertainty.
“You are conflating the uncertainty and bias of spatial anomaly averages with the uncertainty and bias of individual measurements.”
As Lin and Hubbard point out in their other paper, spatial anomaly averages are not useful for determining widespread correction, there are simply too many microclimate differences between stations to provide any kind of spatial correlation in temperatures. Corrections should be done on a station-by-station basis. Since individual measurements are used to determine anomalies the uncertainty and bias of the individual measurements translates directly into the anomaly. You simply cannot wish those uncertainties away because they are inconvenient to treat.
I can’t tell that anything you just said is addressing my criticism of your post at 6:06am. My criticism is with your insinuation that Lin 2004 made statements regarding the uncertainty of a spatial average is misleading because Lin 2004 does not make any statements in this regard. The whole publication is addressing the question of temperature measurement errors caused by sensor, analog conditioning, and data acquisition systems. They provide no analysis of how these errors propagate through the anomaly, gridding, and averaging steps required for estimating the global average temperature and providing rankings of the years.
“My criticism is with your insinuation that Lin 2004 made statements regarding the uncertainty of a spatial average is misleading because Lin 2004 does not make any statements in this regard. “
You are lost in the multiverse! I quoted two DIFFERENT papers. Read them both!
Hubbard and Lin, 2004, addresses the uncertainty in electronic sensors. Hubbard and Lin, 2006, addresses the fact that corrections factors should be done station-by-station.
This is just more proof of your cherry-picking things out of the literature without actually understanding what you are cherry-picking. If you had actually *READ* my posts instead of just running off willy-nilly to do some cherry-picking you would have caught this!
Correction factors are done station-by-station almost certainly and at least in part because of Hubbard & Lin’s research.
No, they are not! Using neighboring stations to try and determine “change” points at a station is a prime example.
You stated “If we don’t apply corrections to address the biases caused by station moves, instrument changes, time of observation changes, “
And just how are those corrections usually determined? I assure you it is *NOT* being done on a station-by-station basis. It is being done by comparison with other stations!
The Menne & Williams 2009 method is station-by-station. Menne & Williams 2009 explain how the corrections are determined.
From one of the homogenization references: ” Likewise, alterations to the land use or land cover surrounding a measurement site might induce a sudden or ‘‘creeping’’ change (Carretero et al. 1998; Karl et al. 1988) that could limit the degree to which observations are representative of a particular region.”
Temperature measurements are supposed to represent the climate at the station site. If land use or land cover around the site causes a change in the temperature readings then that *IS* representative of a climate change at that site and that change should be carried through the data. It is *NOT* representative of an inaccuracy that needs to be corrected.
It is this kind of non-scientific logic that appears throughout climate science that makes it unreliable as a indication of what is happening in the climate.
Yes, I agree. Record the data as it comes from the instruments, then argue about what it means afterwards. Don’t change the data. Calibrate the instruments of course, so that when you make changes you do not introduce a step change in the observations which are due to the changes. That isn’t changing the data.
I am not clear that this is how its been managed.
I think there may be a misunderstanding about the process. The raw observations are not changed. They are stored in the ACORN repository as they were recorded. What ACORN does is provide an extra or companion dataset that is intended for climate research. This extra dataset takes the raw observations and converts them to anomalies and applies changepoint detection and correction to the stations as part producing a monthly average. This is true for similar datasets like USHCN and GHCN-M as well. Also, calibration of the instruments does not prevent changepoints especially if the changepoint is caused by station relocation or time-of-observation changes or some other similar situation that causes a jump in the values not because of the instrument reading the wrong temperature but because the instrument is reading a different temperature. What homogenization does is align station timeseries using the overlap period with neighbors thus removing the changepoint discontinuities.
If those “neighbors” are more than 50 miles away or have a different elevation or have different terrain or any of a host of other issues then how can they be used to “remove changepoint discontinuties”?
See Menne & Williams 2009 for the details of how it is done.
You didn’t answer my question. Neither does the link you provided. Your post is nothing but a non sequitur.
Pairwise comparison is useless if the two stations are not highly correlated.
I thought you were asking how pairwise homogenization works including neighbor selection, neighbor correlations, changepoint identification, adjustment calculations, and the evaluation of its effectiveness. The Menne & Williams 2009 publication addresses each of these and answers the question regarding how neighbors are used to remove changepoint discontinuities.
Malarky! The entire publication assumes that neighboring stations are 100% correlated with the temperatures in the missing areas. There isn’t *any* paper that I have read that does an actual calculation of the correlation factors to determine their approriateness.
TG said: “The entire publication assumes that neighboring stations are 100% correlated with the temperatures in the missing areas.”
Menne & Williams do not assume 100% correlation.
TG said: “There isn’t *any* paper that I have read that does an actual calculation of the correlation factors to determine their approriateness.”
Menne & Williams 2009 does. There is a whole section dedicated to explaining it.
“Menne & Williams do not assume 100% correlation.”
Of course they do. There is not one single mention in the paper about determining the actual correlation of neighboring stations. No weighting for elevation or humidity. No weighting for microclimate differences, e.g. ground conditions under the measuring stations. No weighting for terrain, e.g. is one close to a lake and another is not.
They just assume 100% correlation.
They explain nothing about microclimate. If you think the do then provide a quote showing that they do! All you are doing now is using the argumentative fallacy of Name Dropping!
If you aren’t seeing the description of neighbor selection and correlations then you aren’t reading the publication I linked to. Again, I am referring to Menne & Williams 2009.
I didn’t say I couldn’t find anything about neighbor selection and correlations. I’m saying their selection process totally ignores most of the factors directly affecting correlation, e.g. distance, elevation, humidity, terrain, etc.
The selection process is based on station correlation. The bigger the difference in distance, elevation, humidity, terrain, etc. the lower the correlation.
Hmmmm……
“The assumption is that similar variations in climate occur at nearby locations because of the spatial correlation inherent to meteorological fields (Livezey and Chen 1983). ” (bolding mine, tpg)
“The pairwise algorithm starts by finding the 100 nearest neighbors for each temperature station within a network of stations. These neighboring stations are then ranked according to their correlation with the target. The first differences of the monthly anomalies are used to calculate the correlation coefficients [i.e., Corr (Xt − Xt − 1, Yt − Yt − 1)] in order to minimize the impact of artificial shifts in determining the correlation (Peterson et al. 1998b).”
Nowhere in Sections 3 or 4 of Menne is anything but distance considered. First differences of the ANOMALIES tell you nothing about the absolute temperatures themselves. It is even worse that the anomalies are calculated using MONTHLY average temps. You could have two stations with totally different microclimates have the same monthly average temp or monthly anomalies show up as totally correlated when they are both inaccurate!
Menne: “For example, the algorithm relies upon a pairwise comparison of temperature series in order to reliably distinguish artificial changes from true climate variability, even when the changes are undocumented (Caussinus and Mestre 2004). Consequently, the procedure detects inhomogeneities regardless of whether there is a priori knowledge of the date or circumstances of a change in the status of observations (Lund and Reeves 2002).”
I’ve asked you this before and you continue to blow it off – what if the change identified is from the external environment and not from a station change or drift? How do you tell the difference? Why would you want to correct a station that is accurately measuring the temp at the station including the impact of ground cover change or the addition/loss of windbreaks or the addition of a nearby lake?
Think real hard about this statement from Menne: “Likewise, alterations to the land use or land cover surrounding a measurement site might induce a sudden or “creeping” change (Carretero et al. 1998; Karl et al. 1988) that could limit the degree to which observations are representative of a particular region.”
How can an accurate measuring station be limited by external factors such as land use or land cover? It would *still* be measuring properly. And just what temperature reading is representative of a particular region? Is it what an accurate measuring device turns out or the subjective choice of what someone *thinks* it should be churning out?
This is an example of how correlated temps are at any given time. Watch local weather displays of current temps. How do you pick which pairs to use? Remember, these are also integers, they have an uncertainty of +/- 0.5 degrees. This whole process assumes that close stations have small difference. How wrong. Averages of differences like this doesn’t diminish or remove errors and uncertainties caused by individual readings from devices. It is simply a leap of faith that statistics will allow you refine measurements with a specific resolution.
If only that was true. Instead the extra dataset is also used to justify press releases and policy decisions by governments.
It is true. Just because there is are people you don’t approve of using it doesn’t make it any less true. In a hypothetical world you might argue that some people should be banned from using it and then be the first to complain that BoM isn’t be open and transparent.
I remember reading something about the historical measurements being lost because there wasn’t sufficient storage capacity to keep them archived.
I can’t speak for ACORN, but I’ve never heard of that happening with GHCN. In fact, GHCN is always accepting recent observations and newly digitized observations from the past some decades old.
You sound like someone familiar with physical measurements and their variabilities. You need to remember that some folks here are mathematicians who see temperature measurements as only numbers to be manipulated to give an answer.
It hasn’t been “managed”! It something mathematicians have convinced themselves that can be done by looking for “anomalous” readings using computers.
They fail to realize these are measurements subject to numerous environmental conditions that can cause differences in readings between different devices and/or over time.
Another big problem is adjusting readings from the assumed point of change back to the beginning of the station record. This is done with no regard as to how an audit was to determine this is appropriate,
The basis for this is to keep “long records”. The reason for this is so that stations remain in the passband of the period being used for an anomaly baseline. That means there will be no “anomaly” for those stations after stopping their temperature record and it shouldn’t be used in a long term trend. Can’t have that!
My recommendation is that station records be closed whenever technology changes. If LIG thermometers were calibrated as they should have been, then they would have been reading the micro-environment of their location. Changing technology ( or even small location changes of a few feet) can change the micro-environment of the thermometer resulting in a new baseline temperature. Even parallel running can show only that there are different baselines DURING the parallel period. Extending that adjustment to the beginning of the station recording is just not allowable from a confidence standpoint.
Get ready to be frustrated!
Hubbard & Lin 2006 document the instrument change bias effect in the USHCN record as well.
“The homogenized U.S. Historical Climatology Network (HCN) data set contains several statistical adjustments. One of the adjustments directly reflects the effect of instrument changes that occurred in the 1980s. About sixty percent of the U.S. HCN stations were adjusted to reflect this instrument change by use of separate constants applied universally to the monthly average maximum and minimum temperatures regardless of month or location. To test this adjustment, this paper reexamines the effect of instrument change in HCN using available observations. Our results indicate that the magnitudes of bias due to the instrument change at individual stations range from less than −1.0°C to over +1.0°C and some stations show no statistical discontinuities associated with instrument changes while others show a discontinuity for either maximum or minimum but not for both. Therefore, the universal constants to adjust for instrument change in the HCN are not appropriate.”
Not very encouraging…
That 2006 comment refers to USHNN V1. Many improvements were made before USHCN itself (V2.5) became obsolete in 2014. I would expect that by now, individual adjustments for conversion to MMTS are made for stations.
How can individual adjustments be made unless you use a calibrated reference at each site to determine offsets? And since most stations drift gradually over time, how do you actually determine corrections to be made in past measurements by using one point in time?
It is unfortunate that LiGs are biased low relative to MMTS. But we can’t deploy MMTS in the past nor is it feasible to continue to use LiGs so scientists did the next best thing and applied a correction factor to homogenize the timeseries. BTW…USHCN and the broader GHCN began using pairwise homogenization on stations separately shortly after the Hubbard & Lin 2006 publication. See Menne & Wlliams 2009 for details on how it is done. This new method has proven to be very robust as compared against raw USCRN data. See Hausfather et al. 2016 for details. BoM uses PHA as well. See here. The WMO guidelines on homogenization is a good read as well regarding the topic of using station data for climate research.
The only observable reason for applying a correction factor is to keep a “long record” usable for trending.
It has nothing to do with how accurate the prior readings were.
Tell us how a calibrated LIG that reads freezing and boiling accurately can differ significantly from an MMTS temperature reading.
Do you think environmental factors such as housings, moved locations, etc. would cause a change?
Tell us how a “correction” makes previous data more accurate!
Yes. I think environmental factors such as housings, moved locations, etc. would cause a change. The corrections are applied to homogenize the timeseries before and after the changepoint.
You didn’t answer the pertinent issue. Why do you want to apply corrections? Is there a reason other than maintaining a “long record”? Why not just stop the station record and start a new one?
As I’ve told you previously I have no issue with the “scalpel” method that BEST uses.
How else can one maintain fictions such as “The warmest since temperature readings began!” unless one relies on the unexamined assumption that historical measurements as equal in all respects to modern measurements?
Nobody is assuming historical measurements are equal to modern measurements. In fact, it is the exact opposite. It is one reason why homogenization must happen at some stage of the analysis. If you aren’t homogenizing then you are assuming that the past is equivalent to the present in terms of the measurement method and consistency.
Implicit in this is the assumption that newer stations are *always* more accurate than older stations. That is simply an unjustified assumption. As Hubbard and Lin showed, corrections must be done on a station-by-station basis, not through homogenization with other stations
How do you know the past is *NOT* more accurate for any specific location? What formula do you use to determine that?
Do we need to let Hubbard and Lin know that you think their studies on this subject came to an incorrect conclusion and they need to revise it?
Again…homogenization is done on a station-by-station basis.
The point of homogenization is not to identify which station is more accurate. The point is to identify and correct the changepoints. It is not a requirement that the reference be accurate to do this.
Hubbard & Lin’s research is used by Menne & Williams. It is almost certainly because, at least in part, of Hubbard & Lin’s research that GHCN and ACORN perform the homogenization on each station independently.
Again, how to you determine if the change point is due to a change in the measurement device or in the external microclimate? If it’s due to external microclimate changes then the station shouldn’t be corrected using homogenization at all. The purpose of the measurement is to identify the temperature at that location, no matter what is happening with that location. And if a lake is built upwind of the station, why then it *should* show a change. “Correcting” it gives a misleading metric on the microclimate at that location.
Can *you* tell what is the cause for a “change* just using the data from a station? Can *YOU* tell from the data whether the ground below the measurement station changed from fescue to sand? Can *YOU* tell from the data if the change is from a new dam and impoundment upwind of the station? If *YOU* can’t discern the two different scenarios from the temp data then why do you expect others to be able to?
Right on target! BOOM!
Even the newest shelters suffer from micro-environmental biases. E.g. ground cover underneath the shelter. Hubbard and Liu studied this all the way back in 2002, twenty years ago! That being the case there is no way to identify non-climatic impacts merely by looking at the raw data itself. In other words the “corrections” are speculative at best, fraudulent at worst.
The Trenders won’t like this one.
TOB changes should not be accepted at all. If the data isn’t there, then it isn’t there. You don’t get to make changes to it based on what you think it should be. If you need more data, hourly temps instead of just high/low then go start taking those measurements. But you don’t get to put data in where it never was.
“You don’t get to make changes to it”
TOB doesn’t change the data. It makes a correction to the monthly average, based on how often observations show that the data accurately recorded by the min/max thermometer is not the min/max for that day, but a previous day. We know this happens, and how often. It is malpractice not to correct for it.
If you know for a fact that the recorded temperature is wrong, then you should be discarding it. “Correcting” the average is still making a change based on data you do not have.
Averages are made up too.
TOB correction assumes the recorded temperature is right. The thermometer and observer are assumed to have functioned correctly. No data is to be discarded or changed. But it is known that sometimes what the instrument recorded is not the normally understood max, but a carryover from the previous day. That creates a bias. Even that does not need correcting, unless the TOB changed. Such changes were recorded. That change creates an artefact up or down shift which can and must be calculated and corrected. There is ample information for doing that.
Since temperature data is so heavily autocorrelated it shouldn’t matter much whether a recorded Tmax is that of today, yesterday, or tomorrow (in relation to the recorded date). Shift a whole set yearly of daily Tmax temperature readings back or forth by a day – there still would be no reason to assume that monthly (let alone yearly) averages, nor the daily Tmax/Tmin difference, would change noticeably, because +/- 24 hrs is a RIDICULOUS amount of phase change over the 365-day sinusoid circle that is of climatic interest, and any theoretical effect of such a shift is swamped in the day-to-day “noise” we call weather. Signal processing 101: phase differences of less than one degree in a sinusoid signal (i.e. one with a dominating fundamental, in our case the 365-comma-something days period we call one year) simply do not matter when averaging several such sinusoids (namely readings from a bunch of stations), and neither the error nor its correction should make any perceptible difference to the yearly curve, let alone to the multi-decadal trend we call “climate”. The fact that it does proves that something else must be happening than just shifting Tmax readings by a single day, and your explanation is at least incomplete if not wrong.
It doesn’t matter if the TOB remains unchanged. It is when you change the TOB that it matters. That’s the core of the problem. In an effort to improve precipitation reports stations began switching from PM to AM observations. The unintended effect of that was to introduce a carry-over and drift bias into datasets used for climate research.
Tmax/Tmin daily differences change appreciably based on season. This propagates into all downstream averages.
Someone earlier suggested tracking just Tmax and/or Tmin as separate data (Tom Abbott maybe?). That would eliminate Tmax/Tmin difference ranges as an biasing effect.
“Since temperature data is so heavily autocorrelated it shouldn’t matter much whether a recorded Tmax is that of today, yesterday”
The point is that if yesterday was warm, it is likely that both yesterday’s and today’s Tmax were from yesterday afternoon, if the min/max was reset at say 5pm. That warm afternoon was double counted. Cool afternoons are not.
You have no data to prove that two days in a row with the same Tmax is “likely” TOB related. That is pure assumption. This especially true when looking at temps recorded as integers. One day may be 79.5 and the next 80.4 but would look the same. Far from the same temp.
Single days would not affect an average very much, especially over a year.
Tell us how you determine that subsequent days do not have the same temperature?
To make changes to data, some physical proof must be available to show that repeated temperatures and faulty. What is the proof?
Vose et al. 2003 used the Surface Airways Hourly database [1] for their analysis of the TOB bias.
So sequential days can not have the same temperatures? How do you separate those out? You are changing data based solely on numbers and not on any physical investigation, right. How do you KNOW an adjustment is needed?
More of your typical sophistry. The monthly average is used for many things, including coarser temporal-resolution summaries. If the monthly average is changed, it is functionally equivalent to changing the original raw data, except that you get to piously claim the original readings weren’t changed.
How would you deal with time-of-observation changes, station moves, instrument changes, bucket-engine-buoy differences, etc. without making an adjustment at some phase of the analysis?
As Gorman has suggested, moves and instrument changes should be treated as a new and different time-series unless a rigorous analysis can show that differences are negligible and that they can be used interchangeably. In most cases, it is probably better to acknowledge and accept that the data are not as accurate as desired and work with them, rather than applying subjective ‘corrections’ that may be as bad or worse, or simply unknown. Since, in all cases, one is sampling a continuously varying parameter, the first concern should be whether sampling theory is being satisfied, and what impact would result from dropping suspect data, or performing a low-pass filter to remove outliers.
With respect to engine-buoy differences, they should be treated as exactly what they represent — two different measurements. The engine-intake is the least accurate and precise because it was often unknown what the intake depth was, the thermometers were not highly precise, and probably last calibrated just prior to being installed. Putting lipstick on a pig doesn’t make it more attractive.
CS said: “As Gorman has suggested, moves and instrument changes should be treated as a new and different time-series”
That’s the method BEST choose. I have no objections to that method.
CS said: “With respect to engine-buoy differences, they should be treated as exactly what they represent — two different measurements.”
They are treated as different measurements already. Remember, the ocean datasets don’t have the concept of a timeseries because there is no station. They are immediately gridded. If a grid cell has say an 80/20 mix of bucket/engine measurements and then over a period of time transitions to say 20/80 then you’ll erroneously conclude that the cell warmed more because you biased the cell with the warmer engine measurements. A similar problem occurs with the engine/buoy ratio as well. If you don’t homogenize the measurements then how do you solve this problem?
Again, how do you *KNOW* which should be the homogenation control? If you don’t *KNOW* and can’t give an objective reason for your choice then you could be introducing unknown biases. What if they are *both* wrong?
BEST does not use pairwise homogenization so they don’t have to know.
Even the process of discarding some data requires judgement. There just needs to be a fundamentally scientifically good reason for doing it. You are always at the mercy of someone who is collecting and analyzing the data. On the other hand, people, especially people who have no basis for it, should not kill the message just because they don’t like the messenger.
Yes, so unless it is obvious that a mistake has been made, such as an impossible temperature, or a verified transcription error going from field notes to a database, all recorded data should be retained and considered part of the inevitable uncertainty arising from the process of measuring. Subjective judgement leaves too much room for mischief. One way of dealing with this is to flag individual readings as being suspect by assigning a greater uncertainty with a notation (Always an audit trail!) as to what the suspected problem is. I would assert that it is better to downgrade the claimed precision than to claim unwarranted accuracy and precision.
I think you might be misunderstanding what the TOB bias is. Vose et al. 2003 provide a pretty good explanation for what is going on. In a nutshell there are two significant components in play referred to as the carry-over bias the drift bias. The third bias caused by using daily as opposed to hourly observations turns out to be insignificant at about 0.03 C as compared to the carry-over bias of up to 2.0 C and the drift bias of up to 1.5 C. The TOB bias occurs mostly because the time at which the observations took place changed thus injecting the carry-over and drift biases into the raw record.
Vose describes a very silly way to calculate monthly averages. I must say I never knew that this naive type of “accounting” calculation was actually used in a situation that clearly is a type of signal processing with analogue input. What is needed to correctly weight each day is not a stepwise averaging Jan.1 to 31, Feb.1 to 28/29, etc. that is sensitive to one-day or even hour-wise mistakes, but a lowpass filter (preferably a Gaussian one with no “ringing” artifacts) with a cutoff frequency of 1/12 year, yielding the same number 365 datapoints that went into it, but smoothed to remove the daily “weather noise”. If one datum per month is actually desirable, using the MEDIAN of each monthly set of low-passed daily data would likely be the most reliable and least artifact-prone way of finding it.
I assume however that modern data processing should be able to deal with all 365 values of all stations as direct input to calculate the multiyear climatology. In that case, not even the lowpassing described above should be necessary, as local short-lived weather events are bound to disappear during the averaging process. Just like an audio engineer can average several noisy copies of an analogue recording that are phase-locked reasonably well to average out random noise and small phase/clock distortions, getting a cleaner signal without any arbitrary filtering.
“Simplifying” existing hourly data into daily, daily into monthly, and monthly into yearly, only makes sense to not crash the digital storage and/or computing power – realistic concerns in the 1980s but hardly in 2022, when audio data sampled at 192kHz sampling frequency and 32bit resolution is rapidly becoming an industry standard and is easily processed by simple desktop computers. 192,000 32-bit samples (i.e. _one_ second of _one_ audio channel in current commercial studio standard) of daily temperature data correspond to more than 262 YEARS of readings for one station (assuming 2 readings a day). Most likely, ALL the temperature data ever recorded since thermometers were invented is not much more data than one multichannel recording of a 5-hour Wagnerian opera, a totally routine amount of data to be lowpassed, mixed, averaged, denoised and otherwise mixed by a modern audio workstation that runs on a simple Windows laptop if needed, in little more than real time.
In other words, there is really NO EXCUSE for downsampling daily readings, irrespective of TOB and other typical “analogue-world” mistakes, as brutally as Voss’ method of calculating monthly averages suggests (with gross errors in the low-res output following from very minor errors in the higher-res input, i.e. the crude way of downsampling at least occasionally AMPLIFIES errors instead of averaging them out, as we are always told it happens). On the contrary, NASA’s and similar-sized institution’s computing power should be more than sufficient to UPSAMPLE all historic data to the currently standard resolution (hourly? per minute? per second?) to make them comparable and combinable 1:1, using well-known upsampling algorithms developed for importing low-res audio and video into modern workstations with minimal artefacting and distortion, and still end up with a very manageable quantity of sampling data: A century of HOURLY readings is a measly 876,600 32-bit samples and takes the space of under five seconds of highest-fidelity music, assuming a century of MINUTELY readings we multiply that by sixty and get the equivalent of four-and-a-half minutes of music, or 52,596,000 samples, or 200 MByte of data when using the garden-variety WAV format for 32-bit audio to store it – lossless compression easily cuts that already modest space in half or less.
Please don’t tell me that these are frightful, unmanageable amounts of data that make brutally cutting them down and introducing grotesque aliasing artefacts (like 2 deg. C error originating from a month of dates being misreported by 24 hours) inevitable and somehow acceptable. This déjà-vu of the earliest years of the Internet with their ghastly narrow-band 8-bit audio and video for lack of bandwidth and processing power is beyond ridiculous!!
When the uncertainty of each temperature measurement can be as much as +/- 0.5C, it wouldn’t seem to be of much use doing all the data manipulation you are speaking of.
I have long been an advocate of using cooling and heating degree-day measurements. They are an integral of the temperature curve above and below a specified set point. Degree-day values are used by HVAC engineers to size heating and cooling units in buildings. Assuming the temperature curves at least approximate a sine wave, the uncertainty of the integral should be far less than the uncertainty of individual temperature measurements. This would also provide at least some attempt at converting a time dependent variable into a non-time dependent value.
Doing this integral shouldn’t be any more complicated than calculating the power in a digitized audio tone. The integral of the temp profile would be essentially the same, calculating the power of the digitized temp data. A few outlier data points wouldn’t affect the overall integral very much so I’m not sure using a low-pass filter on the data would be of much use.
There are a lot of stations out there that can provide 1 sec, 5 sec, 1 min, or 5 min data throughout the day. This data goes back at least twenty years for most of the stations.
It’s high time the climate scientists look at different methods for coming up with a metric that can evaluate the globe rather than an “average temp” with all of its inherent problems. The global temperatures provide a multi-modal data set (think summer in the NH and winter in the SH – each with a different range between min and max depending on the season). Using averages with a multi-modal set of data is almost useless for describing what is actually going on. Using cooling and heating degree-days would help with this problem.
I’m not sure I’m understanding the criticism. Vose et al. describes what I consider the obvious method for determining the TOB bias based on a comparative hourly observation dataset. It sounds like you would prefer them to use a low pass filter that smooths the daily noise. I have no idea how you would quantify the TOB bias with such an approach. And your comment “that is sensitive to one-day or even hour-wise mistakes” regarding what I think is reference to current monthly averaging period is odd because it is the carry-over and drift mistakes that they want to quantify. That’s the whole point of the publication. And I’m not understanding how the bottom 4 paragraphs even related to the TOB bias.
From Vose et al: “Spatial averaging was accomplished by interpolating station-based values to the nodes of a 0.25deg x 0.25deg latitude-longitude grid, …. ”
Really? That’s a grid about 20 miles in latitude and 15 miles in longitude. If there were a station in the grid there would be no reason to “interpolate”. So we can assume that the stations being used to calculate the interpolation are at least two grid squares apart if not more, i.e. 40 miles in latitude and 30 miles in longitude. As I’ve pointed out several times here, these distances are far enough apart that the correlation factor is too low to be useful. That puts the interpolated, intermediate temperature in question.
It is simply unbelievable to me that this obvious of a problem is ignored by climate scientists.
It also indicates to me that you are back to your cherry-picking ways while not understanding what you are quoting.
And I’m not surprised you don’t understand what Mr. Berlin is saying in his last four paragraphs. A low-pass filter is not just a “smoothing” function. A low-pass filter is used to eliminate high frequency “artifacts” (think outliers) such as phase differences caused by TOB inconsistencies (for that is really what the different TOB’s are).
Since we have a multiplicity of measurement stations today, and have had since the 90’s, that can record temperatures with small time intervals we should be using the entirety of their measurement records to generate daily, monthly, and annual means. That would give us a 30 year baseline from which to analyze what is happening around the globe. Signal analysis algorithms that are well known today would be useful in eliminating the high frequency “noise” from these analyses
In fact, many of these stations are capable of recording elevation and humidity as well as temperature. This would allow analysis of the true heat content station by station by calculating enthalpy instead of using the very poor proxy of temperature alone.
It’s time climate science moved into the modern world.
Why do you need a low-pass filter to see if an observation was carried over to the next calendar day or drifted into the next month? Can you post a link to a publication that uses this method so that I can review it?
Once again, you are cherry-picking without the slightest understanding of what you are speaking about.
tim: “A low-pass filter is used to eliminate high frequency “artifacts” (think outliers) such as phase differences caused by TOB inconsistencies (for that is really what the different TOB’s are).”
What do you think TOB carry over is but a phase difference in the temperature profile signal? What causes a phase difference but a high-frequency impacts on the signal? Think sin(x) suddenly jumping to sin(x + a). What causes that?
Attached is a picture of a BPSK demodulator (binary phase shift keying). Note the low-pass filter.
Maybe you can explain how applying a low pass filter can help identify and quantify the TOB carry-over and drift bias. I’ll download the Tmin/Tmax observations and the hourly observations for the same station and we’ll work on this together. What is the first step you want me to do?
In other words you don’t have a clue as to how phase shift keying protocols work.
I’ve already posted you a graph of Tmax for the US between 1900 and 2018 based on all US reporting stations.
Do it this way and why does carry over even become a problem? If the Tmax for a few stations are out of sync with the majority what difference will it make? Do the same for Tmin.
Have the climate models predict global Tmax and Tmin instead of some funky “global average”.
I can download the data. That’s not the problem. The problem is I don’t how you want me to apply a low pass filter to either the Tmin/Tmax data or the hourly data to identify when a Tmin or Tmax observation got incorrectly applied to the wrong day. I don’t see what the big deal is in explain how you would do it and why it would be better at identifying carry-over and drift instances and quantify the effect.
If the wrong data is posted it will stand out as an outlier. Either there will be hole in the data for the prior day or an outlier for the next day, i.e. two Tmax entries. Either one represents noise (i.e. a phase difference) in the data which can be taken out using a low pass filter.
I’m not here to teach you how to implement digital signaling processes, e.g. a low pass filter. Go learn how to do that on your own time. You might start with the book “Digital Signal Processing Technology” from the American Radio Relay League. Page 2-14 and Appdx D might be of interest.
The issue isn’t trying to identify carry-over but eliminating its impact.
How about this. Post 2 Tmax and 2 Tmin observations on back-to-back days and the corresponding 48 hours worth of observations. Then provide a simple algorithm using a low pass filter that 1) can be used to answer the question whether the second Tmin or Tmax was duplicated and 2) the magnitude of the bias it causes. Do this with a low pass filter.
How about I give a number of sequential days where Tmax was the same? Can *YOU* tell whether that was reality or a carryover affect? If you can’t then how do you write an algorithm to do so?
How about 12/27/2021 and 12/28/2021? Or 9/1/2021 and 9/2/2021?
Which of those two pairs were due to the same Tmax on each day and which of those two pairs were due to carryover from one to the next?
The way you identify carryover is by running the entire temperature protocol through a low pass filter. If the 3pm temp was 81 and the 4pm temp was 95 that is not following a sine wave. As the sine wave reaches maximum the slope of the sine wave goes to zero. The slope between 81 and 95 is obviously not a slope approaching zero. That means that there is a high frequency component that is affecting the slope of the curve. A low-pass filter will remove that high frequency component.
I told you once I’m not going to try and teach you digital signal processing. If you want to learn it then do what I did and get some books and study up on it.
I’ve attached a sample program for a low-pass filter using a sinc window.
Here's a python stub: from __future__ import division import numpy as np fc = 0.1 # Cutoff frequency as a fraction of the sampling rate (in (0, 0.5)). b = 0.08 # Transition band, as a fraction of the sampling rate (in (0, 0.5)). N = int(np.ceil((4 / b))) if not N % 2: N += 1 # Make sure that N is odd. n = np.arange(N) # Compute sinc filter. h = np.sinc(2 * fc * (n - (N - 1) / 2)) # Compute Blackman window. w = 0.42 - 0.5 * np.cos(2 * np.pi * n / (N - 1)) + \ 0.08 * np.cos(4 * np.pi * n / (N - 1)) # Multiply sinc filter by window. h = h * w # Normalize to get unity gain. h = h / np.sum(h)I doubt it will help you any. As usual it’s not obvious that you have even the slightest grasp of the subject.
TG said: “How about I give a number of sequential days where Tmax was the same? Can *YOU* tell whether that was reality or a carryover affect? If you can’t then how do you write an algorithm to do so?”
Yes. I believe I could tell given the daily observations being evaluated and the hourly observations used for testing. The Vose method is to compare the 2nd Tmax with the maximum from the hourly observations. If they disagree then you have a possible carryover.
TG said: “Which of those two pairs were due to the same Tmax on each day and which of those two pairs were due to carryover from one to the next?”
What is the Tmax and the 24 hourly observations for each day?
TG said: “A low-pass filter will remove that high frequency component.”
Why do you want to remove the high frequency component? How does that help identify a carry-over?
TG said: “I’ve attached a sample program for a low-pass filter using a sinc window.”
Thanks. Lucky for me I know both python and Basic so I can see what is going on. The question is…how does this help you identify Tmax or Tmin carry-overs and drifts? What are you wanting to apply this to? The Tmin or Tmax daily timeseries or the hourly observation timeseries?
Really? This would be impossible with the two days I gave you since they both had the same sine curve.
You might catch errors in data propagation this way but single errors shouldn’t generate a correction in the temperature measurements. If this happens on a continuous basis where the Tmax entry doesn’t match with temp measurements for that day then the answer is not to correct all of the temperature measurements but to substitute the actual Tmax based on the current day’s measurements. It’s not like you don’t have the data because if you didn’t have it then you couldn’t identify that a difference exists.
The temperature profiles are the same.
Did you *really* think about this before posting it?
You said: “Yes. I believe I could tell given the daily observations being evaluated and the hourly observations used for testing.”
What would you be doing but looking for high freq noise in the temp curve on a visual basis? Why can’t you just run it through a low-pass filter? What’s the difference?
Really? You can see what is going on? That’s just not obvious based on your assertions.
You apply the filter to the *entire* temperature curve data. This just tells me that you don’t have a clue as to what those code snippets are doing!
Again, for the tenth time, the temperature curve represents a sine wave, at least approximately. That sine curve has a base frequency plus high frequency components that drive the data away from a pure sine wave. One of those high freq components would be phase shifts caused by data that is out of sequence. A low-pass filter will remove those high freq components such as the phase shifts caused by out-of-sequence data.
The digital LPF is nothing more than a windowed fast fourier transform. the fourier transform breaks out the frequencies associated with the data and the window provides side lobe reductions, a inherent problem with the FFT. With this you can identify the components causing the phase shifts.
It’s pretty obvious that climate scientists are living in the 70’s before digital signal analysis began to be used by geologists and petroleum engineers. They are at least 40 years behind modern techniques. Thus their dependence on using processes that inherently give no or ambguous results. That you think these processes are the sine qua non of temperature analysis only shows how far behind the curve *you* are.
This doesn’t even begin to address wavelet analysis!
TG said: “Really? This would be impossible with the two days I gave you since they both had the same sine curve.”
If the two days have the same temperature curve then there wouldn’t be a TOB issue in that case. In other words, both reported Tmax and Tmin values would be correct in terms of the TOB.
TG said: “What would you be doing but looking for high freq noise in the temp curve on a visual basis? Why can’t you just run it through a low-pass filter? What’s the difference?”
We’re not looking for high frequency noise. We’re looking for instances where the reported Tmax or Tmin is incorrect due to the temperature being higher (for a Tmax) or lower (for a Tmin) at the previous day’s reset than it is at any point in the current day.
TG said: “You apply the filter to the *entire* temperature curve data.”
Which one? There are two. There is the daily Tmax and Tmin timeseries and there is the hourly T observation timeseries.
TG said: “A low-pass filter will remove those high freq components such as the phase shifts caused by out-of-sequence data.”
I get that the temperature profile can resemble a sine wave and that instantaneous observations can deviate from it. I also get that low-pass filters, Kalman filters, etc. can be useful tools for analysis. What I don’t get is how you and AlexBerlin think they are useful in this particular case. Based on some of your commentary I’m wondering if you are misunderstanding what the TOB problem is. Can you provide a brief description of the problem you are trying to solve?
“If the two days have the same temperature curve then there wouldn’t be a TOB issue in that case. In other words, both reported Tmax and Tmin values would be correct in terms of the TOB.”
In other words, one of the stations reading incorrectly but matching a neighboring station would be missed! And *you* think the process works?
“We’re not looking for high frequency noise. We’re looking for instances where the reported Tmax or Tmin is incorrect due to the temperature being higher (for a Tmax) or lower (for a Tmin) at the previous day’s reset than it is at any point in the current day.”
That *IS* the very definition of high frequency noise. You don’t know enough about the subject to even understand that simplistic concept.
“Which one? There are two. There is the daily Tmax and Tmin timeseries and there is the hourly T observation timeseries.”
Please stop showing off your ignorance. Fourier transform assumes a continuous signal. The windowing takes care of truncating the signal. If you want to make the signal cover two cycles then do so! How many cycles of the lowest frequency signal show up in an audio clip lasting several seconds? The daily temperature profile takes in *both* Tmax and Tmin. That’s why the daily temp profile looks like a sine wave.
“What I don’t get is how you and AlexBerlin think they are useful in this particular case.”
You don’t understand because you apparently know nothing about signal analysls. Fourier analysis is very useful in studying things like sound waves, e.g. sounding pulses used by geologists. This is no different than studying the temperature.
“Based on some of your commentary I’m wondering if you are misunderstanding what the TOB problem is. Can you provide a brief description of the problem you are trying to solve?”
I’ve given you this at least twice in this thread. Repeating it one more time won’t help you understand it any better. But here goes.
You say you want to look for discrepancies in the temperature curves that would indicate a TOB carry over to the next day. That causes a “bump” in the sine wave representing the temperature profile for the next day. That “bump* is caused by a high frequency component causing a phase or amplitude shift in the base curve. A low-pass filter using a windowed fast fourier transform will identify the frequency components of that daily temperature signal and indicate whether you are seeing a TOB discrepancy.
It doesn’t get a simpler than that. If you can’t understand it then do what I said, get some modern books on digital signal analysis. This has been around since the 80’s. It’s not magic hand waving. The same apparently needs to be done by a lot of climate scientists. I haven’t seen a single paper doing any type of modern signal analysis on the temperature profile curves.
Let me make this clear. I’m not questioning the usefulness of signal processing. I’m not questioning low-pass filters. What I’m questioning is how you apply it to this problem and why you think it is a better method than Vose et al. 2003?
Perhaps the best place to start is from the very beginning. Given the following 7-day Tmax/Tmin timeseries how do you use a low-pass filter to identify TOB problems? If I told you one of the values is a carry-over which one does your algorithm say it is? Does your algorithm need extra information to answer the question? If so, what extra information does it need?
89/62, 90/61, 90/63, 91/61, 89/61, 89/64, 90/61
Did you not bother to read my post? Geologists and petroleum engineers have been using this technique for DECADES!
Who are you to question how digital signal processing is accomplished and how it is used? Are all those geologists and petroleum engineers over the past 4 decades only fooling themselves?
Again, for the umpteeenth time, the daily temperature profile is a repeating sine wave – the VERY THING DIGITAL SIGNAL PROCESSING WAS DEVELELOPED FOR! It’s what the Fourier process is made for.
A better method? Pairwise comparisons suffer from the typical climate science problem – you simply don’t know what is going on with the stations in reality. Digital signal processing works on a station-by-station basis. You analyze the data from one single station at a time. That eliminates the problems with trying to do comparisons with multiple stations when you don’t have full information on any of the stations!
Wow! Just WOW! You are lost in your own multiverse! The temperature profile is more than just the Tmax/Tmin temperatures – AS YOU YOURSELF ACKNOWLEGED! You said you would need all the temperatures throughout the day in order to identify a phase shift such as a carry-over of a Tmax.
You don’t even understand the subject well enough to ask on-point questions. Stop. Just stop. Go learn about Fourier analysis, Fast Fourier analysis, and Digital Fast Fourier Transforms.
Don’t get too carried away with “upsampling”, there simply isn’t enough data to get better resolution. To use your example of a Wagnerian opera, imagine doing that with only very small samples every minute. There simply is not enough data. Two temps a day just won’t suffice. That isn’t true with later thermometers and your suggestion is a good one for those.
If you need to change data then the data must be thrown out as being rubbish. This is how the mining industry works. Changing the commodity grade data used as an input for a geological block model will change the overall grade of the model. That is go to gaol stuff for mining companies and their directors. Why is it illegal for us but not illegal for the BOM?
As I’ve said, people who deal in the real physical world understand the value in measurements and their accuracy and precision. Mathematicians only see data as numbers to be manipulated to achieve a goal.
And, it has been my experience, over years of observation, that mathematicians are conditioned to work with exact numbers, such as integers, and overlook the fine points of finite precision and how to properly handle it. I can remember spending time in an integral calculus course going over a chapter on errors, and then never using it again.
An alternative (possibly more compelling) way to shoot-down ACORN-SAT is to check its internal self-consistency. There are many clusters of stations that are climatically close enough to do this by simple subtraction and plots of 12-month moving averages of the differences. You will find clear step changes, some at exactly the dates that adjustments are made, indicating the invalidity of those adjustments.
This conclusion needs to be checked for the recent version of ACORN-SAT.
I had to read this post five times before I realised that it uses the term “anomaly” to mean the temperature trend over time. This is an extremely anomalous use of the word “anomaly”. Once that was out of the way, the conclusion is obvious:
“Out of tiny Acorns grow mighty hoax”
Sorry, I couldn’t resist it.
Just came across this animated video of a plane departing from Athens airport in 1978 and almost crashing. The interesting thing is that the commentary says (17.45) that the temperature ‘tends to be higher over the city centre’ and that as the plane moves away from the city centre to the less-built-up part of Athens the outside temperature drops from 43 degrees C down to approximately 38 degrees C. This is one of the factors which helps to save the plane from crashing. An interesting example of the urban heat island effect.
I don’t understand that, if the raw data is uncertain for any reason don’t use it — at all.
The individual temperature increments involved are so relatively minuscule that the compounded ‘corrections’ over a 100 year period produce a trend that is entirely artificial.
Instead of +1C over 100 years, in a different mood I’m sure the compilers could ‘correct’ the data to produce a zero or negative trend and equally justify the result.
All measuring stations have uncertainty. All stated values should include an uncertainty value and that uncertainty should be propagated into any metric calculated from those uncertain values. The AGW clique, however, just arbitrarily assume that all stated temperature values are 100% accurate. Trying to propagate the uncertainty through the metrics is *difficult* so they just ignore it.
If they were 100 % accurate, why correct them ?
Even if they were 100% accurate (they aren’t) you’d still have to apply corrections to address station relocations, non-climatic sighting changes, time of observation changes, etc. BTW…no one assumes temperature observations are 100% correct. That is just a patently false statement. In fact, a lot of effort goes into propagating uncertainty through the averaging process. See Christy et al. 2003, Mears at el. 2009, Rhode et al. 2013, Lenssen et al. 2019, Haung et al. 2020, etc. for details.
We moved all these stations, quick let’s use it as an excuse to adjust the temperature up.
Why does moving stations increase readings 99% of the time
Movement alone means you are not measuring the same thing. Any discrepancy between the old and new stations is reason to end the old record and start a new one. Without special procedures you can not assume the discrepancy is an inaccuracy that requires correction.
Why have you not commented on the obvious desire to keep “long” records? Why are stopping old records and starting new ones such a major problem? Why do calibrated LIG thermometers have a need to be corrected? Is there a systematic error in LIG thermometers?
Look I don’t have a problem if someone will scientifically determine a constant systematic error between LIG and MMTS. I do have a problem with not recognizing that different enclosures, microclimates, locations, etc. can introduce differences that are unamenable to “corrections” because they are accurate as they are.
My comment is going to be the same the last time you brought it up and the time before that. BEST already does it that way. I have no criticism with that method.
Obviously true. However, with rigorous analysis it might be possible to demonstrate that the differences are negligible within the context of the accuracy and precision of the original station, and either can then serve as a proxy for the other. However, I’m of the opinion that typically little effort is expended to prove the changes are negligible, particularly when the distance is (undefined) small. It is an unstated, and unexamined assumption.
How do you justify subjectively ‘correcting’ suspect values, to maintain an apparent precision, instead of assigning greater uncertainty to the ‘outlier’ to reflect the fact that the data set is not optimal?
I’m not understanding the question. Homogenization has nothing to do identifying outliers. Homogenization is the process identifying and correcting changepoints.
This sub-thread wasn’t addressing homogenization.
Krishna Gans asked: “If they were 100 % accurate, why correct them ?”
They correct them because of station moves, time of observation changes, etc. It is done by applying pairwise homogenization. It would still be required even if the observations were 100% accurate and even if there were no outliers.
For at least the 5th time, Hubbard and Lin showed that using correction factors like this can introduce more bias than you originally had. Homogenization REQUIRES you to decide which stations are more accurate than others – a SUBJECTIVE choice at best. Differences in the microclimate at each station can be the complete reason for different readings – meaning both stations may be accurately measuring the temperature!
Where does Hubbard & Lin say the Menne & Williams 2009 method introduces more bias than already exists in the raw record? Where do they say pairwise homogenization requires deciding which stations are more accurate?
I’ve given you the paper titles and excerpts from them. Are you incapable of reading the entirety of papers yourself? Or do you just not understand what they say?
They say that corrections must be done on a station-by-station basis. Why is that so hard for you to understand? Micro-climate differences make it impossible to determine corrections on a widespread basis like homogenization. A station located over fescue grass will encounter different impacts than one located over bermuda grass or tarmac. How do you homogenization anything when you can’t identify what these differences are? And if you can’t identify the differences they you will unwittingly introduce biases you don’t even know about.
As usual, tell us how you can homogenize a station in Denver with one on top of Pikes Peak? Totally different environments and if you can’t identify the impacts of those different environments then how does one keep from introducing errors in the record?
TG said: “They say that corrections must be done on a station-by-station basis. Why is that so hard for you to understand?”
They are done station by station.
TG said: “As usual, tell us how you can homogenize a station in Denver with one on top of Pikes Peak?”
I don’t know that you can. I don’t know if it meets the neighbor selection criteria.
“They are done station by station.”
Identifying stations that have a change is done by comparing to other stations. How is that a station-by-station correction?
“I don’t know that you can. I don’t know if it meets the neighbor selection criteria.”
Since elevation, humidity, and terrain are not considered when selecting neighbors then they would certainly meet the selection criteria you keep trying to use.
Stations are compared to neighboring stations. That’s why it is called pairwise homogenization. It is done station-by-station because the pairwise is done for each station separately.
Elevation, humidity, terrain, etc. are all examples of reasons why stations are not correlated at 100%. And as you can see in the publication neighbors are ordered by their correlation score prior to selection for pairwise comparison. So yes those examples plus many others are factors that are considered.
How do you determine a changepoint except by comparison with another station that may or may not be more accurate than the one being studied? The only way to SCIENTIFICALLY do this would be to do a calibration on the station in question in a qualified lab. Otherwise you could be introducing more inaccuracy than just not doing any correction at all!