I mentioned the poster earlier here. Now I have the video interview and the poster in high detail. See below.
Here is the poster:
And in PDF form here, where you can read everything in great detail:
I mentioned the poster earlier here. Now I have the video interview and the poster in high detail. See below.
Here is the poster:
And in PDF form here, where you can read everything in great detail:
@ur momisugly Mosher et al , Best data;
“You can have your opinion about the data but not your own data”, it appears you are making up data “40,000 station record segments or station fragments from about 7,000 weather stations” ?
I would add the 1930’s and 1940’s adjusted or made up data.
Paul Homewood says:
December 19, 2013 at 9:59 am
Zeke
Here is the detailed Berkeley analysis of that station in Iceland: http://berkeleyearth.lbl.gov/stations/155466
You can see that two breakpoints are detected; one empirically detected one ~1860 and a second associated with a documented station move around 1940. The net effect of these two breakpoint adjustments is to slightly decrease the trend relative to the raw data
That’s interesting because the GHCN set adds about half a degree of warming on from 1965 for Stykkisholmur
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/products/stnplots/6/62004013000.gif
They seem to have been confused by the well documented sharp drop in Icelandic temperatures then, a time known in Iceland as the “sea ice years”. The GHCN algorithm seems to think this was a measurement error.
++++++++
History be damned. The past can be changed by creating new and improved literature. Some day, these folks will look back and reflect on their lives. They will have to come to terms with the deceit. If these people have a conscience they won’t be feeling any sense of self worth as their carbon 14 levels begin their half-life journey downwards.
Sorry, Anthony, I can’t help chuckling at the way Zeke keeps referring to you as ‘So’ in his replies. (OK….I’ll go now….)
“Hansen Stitched together stations that were NOT the same station.”
There are no stations nearby each other for them to be stitched up based in the Arctic. The Arctic stations show the warming in the 1930s and 1940s were like it has been recently.
http://img141.imageshack.us/img141/7617/arctictempstrend.png
Hi Sun Spot,
There are 40,000 stations, not 40,000 fragments of 7,000 stations. You can find them all here: http://berkeleyearth.org/source-files
Hi Anthony,
I agree that not all UHI is step changes (there is likely a combination of trend biases and step changes). However, by comparing rural and urban stations we can bound the magnitude of UHI. Our paper found a bias of about 20% of the century scale min temperature trend in the raw/TOBs data that was eliminated post-homogenization. This is true even if you completely toss out urban stations and only use rural stations to homogenize and to construct a temperature record.
We’ve looked in the past at well-sited stations (no detectable urbanization via MODIS, ISA, Nightlights, etc.) and found trends that are slightly lower than urban stations. Our poster at the AGU two years ago covered some of this work. UHI is certainly real, but its effect is relatively small compared to mean temperature rises, and it should in principle be detectable and removable.
Modern networks like the CRN will help us determine going forward if actual temperatures are more comparable to homogenized datasets where breakpoints are detected and removed or raw datasets where they are not.
Let’s use some real data about the region of Montreal Canada. 1953 and 2010 are possibly the warmest on record. 1949 and 2012 are also warm. 2009 and 1954 were colder with colder winters.
Stations Distance, Elevation, GHCN
McGill : ———- , 56.9m, CA007025280
Airport : 13.8 km , 36.0m, CA007025250
L’Assomption: 36.4 km , 21.0m, CA007014160
1949 max, mean, min
McGill : 12.1, 8.0, 3.8
Airport : 12.3, 7.7, 3.0
L’Assomption: 12.1, 6.7, 1.1
1953 max, mean, min
McGill : 12.5, 8.7, 4.9
Airport : 12.9, 8.3, 3.7
L’Assomption: 12.8, 7.1, 1.4
1954 max, mean, min
McGill : 10.4, 6.8, 3.3
Airport : 10.5, 6.4, 2.2
L’Assomption: 10.4, 5.1, -0.3
2009 max, mean, min
Airport : 11.2, 6.6, 2.1
L’Assomption: 10.6, 5.3, -0.1
2010 max, mean, min
Airport : 12.7, 8.4, 4.1
L’Assomption: 12.4, 7.3, 2.2
2012 max, mean, min
Airport : 13.3, 8.5, 3.7
L’Assomption: 12.6, 7.2, 1.7
McGill and L’Assomption have a difference of 1.5C during the 1950s. The airport has possibly warmed from more snow melting during the colder months. L’Assomption has colder winters and much less pavement around. The change associated with snow melting on streets and roofs could easily make a difference on distances greater than the height of the atmosphere, so the 2 sites could still be part of the same UHI.
So how much time did it take for McGill to become 1.5C warmer than L’Assomption? How much warming does it suggest for the last 60 years? Don’t forget, L’Assomption already had some urban development in 1950 and America was discovered in 1492.
@Zeke Hausfather at 11:11 am
There are 40,000 stations, not 40,000 fragments of 7,000 stations. You can find them all here: http://berkeleyearth.org/source-files
Your link is to a page with 14 separate datasets, I guess each with parallel of data files such as a site_stations.txt. Some records have -999 lat lons.
Do you have a link to some intermediate files where there is one table of station sites with lat longs keyed on station ID and datasource?
@Zeke Hausfather at 11:11 am
Looking at just the Global Summary of Day TAVG and the Monthly Climatic Data of the world,
The data_charicteristics file.txt.
There are 27,233 sites from the merge of these datasets.
But 10,267 have fewer than 60 “Unique times” (5 years)
14,286 have fewer than 120 “Unique Times” (10 years)
only 805 have greater than 600 “Unique Times” or 50 years.
2885 sites have at least 360 Unique times (30 years) with less than 100 Missing values.
40,000 stations? By some measure maybe, but not all stations are created equal. But are fewer than 20% of them shorter than 30 years? Half shorter than 10 years. These stats aren’t looking at the segments yet.
Here is a map of the stations from
the Global Summary of Day TAVG and the Monthly Climatic Data of the world,
25,569 sites, colored by elevation. No distinction yet between stations with 1 year of records or 50 years. That will come tomorrow.
http://i1367.photobucket.com/albums/r782/Stephen_Rasey/131219_Spotfire_A_Best_Sites_2Sets_zps2472c844.jpg
Steven Mosher says:
December 19, 2013 at 8:23 am
“I wanted to investigate the BEST method for this effect when it first came out but they only provided massive files that could not even be loaded on a PC. Despite the claims of total openness, this effectively meant it was not available to be checked.”
Before I actually Joined the best team I wrote software to download and use it on a PC. takes up about 2GB of memory. maybe you dont know what you are doing
—-
Care to share the source code so that others with possibly less prior knowledge can also use it?
Zeke Hausfather says:
December 19, 2013 at 11:11 am
“There are 40,000 stations, not 40,000 fragments of 7,000 stations. You can find them all here: http://berkeleyearth.org/source-files”
Care to do a plot of number of unique ID’s against data lengths for those without serious database skills and resources?
Stephen Rasey says:
December 19, 2013 at 11:20 pm
“No distinction yet between stations with 1 year of records or 50 years. That will come tomorrow”
May I suggest a histogram of number of unique IDs against record length (in yearly buckets?)
Now if we can only get a similar histogram for segment lengths (may have to be monthly!) as well then a much clearer picture may emerge.
There are about 1200 USHCN stations, regarded as a long record, high quality dataset.
What trends would we get if we just used them?
By adding a whole load of lower quality, and potentially very dodgy, stations, aren’t we risking contaminating the good stuff?
Is this a case of “more = less”?
@RichardLH at 3:03 am
May I suggest a histogram of number of unique IDs against record length (in yearly buckets?)
How about a reverse cumulative Distribution of Number of Sites that exceed X number of years of data at each site?
http://i1367.photobucket.com/albums/r782/Stephen_Rasey/131220_BEST_SiteDistrib_by_LengthofRec_3Sets_zpsa2f13000.jpg
Data sets used: 3 of 14 TAVG from http://berkeleyearth.org/source-files
data_characterization.txt files from:
Global Summary of Day TAVG
Monthly Climatic Data of the world
GHCN Monthly Version 3
There are three lines on this chart. The top line is a simple
DCOUNT(database, 1, Unique_Time > [X axis Value]) in Excel.
I have NOT yet validated for duplicate stations that might be in the 3 datasets, so it could be lower. With this line, there are 34,513 stations with at least one month. so Zeke’s 40,000 stations isn’t wrong. but there are only 20,140 stations will more than 120 Unique_Times (10 years) and 10,140 with more than 360 months (30 years).
In my opinion, if you are going to discard the absolute value of the temperature reading and rely on the trend of temperatures, I think any station will less than 30 years will do more harm than good.
Stephen: Well with a minimum 30 year span at an individual station and expecting >90% data coverage, when using all of the databases listed above and merged together, I get only 31,485 Unique IDs.
If I extend that to >95% data coverage the number drops to 24,088 Unique IDs.
That is from everything combined and, as you say, that is somewhat less than Zeke’s 40,000 stations with only what I would consider to be a reasonable data validation criteria.
Continuation of 8:09 am
As I said, there are three lines on this plot. The upper blue line is raw count based upon how many Unique_Time are greater than X.
In the data_Characterization.txt files, there is a “Missing Values” column. There is a Begining data column and an Ending date column. Missing Values column should be about equal to the (months between beginning dates and ending dates) minus the Unique_Times column. Ok. So a 30 year station might have a gap of a month or two, no problem if you make reasonable assuptions that preserve long term trends. But what is the sense of the number of gaps in the record?
The lowest red line distribution is the blue line with the added criteria that “# of Missing Values is less than 10% of Unique Values” or no more than 1 in 11 months are missing. Under this measure, only
9,717 sites have at least ONE year of data,
7,113 sites have at least 10 years of data,
4,343 sites have at least 30 years of data with less than 36 months of gaps.
I think a 10% gap rate is generous, but if you treat the data well, it can still deliver good climate data. (Mind you, slicing it up into bit size chunks does not qualify as “treating the data well”, but that is my opinion). But that “Less than 10% missing values” only leaves us with 7,113 sites with as few as 10 years of data.
What if we relax the missing values criteria to 20%. It didn’t make much difference!
I had to relax the missing values criteria to 50% of Unique_Times, I.e. 1 month in 3 is missing! to get the green line. With this very loose criteria, we get
13,807 sites with at least 2 years of data,
11,339 sites greater than 10 years of data,
8,394 sites with 30 years of data.
Compare the green and red lines. 11,339-7,113 = 3800 sites have more than 10 years of data, but is missing between 12 to 60 months (1 – 5 years worth) of data.
@Paul Homewood at 3:28 am
Is this a case of “more = less”?
Yes. It is a case of sausage making that would turn Upton Sinclair’s stomach.
I think this WUWT post and comment thread from March 31, 2011 is still on point.
Correction to Rasey:12/19 11:03 pm
40,000 stations? By some measure maybe, but not all stations are created equal. But are fewer than 20% of them
shorterLONGER than 30 years? Half shorter than 10 years. These stats aren’t looking at the segments yet.(sorry, it was well after midnight.)
RichardLH at 8:31 am
Well with a minimum 30 year span at an individual station and expecting >90% data coverage, when using all of the databases listed above and merged together, I get only 31,485 Unique IDs.
If I extend that to >95% data coverage the number drops to 24,088 Unique IDs.
I’m still using the three: GHCN Monthly V3, Global Summary of the Day, Monthly Climate Data of the World. When I do a DCOUNT( database, ID, (Unique_Times greater than X AND Missing Values Less than X)), i.e. 50% data coverage from begining to end of station life, I get a maximum of 15,051 at 2 years, pretty flat at 14,500 to 13,995 at 17 years, then it gradually merges with the blue.
As anyone taken a second look at the jackknife confidence interval calculation yet?
@Jeff Id 9:24 am.
Do you know where there is a FIRST look at jackknife confidence intervals?
What parameter do you want to analyze? There are thousands of degrees of freedom in the BEST process, because each breakpoint changes the jackknife of every other parameter.
The answer for me is no. That would appear to me to be a monumental calculation if you had all the data in memory. What are your ideas?
Stephen Rasey at 9:20 am (continuation)
I just did a first pass on duplicate station IDs in the three dataset and I did not find any.
Zeke Hausfather says:
December 19, 2013 at 8:55 am
Mario Lento,
Reanalysis products use weather models (not climate ones) fed by observations to estimate changes in temperature, precipitation, etc. at both the surface and various levels of the atmosphere. You can find more about them here: https://climatedataguide.ucar.edu/climate-data/atmospheric-reanalysis-overview-comparison-tables
+++++++++
Thank you Zeke: I’m a process control engineer. Typically I try to back up and see the big picture before delving into the process control science and engineering. Before drilling down to the details, I try to sort out what’s being done.
From what I read, the BEST results are much more consistent with weather “models” than (observations?).
That the BEST results agree with the models, is evidence that the BEST (version of the) temperature data were good? In conclusion, the models prove the BEST results were right?
My understanding is that the “models” are programmed show that CO2 is the driver to warming and that these models show more warming than observations. So a product (the models) that shows more warming than observations agrees with BEST versions of data means the BEST versions of data also show more warming than observations. Right?
Can I conclude that if temperature readings from the urban areas were completely removed and only rural areas were observed, the results would show less warming, and no longer fit the CO2 tuned models that show warming that is not observed.
Pardon all the ways I am trying to state this, but the big picture can sometimes be obscured by the details.
@RichardLH at 8:31 am
Well with a minimum 30 year span at an individual station and expecting >90% data coverage, when using all of the databases listed above and merged together, I get only 31,485 Unique IDs.
RE: Stephen Rasey 8:47 am
The lowest red line distribution is the blue line with the added criteria [AND] that “# of Missing Values is less than 10% of Unique Values” or no more than 1 in 11 months are missing. Under this measure, only
….
4,343 sites have at least 30 years of data with less than 36 months of gaps.
So it looks like both of us were looking for something similar here: 30 years of station records or 360 Unique_Time value and no more than 36 months missing.
We get very different results: your 31,485 to my 4,343.
You say, “using all of the databases listed above and merged together”… Above where? All of the BEST data sources and not just the three I chose to experiment with? Were you using the data_characterization file or links to the actual dated temperature records? Is the problem that I missed a big database? If so, what are the likely candidate databases that have such full coverage. It is impossible to tell from the page Zeke links to the size and scope of the dataset without first downloading them, so some hints to the one or two that might be more complete than the three I used (GHCN Monthly V3, Global Summary of the Day, Monthly Climate Data of the World), would be helpful.