UPDATE: 08/07 2:PM PST MLO responds with improvements to the CO2 data reporting
Approximately 24 hours after I published my story on the January to July trend reversal of CO2 at Mauna Loa, the monthly mean graph that is displayed on the NOAA web page for Mauna Loa Observatory has changed. I’ve setup a blink comparator to show what has occurred:
For those who don’t know, a blink comparator is an animated GIF image with a 1 second delay consisting only of the two original images from NOAA MLO. Individual image URLS for: August 3rd ML CO2 graph | August 4th CO2 Graph
Now the there is no longer the dip I saw yesterday. Oddly the MLO CO2 dataset available by FTP still shows the timestamp from yesterday: File Creation: Sun Aug 3 02:55:42 2008, and the July monthly mean value is unchanged in it to reflect the change on the graph.
[UPDATE: a few minutes after I posted this entry, the data changed at the FTP site] here is the new data for 2008:
# decimal mean interpolated trend
# date (season corr)
2008 1 2008.042 385.37 385.37 385.18
2008 2 2008.125 385.69 385.69 384.77
2008 3 2008.208 385.94 385.94 384.50
2008 4 2008.292 387.21 387.21 384.46
2008 5 2008.375 388.47 388.47 385.46
2008 6 2008.458 387.87 387.87 385.51
2008 7 2008.542 385.60 385.60 385.25
and here is the 2008 data from Sunday, August 3rd:
2008 1 2008.042 385.35 385.35 385.11
2008 2 2008.125 385.70 385.70 384.85
2008 3 2008.208 385.92 385.92 384.38
2008 4 2008.292 387.21 387.21 384.59
2008 5 2008.375 388.48 388.48 385.33
2008 6 2008.458 387.99 387.99 385.76
2008 7 2008.542 384.93 384.93 384.54
Here is the MLO data file I saved yesterday (text converted to PDF) from their FTP site.
Here is the URL for the current data FTP:
ftp://ftp.cmdl.noaa.gov/ccg/co2/trends/co2_mm_mlo.txt
I have put in a query to Pieter Tans, the contact listed in the data file, asking for an explanation and change log if one exists.
UPDATE 08/05 8:55AM PST I have received a response from MLO:
Anthony,
We appreciate your interest in the CO2 data. The reason was simply that
we had a problem with the equipment for the first half of July, with the
result that the earlier monthly average consisted of only the last 10
days. Since CO2 always goes down fast during July the monthly average
came out low. I have now changed the program to take this effect into
account, and adjusting back to the middle of the month using the
multi-year average seasonal cycle. This change also affected the entire
record because there are missing days here and there. The other
adjustments were minor, typically less than 0.1 ppm.
Best regards,
Pieter Tans
UPDATE 08/05 4:03PM PST
I have been in dialog with Dr. Tans at MLO through the day and I’m now satisfied as to what has occurred and why. Look for a follow-up post on the subject. – Anthony
UPDATE 08/06 3:00PM PST
A post-mortem of the Mauna Loa issue has been posted here:
http://wattsupwiththat.wordpress.com/2008/08/06/post-mortem-on-the-mauna-loa-co2-data-eruption/
– Anthony
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Beaker: current ENSO view neutral-
•ENSO-neutral conditions are present in the equatorial Pacific Ocean.
I have been having an ongoing email exchange with Dr Tans. In the last go round, I asked him to confirm my understanding of the nature of the adjustment.
I wrote:
“Am I correct that when you changed the program to account for the missing 20 days in July, there was a backward propagation of adjustments filling in for other missing days?”
Dr Tans replied:
“You are good.
When I was at it, I made another adjustment to the program. I used to fit 4 harmonics (sine, cosine with frequencies 1/year through 4/year) to describe the average seasonal cycle. I changed that to 6 harmonics.
Therefore, there will be small systematic differences as a function of time-of-year in the de-seasonalized trend. That will be on top of adjustments caused by months in the past during which there were a number of missing days not symmetrically distributed during that month.”
I think we are too conditioned to data getting Hansenized and may be jumping to conclusions. So far, unlike Hansen, Dr Tans has been forthright with communicating his approach.
Pieter Tans has replied to several questioners who asked why the data have been altered. His reply says:
“The reason was simply that we had a problem with the equipment for the first half of July, with the result that the earlier monthly average consisted of only the last 10 days. Since CO2 always goes down fast during July the monthly average came out low. I have now changed the program to take this effect into account, and adjusting back to the middle of the month using the multi-year average seasonal cycle. This change also affected the entire record because there are missing days here and there.”
But this explanation is extremely implausible in the light of the observation (above) by Kim Mackey (03:49:35).
Kim Mackey lists the data adjusted data in chronological order and shows that the adjustments were plus, minus, plus, minus, etc. throughout the series. And – according to Pieter Tans – this pattern of alternate increases and reductions is a result of corrections to days when data is missing as a result of faulty equipment and/or weather.
But the odds of this pattern happening by chance are the same as tossing a coin 25 times and gaining a sequence of heads-then-tails throughout all 25 tosses. It could happen, but the odds of it happening are comparable to the odds of winning the grand prize in the National Lottery.
Indeed, the extremely improbable pattern of the changes is reason to reject the (above) suggestion by crosspatch (10:26:49) that the data alteration is insertion of simple “fill” values unless evidence is provided to demonstrate that such values were calculated and used.
When I smell a rat then I look to see if a rat is present so I can eliminate it.
We need to see
(a) the original data,
(b) the adjusted data, and
(c) the algorithm used to make the adjustments.
Until these three sets of information are provided then I think it is naïve to accept the data as being valid.
Richard S Courtney
I’m skeptic of the explanation you received.
Of course, it is possible to have problem with equipment. It is possible that partial data would have an effect on average.
Where I am skeptical is that the adjustment seem to be only made when data don’t correspond to the preconceive notion of the people taking the measurement. Even more many rely on this data set which has no other corroborating evidence. Climate science is really a world leader in data adjustment and creation.
Dee Norris:
“I think we are too conditioned to data getting Hansenized and may be jumping to conclusions. So far, unlike Hansen, Dr Tans has been forthright with communicating his approach.”
I agree. Lets not go making mountains out of molehills. The overall impact of the change is trivial and I am satisfied with Dr. Tans’ explanation. We should be applauding his transparent approach to what happened and what he has done and move along. Personally, I am satisfied with the explanation he has given.
I am still bothered, though, that only data that goes unexpectedly in one direction gets attention but that is more of an overall issue and not something directly aimed at Dr. Tans’ work.
@richard S Courtney
I think that your post and my update from Dr Tans missed each other.
If Dr Tans changed the harmonics that were used to compute the missing days and groom the data, it could very well lead to the adjustment pattern we are seeing on the plot here: http://tinyurl.com/6qb3sg
without a complete description of his methodology before and after, it is hard to really be certain.
Magnus: Weather data from the Kona airport (FAA) should show wind direction and velocity daily for all 24 hours. Whenever the wind direction from the south (Kona wind), the VOG (outgasing) will corrupt data collection on Mauna Loa. I should not be too difficult to compare NOAA data that is discarded with FAA actual wind data to determine if NOAA is cooking the books rather than accumulating data honestly.
To Dee Norris:
You accurately point out to me:
“If Dr Tans changed the harmonics that were used to compute the missing days and groom the data, it could very well lead to the adjustment pattern we are seeing on the plot here: http://tinyurl.com/6qb3sg
without a complete description of his methodology before and after, it is hard to really be certain.”
Yes, I completely agree. Indeed, I take your pint to corroborate my assertion that:
When I smell a rat then I look to see if a rat is present so I can eliminate it.
We need to see
(a) the original data,
(b) the adjusted data, and
(c) the algorithm used to make the adjustments.
Until these three sets of information are provided then I think it is naïve to accept the data as being valid.
Richard
Dr Tans replied:
“You are good.
When I was at it, I made another adjustment to the program. I used to fit 4 harmonics (sine, cosine with frequencies 1/year through 4/year) to describe the average seasonal cycle. I changed that to 6 harmonics.
Therefore, there will be small systematic differences as a function of time-of-year in the de-seasonalized trend. That will be on top of adjustments caused by months in the past during which there were a number of missing days not symmetrically distributed during that month.”
So, they can just do this? A guy can sit down to fix a problem from three weeks ago that has supposedly just been discovered, and while he is “at it,” go ahead and make another adjustment to the historical record just because he feels like it at the moment?
Is that how it works? Are these changes logged somewhere? Is the rationale for these changes the product of previous discussions, of which a record is kept, or can it be just a spur of the moment thing that someone happens to think of while doing something else? Can the changes he describes be replicated?
In short, the July fix, plus the harmonics change, plus the previous “missing days” fix had to be done all together, and they had to be done precisely last night?
@crosspatch
If my understanding of the July CO2 level changes is correct, the rate of (negative) change increases throughout the month and into August.
So, if we only average the last 10 days of the month to get the mean, we are injecting a bias into the value cause we will show a greater decrease for the month. Likewise, had the data been for first 10 days, we would see a lessor decrease for the month. Each monthly mean is adjusted to the mid-month according to the data-set description.
It then becomes a safe bet to say that the old July 2008 data was incorrect, but we don’t know how correct the new data set is. To get an estimation of the accuracy of the adjustment, I would want to see the last 10 days of July for several years past and compare them to 2008. If the fit is good, then adjusting using prior history is reasonable. If the fit is not not good, then we have a problem.
Any unusual acceleration in CO2 levels for July should also manifest itself in August to some extent. We will just have to wait and see.
The adjustment made is minor as far at the trend is concerned. The issue is not likely malicious intent, so I think people should calm down, (my opinion, not acting as moderator).
But…there ARE two issues that this raises, noted earlier.
1. The mindset or groupthink that triggers such scrutiny by those reporting the data. That is easy. If people believe a theory, they will only notice data as aberrant in conflict with that theory. So groupthink or a belief system does have an unconscious influence on actions like this.
2. The audit trail that should be in place when such adjustments are made. This is the big unresolved issue.
Talk about observational bias writ large. Every other science is acutely aware of observational bias but the climate crowd seems to think they are scientists so it doesn’t apply. Dr. Tan only “noticed” the distortion when it went outside his preconceived notions of what the correct values “should be.” How many bad readings that matched his expectations were never investigated because there was nothing obviously wrong? At the NIH my blood pressure was measured on the same machine that measured my grandparents’ in 1947. A gorgeous walnut cabinet with double blind calibrated mercury pressure gauges. The researcher explained that without the double blind the readers would unconciously bias the readings. I notice the same possibility in the manual entries for my local NOAA manual entries where the daily readings are suspiciously too often the low of the day and the low is rarely lower than recent past lows. It is classical observational bias to resist a new lower low when all the past readings are there to compare.
Somebody needs to graph the last 10 days of the month unadulterated data going back and see what that looks like.
@richard S Courtney
At this point, I don’t smell a rat, I smell a lack of transparency and an audience who is worried that Hansen’s behavior is spreading to other scientists.
He’s just following the Hansen Rule . . all adjustments have to be up.
So simple, so easy. Keeps the fear factor high and the Grant money flowing.
Beaker:
Thanks for the info. I mistakenly identified the base index year as 1977, it was 1973-which was an El Nino year.
@Xavier
I would bet that the change from 4 harmonics to 6 harmonics was planned for a while. That sort of thing you don’t do on the fly. The needful correction of the July 2008 data may have been the trigger to executed it.
As I understood his email the change in harmonics was done to further reduce the influence of seasonal variation (de-seasonalize) in the C02 record. If this is correct and the change does accomplish that, it is a good thing.
@Paddy
Since the Mauna Loa trend is pretty much in agreement with the global trend, I doubt they are ‘cooking the books’ with VOG.
the explanation is a good one.
if you discover an error you MUST find a solution for the WHOLE dataset.
doing anything else, would NOT work.
that it has a “dampening” effect on the wiggles sounds pretty logic to me. looks like many of those outliers were actually produced by missing data.
lesson to learn: don t get too excited over a single data point. even when he fits your believes…
Reply:Hey! I agree with sod. Time to buy a lottery ticket~charles the moderator
Like Dee, I also don’t smell a rat. I know to those who suspect all adjustments it seems odd that the big negative number caught Dr. Tans eye. But bear in mind, this data point was a major outlier relative to the linear fit I show above. Any experimentalists would notice the big excursion and investigate what might have happened.
The answer is straight forward enough.
That other changes were made suggests that this measurement program is a bit informal. But that happens sometimes. The degree of formality associated with measurements varies from program to program, and will often depend on the use of the data. As interested as climatologists and bloggers are in this data, they are not direct inputs to weather prediction, health or safety programs or anything else.
Remember: No one was physically injured as a result of the August CO2 blog kerfuffle.
It doesn’t seem like Dr. Tans did anything malicious or with nefarious intent. His explanation makes sense.
(But I’ll bet $1 he wouldnt have gone looking for a new algorithm if they had instead lost data from the 2nd half of July and the chart was spiking erroneously high. Any takers?)
“My objection goes to bias your honor.”
Ed Ried,
I think you understand the point I was making. Furthermore duplicate samples tested at an independent laboratory would certainly lend credibiliy to their data set and also avoid having lengthy periods with no data recorded. With regard to the subsequent adjustments to the graph I couldn’t agree with you more. As for why it took twenty days to get their instruments fixed well maybe the pack mule they use to carry the equipment up and down Mauna Loa was lame.
Frank L/ Denmark (07:45:57) :
And who believes that is just a coincidence. If the data was really in need of adjustment, you would not be able to tell the positive count from the negative count of adjustments. Nothing in nature is this one sided, and surely not data taken by any monitoring system. The odds are infinite that this is just fudging the facts.
So where is the raw data, without the adjustments. We are big boys, we can understand the need to try and make the picture clearer, we just demand to know the facts about what is going on. Honest science should be open science.
Has anyone heard of the Data Quality Act of 2001? It requires the federal government and it’s agencies and contractors to put all the science data in the public domain, unmolested … With no corrections and fudge I might add. Corrections can be applied, but must be documented. I wonder who this qualifies under DQA.
I would agree that the change is for the better during any kind of month that shows rapid variability. I don’t see anything statistically wrong with doing that. It clears the noise so that trends can be seen.
However, sometimes change in variability is an important statistic. Months that fly all over the graph may show trends in raw data variability that may be much more sensitive to climate change than steady as she goes months.
I would sure lurve to have the raw data for every moment of measure.
Both the explanation and the correction seems reasonable to me, except for one thing, that weird seesaw where every second month is corrected up and every second down. To me this means either that there is something wrong with the correction algorithm (easy to do when in a hurry), or that for some reason data is predominantly missing at the beginning of one month and at the end of the next. Do they by any chance take their equipment off-line for maintenance at two-month intervals?
Dee:
Thanks for the leg work and the level-headed interpretation. If you can ask a follow up question of Dr. Tan, perhaps you can ask him (?) what prompted the move form 4 to 6 harmonics that would at least help understand what issues they are trying to handle.
Thanks