Cores!
Guest post by David Middleton
A precious collection of ice cores from the Canadian Arctic has suffered a catastrophic meltdown. A freezer failure at a cold storage facility in Edmonton run by the University of Alberta (UA) caused 180 of the meter-long ice cylinders to melt, depriving scientists of some of the oldest records of climate change in Canada’s far north.
The 2 April failure left “pools of water all over the floor and steam in the room,” UA glaciologist Martin Sharp told ScienceInsider. “It was like a changing room in a swimming pool.”
The melted cores represented 12.8% of the collection, which held 1408 samples taken from across the Canadian Arctic. The cores hold air bubbles, dust grains, pollen, and other evidence that can provide crucial information about past climates and environments, and inform predictions about the future.
The storage facility is normally chilled to –37°C. But the equipment failure allowed temperatures to rise to 40°C, melting tens of thousands of years of history. Among the losses: some of the oldest ice cores from Mount Logan, a 5595-meter-high mountain in northern Canada. “We only lost 15 meters [of core], but because it was from the bottom of the core, that’s 16,000 years out of the 17,700 years that was originally represented,” Sharp says.
Scientists also lost 66 meters of core from Baffin Island’s Penny Ice Cap, which accounts for 22,000 years—a quarter of the record. That leaves “a gap for the oldest part, which is really the last glaciation before the warming that brought us into the present interglacial,” Sharp says.
[…]
Apparently the meltdown was due to two three malfunctions:
Investigation points to two malfunctions
An investigation into the freezer malfunction found fault with the cooling system. Specifically, the refrigeration chillers shut down due to “high head pressure” conditions. Essentially, the chillers were not able to reject their heat through the condenser water system—heat instead of cold circulated through the freezer.
Compounding matters, the system monitoring the freezer temperatures failed due to a database corruption. The freezer’s computer system was actually sending out alarm signals that the temperature was rising, but those signals never made it to the university’s service provider or the on-campus control centre.
In the short term, refrigeration technicians are monitoring the freezers through twice-daily checks, Sharman said. The computer database corruption was resolved by adding a second monitoring controller, which is now issuing real-time messaging updates every eight hours.
[…]
- The “chillers” circulated heat through the freezer.
- Database corruption prevented the alarm signals from reaching their destination.
- The failure to visually inspect the freezers on a daily basis before 1 & 2.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
There’s a strong temptation to say how convenient…..
I had to fiercely resist that temptation.
Database corruption…..wait for it……Russians
Sounds like a perfect opportunity for a little Field Work recoring the ice
Bryan A’s comment is on target. Just go out and recollect the cores.
Bryan A,
But the cores are irreplaceable because Griffy assures us that the Arctic glaciers have all melted.
Careful there, the Griffmeister is like old Lord Moldywart, you invoke his name and he magically appears
Bryan A, that was exactly my first thought. Even if the glaciers have melted a bit, that is only a few years lost. Fire up the coring machines and get some new cores.
PS, I wonder what power source the coring machines use?
Yah–a few too many coincident “failures” to be, uh, “coincidence.” Methinks we’re hiding more inconvenient truths . . .
First thought as well. If these cores were so important, then it is sheer incompetence to have let this melt occur.
In the real world, anything that important is not only monitored using multiple independent systems, but regular tests are made of all components to ensure that they are working correctly.
For example, you would throw a switch on the thermometers to cause then to send out alarm signals and it would be verified that all the proper authorities received a notification in a timely manner.
Dog ate my homework?
Lack of pride in the work environment.
It was sheer incompetence to not have redundancy in the cooling system and the monitoring system, Maybe even more incompetent to not have split the cores (lengthwise [timewise]) in tow and store them in two different facilities. Sort of like the CRU losing the raw data because they couldn’t afford a server. If you are spending tons of our (taxpayers’), at least show some sense and spend it wisely. But, of course, these are scientists, not engineers – never, ever consider the possibility of failure and provide for it.
Computers enable mass incompetence to happen and delving in and determining the reality of the situation takes too long so people let it slide:
– “My machine crashed.”
– “I got hacked.”
– “I can’t do that because I need an upgrade.”
– “Oh – that was autocorrect.”
– “There is no backup.”
– “The file got corrupted.”
– “I’m waiting for IT to fix it.”
– “Typo.”
– “I wrote that all by myself.”
Puts a whole new light on ‘the warm bias’ in data adjustment!!
Canadian incompetence on display
If they were under my control I’d have had sirens fitted. seriously, I would .. not just one either.
Incompetence?… I can just imagine the newly indoctrinated, scientifically trained climate scientists getting their new ice cores. Get to the top of a high, snow covered mountain, find a nice slope of appx. 45 degrees, start drilling down the side of the mountain slope at 45 degrees and there’s yer new ice core.
New Democrat Party (NDP) socialist/commie gov’t elected less than 2 years ago has a goal of outright destroying Alberta. Very similar and maybe even worse than South Oz…a real basket case of governance.
This is the Gov’t that had just cut rapid initial attack for forest fires after being elected, and then Ft. McMurray burns down and they try and blame it on climate change. Sad.
Ron Williams says “This is the Gov’t that had just cut rapid initial attack for forest fires after being elected”
I disagree with Alberta NDP on nearly everything but firefighting money ALWAYS comes out of emergency budget due to the wild swings in resources required from year to year.. Their claims that fires had anything to do with climate are as you describe it, sad. But you can not claim that budget shuffling in any way impeded fighting the Ft. Mac fire. The real culprit was dry, warm winds and poor fuel load management. After that, all the money in the world cant stop a fire like that as any seasoned fire fighter will tell you.
Sorry Dave in Canmore, you are outright wrong on this and ask why you are providing cover for NDP in Alberta? A little biased are we? This story below was published April 19th, 2016, and the Fort Mac fire started May 1st/16 causing $3.58 billion (insured damages); and $9.5 billion (direct and indirect costs) which was almost equal to the entire budget deficit of over $10 Billion this fiscal year. It is absolutely despicable that any Gov’t would cut budgets to fire fighting, but this is what happened. Don’t lie for the NDP Dave. The result of the cutbacks caused the confusion that led to a momentary delay when they could have had a chance at knocking that fire down.
http://edmontonjournal.com/news/local-news/alberta-cuts-nearly-15-million-from-wildfire-management-budget
Furthermore, it was absolutely declared as a human caused fire, with lightening having been completely ruled out. Some theorize it was a wildfire 15 Km out of town that was the cause of the fire. But a second fire was already burning within town in the north end near a garbage dump, and resources WERE NOT immediately dispatched to it, but were dispatched to the the fire burning 15 Km out of town.
http://www.theglobeandmail.com/news/national/fort-mcmurray-wildfire-most-likely-human-caused-alberta-senior-wildfire-manager-says/article30279836/
“Complicating matters was a second fire burning at the same time within Fort McMurray near an industrial area in the city’s north end. That fire was moving up a hill towards a row of houses on May 1. Mr. Spring says that fire crews had to decide which to tackle first: the fire in a remote area south of town or the one bearing down on homes.
Video from May 1 shows tankers dropping fire retardant in Fort McMurray as helicopters poured buckets of water. Crews had decided to confront the blaze burning in the industrial north end–because it was first spotted within the city it doesn’t have a wildfire name like MWF-009.
“The choice had to be made between fire 009 and that second fire headed towards houses. Five out of five times anyone would choose to go after the second fire,” Mr. Spring said.
His company wasn’t asked to dispatch a helicopter to MWF-009 on May 1. Within two hours that fire had grown to 60 hectares, fed by strong winds. The first evacuation notices went out before dusk that evening.
Fort McMurray’s 80,000 residents were evacuated two days later, on May 3.”
Gosh! Surely you not hinting at some sort of conspiracy rather than cock-up! That would be a foul slander on scientists who we know are of the highest integrity. I am shocked – shocked that any such suggestion could even be conceived. let alone expressed in public. I have never heard of
(Continued on page 94)
It’s worse than we thought. Global warming is now causing ice cores to melt inside of freezers. The warming has truly become “global.” We will have to move freezers off planet if we want to keep things frozen.
Problem with temps noted. 37C is way above freezing.
David,
I’m guilty of missing the minus sign and stand corrected.
I can tell you those things make researchers go more than Hmmm. Freezer failures were a bain where I was facility manager for a state college with research facilities. The problem was that the low bid got the sale, so we had cheap stuff.
They should consider themselves lucky that ice cores merely melt and don’t get rancid like tissue in a medical lab freezer.
The biggest problem is that they outlawed the only refrigerants that work in a cascading cryo-system without reaching ridiculously high pressures. newer refrigeration equipment is much more prone to freon leak-down failures.
They had better get out and cut down some old trees and start counting the rings to make up for the lost data.
Just go back to the site and get another core, I’m sure the ice is till there…
I saw on Ebay that supplies were limited. Better act now before they’re all gone!
I’m a software developer. It simply is not smart to rely on software for something this important, if there are other options.
Software’s greatest risk of malfunction occurs when something unusual happens – the code to handle unusual situations is rarely executed and difficult to test, so it is the part of the system most likely to harbour undetected defects.
Also lack of backups and monitoring of critical systems.
“The first shuttle had 3 computers that did all flight calculations.”
Flight Control Software is still written that way. A single specification, but three separate programs written by three separate software teams, if possible using three different compilers.
When I worked at Rockwell/Collins we supported 3 to 5 computers, however they were all our computers and all used the same software.
It’s not that hard to test, all you need is a proper test rig. Either the hardware is set up to use a variable resistor to represent the temperature measuring unit (or voltage input if that is the kind you are using).
Then switches and lights to represent the other inputs and outputs of the system. Then you can completely exercise even the most unlikely of scenarios.
I’ve built systems like that this to emulate nuclear power plants. A single thermometer for a single room should be trivial.
Even without that, you can put the sensor in a box, use dry ice to cool it and a heater to warm it. Then go ahead and test every scenario in your test suite.
Mark,
Yes, I built a small model of a grain mill once that I was rewriting the control software for (because the original developer was incompetent). All the sensors were represented with simple switches and the motors/actuators with LEDs. This allowed me to test every conceivable failure and operating condition. The new software not only was reliable, but also helped find faults with the hardware (bad relays, bypass diodes, etc.).
The kind of software monitoring mentioned in the article is quite simple by comparison, and not that hard to test, as you mentioned. Testing the complete system from time to time would have uncovered the corrupted database and probably saved the ice cores. There really is no excuse for this level of incompetence. The solutions for these kinds of problems are well known and commonly implemented.
Perhaps having an analogue back-up alarm independent of any digital higher tech, would at least alert the problem to those responsible for the maintenance. If it had its own redundant back-up analogue alarm separately with its own sensor etc, in conjunction with the main digital alarms, then the analogue alarm bells would have been ringing the problem regardless of the high tech monitoring equipment. Sort of like having an analogue volt meter wired independently into a electrical panel for a visual back-up of a digital meter. (in case the display burns out) Those damn cosmic rays…
The first shuttle had 3 computers that did all flight calculations. The calculations were compared against each other before any decision was made. Two of the computers were made by one manufacturer and the 3rd was made by a different one. This helped to ensure that all three computers wouldn’t make the same mistake because they all had the same software bugs.
Eric,
I come form an aviation background with computer-controlled, FBW / FBL, uninhabited aricraft. We ALWAYS include software in our flight control systems to “handle the unusual situations” – at least all the ones we can anticipate. That’s the purpose of safety cases. You are right, it is difficult to test, but it is better than a smoking hole in the ground.
And I have to disagree – unless you are working in AI with fuzzy logic, software never malfunctions – give it the same inputs, you get the same outputs. Now, in my experience, failures owing to hardware inadequacies have often been blamed on “software glitches”. Ain’t fair. Also, of course, if you chosen to use a computer language known for inadequate design, such as doing garbage collection poorly, or not cleaning up unused allocations, or handling thrown exceptions poorly, then maybe you can call that software failure, as it is a failure in the language choice / design. But there are languages out there that can be used and are better designed / implemented.
Now, having disagree, it is merely a gentleman’s disagreement – please keep up your articles.
Jim
Tge fact is that random errors can and do occur. Most are probably the result of stray cosmic particles, but we obviously cannot be certain. Others are, as you say, the results of programming errors, and as software becomes increasingly complex, they are more difficult to diagnose. As we start getting AI to develop software, this will get much worse.
The best solution is multiple identical redundant systems, allowing a consensus. I believe this is what is done in space missions. This cannot rule out hardware faults, however, and as I understand it, the Intel 386 chip is the only verified bug-free hardware in common use.
The punchline to the joke is “I think we should push the car back up the road and see if it happens again.”
All good Jim. I doubt aviation QA standards were being applied in this case 😉
While working at a large avionics company, I had to write the software that performed the power on self tests for the error detection circuitry for the memory modules. There was a trick that allowed us to deactivate the circuitry long enough to write bad data to the memory locations, then when we read it back, it would trip the circuitry.
To the best of our ability, every circuit in the hardware was validated each time the system was powered up.
PS: If you think Unix takes a long time to boot up, you should have seen these systems.
As a fellow s/w developer, I tend to agree with you in general. But s/w specifically designed to monitor temperature not being tested to detect high temperatures or complete loss of data? Kinda hard to believe.
Yes, it is like backing up data. It happens every night without fail, but it’s only when you lose the original data, and try to restore that you find that the backup process had a bug.
I thought that you should always do a test restore of the backup to make sure that the backup was valid.
Eric, this is exactly the sort of thing fire alarm systems handle all the time with great reliability. Commercial fire alarms that report to a central station send a test signal every day. Under the latest code a test signal will be required 4x a day. There is no excuse for not using some similarly reliable system to monitor the situation.
Well it looks like they will just have to make it up as they go along then.
so normal procedure then
Keep in mind – the ice is still there, albeit not in the warmed up freezer, but Baffin island is still there and the bottom of the ice is still reachable! All is not “lost” – just what had been archived!
They better get to drilling more cores… ASAP!
https://wattsupwiththat.com/2017/03/22/oh-noes-last-remnant-of-ancient-north-american-ice-sheet-on-track-to-vanish/
My thoughts, too, tom … it’s not as if those glaciers are going anywhere for the next few thousand years, at least … particularly the ice at the bottom of the coreholes. All it takes is time and money to recollect.
They can just drill, baby drill!
Inexcusable! System checks, eyes on monitoring and backup must be in place. To claim a perfect storm of failures is to show incompetent planning and poor execution of preventive protocols. Somebody didn’t do their FMEA!!!!
The fact that the database corruption had not been detected shows that they were not running periodic systems checks.
Database corruption??? … Red flag here … Seems this is a ‘it’s nobody’s fault excuse’ since the consequences would likely result in someone getting terminated.
Alternate scenario. If somebody did something stupid like use their own credentials when setting up the system and said person is no longer with the university and said persons credentials were disabled, well … The system would no longer have valid credentials to communicate with the database. I have seen this scenario played out a million times. The person either doesn’t understand or decided’s they will create a service account later but never gets around to it. If the said person’s supervisor is still around it may not be career enhancing to be responsible for loosing such a valuable asset.
I’m a software developer too. I have a hard time believing “database corruption“. Especially database corruption going undetected.
It is because of all the Donald Trump cut backs
Trump controls the budget of the University of Alberta?
The man has more influence than I imagined.
I’m expecting more data/evidence destruction during Trump’s administration, to be blamed on budgets etc. while hiding the decline, and other lies.
Why not just freeze them again?
Do you have any idea how hard it is to get 16000 year old Freon to re-freeze this with?
How convenient the cores were only 16,000 yrs old. If they were 32,000 yrs old you would find that 32,000 yr old Freon is IMPOSSIBLE!
Would that be ‘flattening the data’?
True Homogenization
The ice is meaningless. It’s the captured air holes. Once the air in the holes got out, refreezing is pointless.
Sarcasm is best served without a /SARC tag… 😉
Bottle the water and sell it to those who would use water to ruin a good single malt.
Maybe it’s all a coverup for a bunch of grad assistants stealing the ice for mixed drinks, because, “16,000 year old ice, man!”
How does anyone know the water on the floor is REALLY from the ice?
if you’re going to ruin 100 year old Scotch, you may as well use 17,000 year old water. it at least SOUNDS classy. ^_^
Pour the water into ice cube trays, and see how you go.
Let’s get a book going on how long it will take before this is claimed to be Trump destroying climate data…
” Christ, 7 years of college down the drain.”
Thanks for that memory…
Comment of the day.
Database corruption? Not again.
“depriving scientists of some of the oldest records of climate change [16,000 years out of the 17,700 years] in Canada’s far north.
Hahaha. Everybody knows there was no climate change before 1850.
The records are still there as tomwsy1 so aptly pointed out. Just no longer in the laboratory.
But without the original data source they can’t verify that their adjustments go in the proper directions
Bryan A, those old records have been sufficiently disproved by the latest methods of consensus that people could not detect or record the proper temperatures before computers and satellites were in place.
Well it looks like they are just going to have to go 100% with sophisticated computer models of the ice now.
Well, some entrepreneurs used iceberg water to make signature vodka…
Just sayin’
Am I reading this right? The article has ice floating around that’s 16000 yesrs old? Meaning the artic has never been ice free this entire interglacial?
Doh! The artic not article
These are Ice Cores on Land Fast ice
There is very little Arctic Ocean ice older than 5 years (about 3% of total in 2012)
Mount Logan is the highest mountain in the Canadian Arctic. Not sea ice. Glacier ice.
Not really in the Arctic per say, but in the southern Yukon at 60 degrees, 34 minutes North, sort of close to Alaska. Ok, splitting hairs…
How the heck does a freezer become a sauna? When my freezer fails it eventually goes to 0C not 40C.
Mine goes to room temperature, about 30C where I am.
Their explanation is wrong and shows no understanding of how these systems work. The most likely cause is the tripping of the high pressure safeties as stated ( for any number of reasons), with the result being the constant input of heat from the evaporator motors which may run continuously even though the compressors are not. Still some questions I would have about that. 25 years as a journeyman refrigeration tech , system designer and even salesman (under duress). With a well insulated room, even lighting can drive up temps quite quickly and to pretty high temps. Lots of big evap fans running? – X 10.
Feedbacks
sigh /sarc
So what’s the problem – its just the same as Kriging, averaging or hommorogerising the data.
Saves a muppet from getting it wrong in Excel. and I’ve always said that an analog computer will gives better answers than any digital one.
Climate Science finally moves on
Top of the range smart freezer’s software hacked by Russian hackers loitering about Canada?
(just wondering)
What’s the medical concept? When you hear hoofbeats, don’t go looking for zebras? Most likely garden variety incompetence. Most mechanical engineers know next to nothing about refrigeration, most University purchasing departments know nothing about hiring competent firms and have to take the lowest bidder with at least superficial qualification. I also worked a lot with controls companies, ALL of which were generally lacking in knowledge of what they were controlling and how those components worked and worked together. I saw dozens of screw ups like this, just none so spectacular.
The next university department that shells out coin for a “top of the range” freezer will be the first. Gotta cover those “fees” for university “overhead” related to the program. Forget about top-of-the-line equipment.
No problem in line with normal procedure in climate ‘science ‘ they will just ‘model the problem away’ and of course they can always refer to the magic tress. Heck the science is so ‘settled ‘ there is hardly any need to collect ice cores now , as they ‘know ‘ the result then want .
In any other field of science this would be assumed to be incompetence.
But Climatology is known for it’s convenient accidents. Remember that poor tree on the beach?
After all, handing over data only let’s people find out when you are wrong, to paraphrase the scandalous Climategate emails.
MSM Report
“Unprecedented warming results in Catastrophic melting of century old Canadian arctic ice”
+1 Major headline.
The Russians hacked UA.
Rachel Notley, Premier of Alberta, with her Green Agenda and buddies (likely left-wing so-called intellectuals @ur momisugly U. of A.) might be asked to account for this inexcusable event. Are these data not as important as Dead Sea Scrolls (and the like) and what opprobrium would be visited on the so-called guardians of the Archive in *that* circumstance?