Climate Audit is down

While we are on the subject of hardware failure (such as has hit the DMSP satellite NSIDC and Cryosphere Today use) Climate Audit is down due to a file system or HD error. It happens. I’m on my way to the Colo (90 miles away) to effect a repair. Comments may be delayed for a few hours if other moderators aren’t online.

UPDATE: 5:30PM

The Climate Audit server is in fact RAIDed, I built it that way for just such an emergency, but some corrupted data was written before the one disk of the array failed. Since I could not stay at the CoLo all day, I’ve brought the CA server to my office for repairs. Hopefully the RAID rebuild goes smoothly (it takes several hours) and I’ll be able to repair the problem areas. Hard drives were both new, RAID quality units, with 3 year warranty. One failed 1.5 years into the warranty – that’s Murphy for ya.

Wish me luck, otherwise I have to rebuild from scratch and restore from backups which is also a chore.

Just for those who like to know about hardware, here is what Climate Audit runs on:

3.4 GHz Intel Pentium D CPU

2 GB ECC DDR2 400 RAM

RAID1 Dual Western Digital 250GB SATAII drives with 16MB cache ram

Running Linux with WordPress in LAMP config

1u Intel Server enclosure like this one:

intel-1u-1325

sr1325tp1

Thanks to those who hit the tip jar.

UPDATE: 8:30PM

One hard drive of the RAID failed. Now before you panic let me say I anticipated this (but like 2 years from now) and this was a RAIDed system with two drives setup to mirror. Normally when one drive fails, I can unplug the other and reboot the system and it will come up and run on the one, then I can install a new second drive and rebuild the RAID, and off we go.

I’ve done that dozens of times in my own systems. It is why I built the CA server the way I did. It is an identical server to 15 others I’m running here.

But for some reason known only to Murphy, this time when the system failed sometime last night, it appears it wrote corrupted data to the “good” drive before the full hardware failure. So at the moment the system is unbootable.

The good news is that most everything should be recoverable, but it takes time. If I can’t repair the boot sector on the good drive, then we have to rebuild two new drives from scratch, mount the one good drive, and pull files over. Though I don’t know just yet how much corruption there is and how much of it can be fixed.

The annoying thing is that these mirrored Western Digital 250GB drives had only 1.5 years on them, and less that 10% full. They were brand new when I purchased and installed them specifically for CA. They have a 3 year warranty. They’ve been in a temperature controlled and dust controlled environment at the CoLo. For one to totally fail now is quite the surprise. I wasn’t all that worried about regular backups due to the RAID mirroring, now the RAID fails with the drive.

I was able to rebuild the RAID, but it appears that the boot sector is corrupted. This will require a mount from a CDROM boot and fix the file system and make copies that way.

Best laid plans….

I anticipate it will be Monday evening before CA is back up and running.

0 0 votes
Article Rating

Discover more from Watts Up With That?

Subscribe to get the latest posts sent to your email.

150 Comments
Inline Feedbacks
View all comments
pwl
February 23, 2009 2:22 pm

The case you’re using looks rather cramped. Make sure that you have enough or way too many fans cooling everything down. Unlike the climate change crisis discussions controlling the climate in a computer case is a lot more straight forward. You essentially want to create a near ice age in the case especially around the cpu, ram, and of course the hard drives and power supply. You’d definitely want to put small thin fans directly on the hard drives if there is room in the case for the fans and the airflow. You’ve gotta move that heat out of there. Another defense is collecting weather data on the climate change in the case. Linux can be set up with system health monitors that track the temperature of all the components that have temperature data. Adding your own thermometers may be a good idea in a tight case like that; place them strategically near or on critical components and in the intake and outgoing air flows. This allows you to monitor the external temperatures of the server room and the total heat being generated by the case in real time. Sometimes adding additional heat vents near or above hot spots to improve the heat dissipation can make a huge difference – get the heat out the new hole rather than move it through the case. This of course assumes that you box isn’t sandwiched in between other cases generating tons of heat themselves. In our server room we keep a good air gap between all cases – no cases ever touch. There are also large fans moving the air around the server room away from the boxes by directly blowing on them.
I prefer larger cases with plenty of fans (at least 6 120mm fans) and ventilation holes. They also can hold more drives so that you can have a live backup set of drives on an independent RAID unit, or you can have that in a second redundant server box (but that tends to increase your colo fees).
Controlling micro climate change is difficult. Leaving hot spots is very likely to lead to premature failures.

pwl
February 23, 2009 2:41 pm

On a personal note, we in Canada wouldn’t mind it being warmer as that would save us money heading to the tropics in the winter months. The main downside to a warmer climate for us will be learning how to make homes out of wood and brick rather than ice. 😉
As for obstructionists, that’s a weird term and meaningless.
(1) If the alarmists are correct, and that’s a huge whopping if, then there is little that anyone can do as all the solutions need to be done on a planetary terraforming scale. You know, like they did in Aliens II or in Total Recall on Mars.
(2) If the alarmists are incorrect, more likely it seems as time goes on, then there is nothing to obstruct.
(3) It’s crucially important to know which because we’d not want to alter our environment intentionally when we are not the cause.
So, yes, I’ve replaced the old light bulbs in my home and save a huge amount of cashola and, well hydroelectric generated power. Yes, I drive less now the last few years, usually only a few times a week. Yes I even sometimes wear a wool tuke (winter hat) while at home and keep the heat set lower. So I’m saving energy and have lowered by consumption of goods and fuel hydrocarbons. It’s not only extreme environmentalists aka the awg alarmists that like living hear on Earth, our mutual precious, us normal humans (the ignorant masse) like it here too.
Would somebody turn up the heat in Canada already? Brrr… these damned ice ages…

Gerald Machnee
Reply to  pwl
February 23, 2009 3:38 pm

Come on, it was not that cold. Winnipeg had only the third coldest December-January in history. We just forgot to burn the coal. The sun was shining, so how can you beat that?

Joe Black
Reply to  pwl
February 23, 2009 4:16 pm

Those who live north of Canada (Check a map) agree.

pwl
Reply to  Joe Black
February 23, 2009 5:46 pm

Meet Joe Black who lives to the North of Canada. (Ok, maybe you don’t live there … It’s more fun if you do though so… )
Ok, Joe where do you live? Iceland? Alaska? While bits of those are north of most of Canada the only land that is North of Canada is the northern tip of Greenland, and only by a the tiniest of smidgens.
So you either live on the north most tip of Greenland or you’re living on the ice pack somewhere?
Say high to Santa for us, unless you really are Santa?
Oh, you could be living on a ship or nuclear submarine?
Or maybe you live here on the northern most island? But it’s not inhabited. http://en.wikipedia.org/wiki/Kaffeklubben_Island
Canada’s most northern community is here: http://www.arcticcircle.ca/Baffin/Grise/ . Well actually Alert Bay has the distinction of being the most northerly community anywhere (on land) here: http://en.wikipedia.org/wiki/Alert,_Nunavut . And Alert Bay is almost at the Northern most tip of Canadian land. Of course it only has five permanent(ly insane) and many temporary(ily insane) inhabitants.
“Alert lies just 817 km (508 mi) from the North Pole. Alert is located 12 km (7.5 mi) west of Cape Sheridan, the northeastern tip of Ellesmere Island.”
“Alert has a polar climate. This means it is very cold and, on average, has snow cover for 10 months of the year. The warmest month, July, has an average temperature of just 3.3 °C (37.9 °F). The climate type also means that Alert is very dry, averaging only 153.8 mm (6.06 in) of precipitation per year. Most of the precipitation is snow and occurs during the months of July, August and September. On average there is 16.1 mm (0.63 in) of rain which occurs between June and September. Alert sees very little snow during the rest of the year.”
No I don’t live there, it’s a bit too cold for me. I did live in Edmonton during eight of it’s coldest years in the 1970s – they don’t close the schools until it’s minus 45c! Brrrr and Grrrr, BrrGrrrrr… at them… since it hovered around -30c to -40c for very long periods – months. I had to walk up hill to school in -40c and uphill back home in -40c with a wind blowing in my face. BrrGrrrr… A -20c day after that was t-shirt weather, seriously we’d play outside in our t-shirts throwing nice packing snow in -20c weather! It is one of my sweeter memories of Edmonton.
Here are the rankings by country.
1) Greenland, Kaffeklubben Island, 83°40’N
2) Canada,Cape Columbia, Ellesmere Island, Nunavut, 83°06’N
3) Russia, Cape Fligeli, Rudolf Island, Franz Josef Land, Arkhangelsk Oblast, 81°51’N
4) Norway, Rossøya, Svalbard, 80°49’N
5) United States, Point Barrow, Alaska, 71°23’N
6) Finland, Nuorgam, Utsjoki, Lapland, 70°05’N
7) Sweden, Treriksröset, Kiruna, Lapland, 69°04’N
8) Iceland, Kolbeinsey, Eyjafjarðarsýsla, 67°08’N
9) Faroe Islands, Cape Enniberg, 62°23’N
10) United Kingdom, Out Stack, Shetland Islands, 60°51’N
What impresses me is that part of the United Kingdom is North of 60! Wow. Canada had a tv show celebrating those who live way up there North of 60: http://en.wikipedia.org/wiki/North_of_60
What is your lat and long Joe Black, or at least of the place you were speaking of?
Well I checked the map and found only one place, an uninhabited island that is the most northern tip of Greenland. At 817 km from the North Pole, Alert Bay, Canada is the winner as it actually has people living on land! They might be few and insane but they live there. Shivers just thinking about it.
Who can beat that? 😉

Larry Sheldon
Reply to  Joe Black
February 23, 2009 8:24 pm

I’ve been north of Canada and didn’t leave the USA.
On other trips I’ve driven the big truck south into Canada.

pwl
Reply to  Joe Black
February 23, 2009 11:42 pm

Larry, it sounds like you were driving a big rig on the ice while remaining within the borders of the USA 200 mile limit. In that case there would still be part of Canada north of the USA! (Unless your on a submarine or some ship that is considered US territory).
What is an example lat long?

Joe Black
Reply to  Joe Black
February 24, 2009 2:39 pm

Well, It looks like there are places in the following states that are directly North of (some part of) Canada:
Alaska, Washington, Minnesota, Michigan, New York, New Hampshire & Maine.

GP
February 23, 2009 3:30 pm

Walt Bennet,
You know I rather suspect that our children (mine have just recently graduated) and their children (if they choose to have any) may indeed curse us in the future but possibly more for the carbon output reducing state of the economy that we seem to be sprinting towards than for any issues of failing to manage the unmanagable.
It is amazing how long things take to build up – civilizations, warm periods of beneficial climate – and how quickly they can be destroyed – humans chasing unrealistic politically created objectives, ice ages.
At this rate all those who call for an 80% reduction in CO2 output by 2050 will see it by 2015. If that happens things will get very ugly indeed, for all of us.
On the plus side, for you at least, one should expect to see the your perfect scenario for saving the planet in its full glory. Heck of a way to forget the banking crisis though.

Reply to  GP
February 24, 2009 7:18 am

GP,
I have no scenario for saving the planet. Hansen has presented all the evidence we need to know that the planet will be ice free. It won’t care, it will carry on just fine.
Life as we know it will cease to exist.

henry
February 23, 2009 3:37 pm

Walt:
“The thanks will go to those who were alive today and chose to suspend laws of physics only where AGW was concerned, who were capable of rational thought except where AGW was concerned, and who learned and loved science except where AGW was concerned.”
The blame, then, must go to those who were alive today and chose to suspend debate on the science where AGW was concerned, who were only capable of irrrational thought where AGW was concerned, and who hid the results of their “scientific” efforts where AGW was concerned.
And as to this statement:
“As Hansen puts it, man is now in charge of climate. The only problem is, he has no idea what he is doing. Thus he is utterly blind to the consequences of his own actions.”
I agree. Hansen DOESN’T know what he is doing. And he IS utterly blind to the consequences of his own actions.

pwl
Reply to  henry
February 23, 2009 5:59 pm

Not only that, think of all the scary scenarios of terraforming Earth, the one place we can exist in the known universe! It’s simply a bad idea terraforming the only place you can exist without knowing the consequences of our actions!
By “simply a bad idea” I mean probable suicide.
Scientists Rank Global Cooling Hacks
http://blog.wired.com/wiredscience/2009/01/georank.html
Oh gee, they are actually planning to terraform the Earth without knowing WTF they are doing… this could start actual wars – seriously.
This is why it’s important that the politicians get out of the way and that everyone let the debate rage on till the science is rock solid (if that’s even possible) and proven.
“Those who fear questions of their science are no longer scientists, they are politicians and activists. Question their science even more intensely!” – pwl
http://PathsToKnowledge.wordpress.com

Reply to  henry
February 24, 2009 7:17 am

Henry,
This will be the only time in your life that I address you directly.
Your dismissal of Dr. Hansen is all I need to know about you. Those who attempt to undermine the good work of knowledgeable people are the least relevant of all the hyenas barking from the sidelines.
No offense, but you are utterly useless. If there are enough of you – and there may be – then you will take the species down with you.

pjm
February 23, 2009 5:42 pm

Here in Southern Australia our biggest problem is not temperature (although there has been record-breaking heat in the current summer) but lack of rainfall. It might be a climate shift, or it might not. If it is it might be due to GW, perhaps even AGW. We don’t know, and I have seen enough to be mistrustful of anybody saying they do know.
What happens if we reduce our CO2 output? Some predict economic disaster, with no climatic effect. Some predict it will trigger the next ice age. Interestingly, I do not remember any predictions from the AGW people about what will happen if we get CO2 back to (say) 1950’s levels – I imagine they think the climate will return to 1950’s patterns. Or perhaps they know they don’t know.

February 23, 2009 6:50 pm

The January 2009 minimum temperature average at melbourne central was the coldest for 5 years. The average for that month was the coldest for 4 years.
Strange how a few abnormal days make it feel like something permanent has happened.

John G. Bell
February 23, 2009 7:23 pm

I jogged around Town Lake on Austin’s hike and bike trail in 113F weather and loved it. You sweat like crazy but the humidity was low and it was great! People are amazing in their ability to adapt to and even enjoy a wide range of climates. I used to jog home from school a couple miles on snow and ice when I was a kid in Maryland and missed the bus. It strikes me that we are talking about a couple of degrees change over a hundred years and that not even certain. Don’t we have something more important to talk about than how to solve a fantasy problem.
Climate change is real, is natural, and is something that humans are particularly well adapted to resist and animals and plants have dealt with for millions of years.

Reply to  John G. Bell
February 24, 2009 7:13 am

John G. Bell,
Please provide for us your no-doubt riveting account of the last time nature spewed millions of years of carbon into the air and water, and then of course let us know what was the result of that.
(P.S. It has actually happened.)

Carrick
Reply to  Walt Bennett
February 24, 2009 8:19 am

nature spewed millions of years of carbon
What the heck is that supposed to mean?

hswiseman
February 23, 2009 8:00 pm

Personally, I prefer big box floor sitters for servers, as you can jam them with drives (including a nice live spare) and fans and huge power supplies that never come near their limits. Of course a COLO usually offers racks only but 1U is pretty tight fit for what a single server solution needs. Raid 5 can take more than 3 drives (3 is minimum), and is the easiest raid config to add capacity to. Also, memory has gotten so cheap that it should be maxed out to what ever the OS/MB will take.

hswiseman
February 23, 2009 8:02 pm

CA is Up in MA on Comcast. Can’t speak for any other DNS. Thanks Greatly AW.

Scott Gibson
February 23, 2009 8:18 pm

Climate Audit is back up!

Larry Sheldon
February 23, 2009 8:20 pm

It works.
Good job, people.
Thankyou.

Gerald Machnee
February 23, 2009 8:28 pm

RE: pwl (17:46:41) :
Just a little correction. Alert is in Nunavut on Ellesmere island. Been there.
Alert Bay is off Vancouver, Island in British Columbia. See the site below. Have not been there.
http://www.alertbay.com/whereis.htm

pwl
Reply to  Gerald Machnee
February 23, 2009 11:38 pm

I stand corrected. You are correct Gerald. Thanks for correcting me on that.
So what is Alert like? Not much eh?
What were you doing up there? For fun? Just living? Family? Science? Military? Commercial?

Gerald Machnee
Reply to  pwl
February 24, 2009 6:51 am

I worked for Environment Canada at Resolute – got a ride to Alert on resupply trip. Got to see the elements, ice, glaciers, etc from the air and ground.

February 23, 2009 10:14 pm

Anthony, curious – what server OS are you running?
REPLY: Ubuntu FF

pwl
February 24, 2009 7:26 pm

Gerald Machnee, what is your take on the whole climate debate?

Gerald Machnee
February 25, 2009 6:36 pm

pwl (19:26:08) :
Re Your question on the debate. I am reading a lot now.
Many moons ago as a member of an organization we showed the public a film on the coming ice age. It was based on ice cores and a few other things. I would like to find that film now. At that time we showed the film but I had not looked into warming or cooling one way or another. And the Canadian government was more concerned with cooling than warming at that time(70’s). National Geographic had an article in 1976 that was sort of split on the issue.
So my point now is that I want to see some real calculations and measurements. All we have is two graphs of temperature and CO2, but no direct proof of causation.
So I have the two questions a) Where are the measurements of how much warming or cooling is caused by CO2 and b) Where is the engineering quality calculation of how much heating or cooling a doubling of CO2 will cause? The second question is easily recognizable from Steve McIntyre at CA.
So we have a lot of arm waving, but two important questions have not been addressed by IPCC.

Gerald Machnee
February 25, 2009 6:43 pm

pwl (19:26:08) :
In my response I missed one point. The big danger to mankind is not warming due to greenhouse gases but pollution of the land, water, air and our food. CO2 is not a pollutant. We are in greater danger of doing away with ourselves but that is not a topic for these blogs.

Gerald Machnee
February 25, 2009 6:50 pm

Walt Bennett (07:23:19) :
There is no point in me replying in reference to those terms. I gave you my question and I stated one more in my reply to pwl. You or anyone else has not responded to those. I and others do not understand why you feel so strongly about being an echo for Hansen. Check David Archibald’s paper on how the effect of CO2 decreases exponentially as it increases.

February 26, 2009 5:48 pm

I wonder if anyone else has asked this: Why all the talk about CO2 output? Why, if CO2 output is bad, don’t we develop ways to capture CO2 out of the atmosphere? If we know how much we put out each day, why can’t we develop capture stations to capture that much out of the atmosphere each day?
That would develop jobs, instead of killing them, wouldn’t it?
And, what harm would it do?

February 26, 2009 6:06 pm

Richard:
“…if CO2 output is bad…”
See, Richard, that’s where the misunderstanding comes in. CO2 is beneficial, it is not “bad.” And more CO2 is desirable.
Once you begin thinking correctly, everything falls into place.
You think like you do because you’ve been bombarded with “CO2 is bad” propaganda. The people putting out that propaganda have a vested financial interest in people believing what they’re selling.
CO2 is beneficial plant food, and when CO2 levels rise, plants benefit. When plants benefit, animals, including humans, benefit.
And finally: as CO2 levels rise, the planet is not warming. Therefore, the “global warming” scare stories are falsified. QED.

February 26, 2009 6:25 pm

Hey Smokey,
Sorry, but I’m not thinking that CO2 output is bad. I said, if CO2 output is bad, what about capturing it out of the atmosphere? It’s a question, not a statement of fact.
I am a computer and cable guy who once took a meteorology class in college, not a climatologist. I’m on the same side as the scientists who are willing to ask the questions. All the questions. Even at the risk of being compared with a “holocaust denier”.
Science, as I learned it in High School, is the collection and classification of facts. The hypothesis is a way of developing tests to determine where facts lead, not a way of stopping debate.
Anyway, I think my question goes really to rethinking the debate: Is stopping the “pollution” the only way of addressing the “problem”? Isn’t removing it also a valid way of addressing it? And wouldn’t it be a better way? What harm could it do?
Richard

February 26, 2009 6:39 pm

Sorry Richard, I must have mis-read your post.
CO2 is already captured out of the atmosphere by increased plant growth. This site had an article showing exactly that: click
Plants are already busy efficiently removing “excess” CO2. We need do nothing.
In addition, anything we do to remove CO2 from the atmosphere will be stupendously expensive and counter-productive. “Sequestering” CO2 underground has to be the most monumentally stupid idea to come down the pike in this whole sorry debate. We would be better off paying people to dig 10X10X10 foot holes in the desert, then moving those holes twenty feet north every six weeks until the holes converge on the geographic North Pole and provide a large access hole for the flying saucers’ easy ingress and egress.
CO2 is NOT a pollutant! And the harm done by treating CO2 as a pollutant will be done to your personal bank account, when $Trillions are wasted trying to fix a non-problem.
[Richard, you should post on a current thread. You’ll get more replies that way, and you can decide among the different points of view presented. I wouldn’t have responded, but I was looking for something else and noticed your post. If you wish to carry on this conversation, I’ll be on the current page.]

1 3 4 5