While we are on the subject of hardware failure (such as has hit the DMSP satellite NSIDC and Cryosphere Today use) Climate Audit is down due to a file system or HD error. It happens. I’m on my way to the Colo (90 miles away) to effect a repair. Comments may be delayed for a few hours if other moderators aren’t online.
UPDATE: 5:30PM
The Climate Audit server is in fact RAIDed, I built it that way for just such an emergency, but some corrupted data was written before the one disk of the array failed. Since I could not stay at the CoLo all day, I’ve brought the CA server to my office for repairs. Hopefully the RAID rebuild goes smoothly (it takes several hours) and I’ll be able to repair the problem areas. Hard drives were both new, RAID quality units, with 3 year warranty. One failed 1.5 years into the warranty – that’s Murphy for ya.
Wish me luck, otherwise I have to rebuild from scratch and restore from backups which is also a chore.
Just for those who like to know about hardware, here is what Climate Audit runs on:
3.4 GHz Intel Pentium D CPU
2 GB ECC DDR2 400 RAM
RAID1 Dual Western Digital 250GB SATAII drives with 16MB cache ram
Running Linux with WordPress in LAMP config
1u Intel Server enclosure like this one:
Thanks to those who hit the tip jar.
UPDATE: 8:30PM
One hard drive of the RAID failed. Now before you panic let me say I anticipated this (but like 2 years from now) and this was a RAIDed system with two drives setup to mirror. Normally when one drive fails, I can unplug the other and reboot the system and it will come up and run on the one, then I can install a new second drive and rebuild the RAID, and off we go.
I’ve done that dozens of times in my own systems. It is why I built the CA server the way I did. It is an identical server to 15 others I’m running here.
But for some reason known only to Murphy, this time when the system failed sometime last night, it appears it wrote corrupted data to the “good” drive before the full hardware failure. So at the moment the system is unbootable.
The good news is that most everything should be recoverable, but it takes time. If I can’t repair the boot sector on the good drive, then we have to rebuild two new drives from scratch, mount the one good drive, and pull files over. Though I don’t know just yet how much corruption there is and how much of it can be fixed.
The annoying thing is that these mirrored Western Digital 250GB drives had only 1.5 years on them, and less that 10% full. They were brand new when I purchased and installed them specifically for CA. They have a 3 year warranty. They’ve been in a temperature controlled and dust controlled environment at the CoLo. For one to totally fail now is quite the surprise. I wasn’t all that worried about regular backups due to the RAID mirroring, now the RAID fails with the drive.
I was able to rebuild the RAID, but it appears that the boot sector is corrupted. This will require a mount from a CDROM boot and fix the file system and make copies that way.
Best laid plans….
I anticipate it will be Monday evening before CA is back up and running.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
thefordprefect (03:37:39) :
Let’s not have a bunfight. If journalists want to mock eachother, then so be it. I don’t want to defend anybody.
The point I wanted to make is that Earth Science strayed from good standard definitions of feedback, and it needs to shoulder some of the responsibility for any resulting confusion.
The mathematical/engineering approach is so simple: in negative feedback, you take one signal away from the other: postive you add them.
Stability analysis can follow with the type of feedback taken as a property of the system – and both types can be stable or unstable (or marginally stable in the specific case of an ideal integrator)
I know what you mean about phase shift, but it would be terribly confusing to talk about negative at some frequencies, positive at some frequencies, and somewhere in between at other frequencies.
Back upstream somebody mad a “so much for infallibility of Linux” remark and I let it go for a while, but I can’t leaveit any longer.
The OS probably had noting what ever to do with it.
And as a “sysadmin” that has scars from before anybody used that stupid term, RAID and Mirrors (not interchangeable terms) are good and at todays prices every high-volume operation ought to use both.
But there are other tools conceptually available, but not being used much it appears.
We used to use periodic backups to safe media and kept “audit trails” (tape records of every transaction as “before” and “after” images).
In the event of a hardware failure (and they don’t all have to be “catastrophic”) repair or replace the hardware, reload from the most recent backups, replay the activity since picking the afterlooks off the audit trail, go back on the air as of seconds (or less) before the failure.
Renegade program? Run the audit trail backward applying beforelooks back to where the trouble started.
Cheaper and people-less is not always “best”.
None of the preceding is a finding of falt with Mr. Watts or his efforts, especially when you consider what he gets paid for his work here.
re, my (KlausB (14:16:36))
I forgot to mention, the SATA drives, especially 2xx GB and bigger,
seem to be worse, we allways have spare parts at our location.
Looks alike, IMHO, as bigger in capacity drives get, and as cheaper they get,
reliability goes down.
And a last one:
The type/size of server, you are describing:
We name them ‘pizza-server’.
REPLY: I’m sure it does…but Linux is my second language, and for such a delicate repair I’m waiting for my Linux Guru in Chief to come in to the office Monday morning since I don’t want to make any mistakes – Anthony
BEFORE playing with boot records and attempting to reboot, the first move is to mount the drive to another system and backup the data partitions.
I once properly mangled a disc with a duff MBR which mounted /root to the swap partition. Not pretty.
The usual thing to do is to jumper write protect on the old drive and copy what you need over to the new drive that you will replace the original with. Do no harm is one of the first rules of data recovery and the best way to observe it is to only read from the original. This is old hat for a “Linux Guru in Chief”.
RAID 1 can not be substituted for back-up.
An important site like this need one extra RAID 1 system for back-up.
tallbloke, if the mirror has already been rebuilt then there is already a backup. Just don’t mount both drives until you know it is working properly.
Does Raid 5 offer much more? If the software starts writing bad data Raid 5 will be corrupted just as fast as Raid 1?
This is a good place to start if one wants to poke around the MBR of hard drives (potentially cheaper and faster than sending them off to the professionals for data recovery):
Having a dedicated computer with various easily accessed drive interfaces is a plus along with having same be able to boot to multiple operating systems and equipped with disk editors, partitioning s/w and sector by sector copying capability is a plus.
I don’t mess with hard drives that have been part of RAID striping though. The idea of semi-automated regular backup makes for a higher probability of self sufficient data recovery. It doesn’t hurt to have multiple drives of the same hardware configuration to be able to swap control boards on occasion.
There are solid state drives that are still a little expensive for me, but would likely be an improvement to the eventually disappointing spinning drives. In the interim, I’m using little IDE interfaces to SD and flash cards (available at newegg) along with some of the 16GB USB thumb drives for in-computer archive file (typically read only) storage. So far so good.
Compact Flash. These have worked the best for me. 8GB ones so far.
Maybe a dumb question, but what distro are we talking about?
REPLY: Ubuntu
This is probably redundant to Tallbloke’s observation, but redundancy is the heart and soul of good recovery. When a system fails, Install the drives as secondaries in a good system and copy everything. Don’t fiddle around. Do NOT try to access files on the disk, but DO run diagnostics. Return the drives to the original system and THEN try to recover. I often start my recovery efforts with a boot disk using a different operating system, just to check the hardware. A windows 98 recovery disk will boot on almost any hardware but will not be able to alter anything on a drive unless you tell it to… as John Bell noted above… “first, do no harm”. If windows 98 can’t find your drive at all, it’s probably no longer there.
RAID may provide redundancy, but it’s not BACKUP. A striped drive is fine for bad clusters, but if the drive fails…. automatically writing boot-sector changes to a secondary drive is just bad practice. John Silver was right as far as he went (can I be Ben Gunn?) but secure backup is transmitting data (and DATA only!) to another server and to off-line media that can be remounted.
I hate to be a shill for a monolithic software company, but there is a product out there that allows you to run a number of “virtual servers” on a single piece of hardware. Each virtual server can have its own net address and will look, for all intents and purposes, like a real box. It even has an “undo” feature, meaning that if a bad configuration gets implemented, the whole problem just be evaporated (ahhh.. along with whatever data was created during that time period…) BUT you can link a number of virtual servers into a server farm living on two or more physical pieces of hardware and have them replicate to each other.
Technology can be good.
As others have said, WUWT and CA are so valuable to us. I’ve no idea if it would help, but I have 500 gigabytes of unused space in my Dreamhost account, and I’d certainly be happy to offer it as a place to back up these important sites. What size of data would we be dealing with?
I’m sure others would do the same – we could end up with a widely distributed and redundant backup situation. Those of us with suitable facilities could even automate the process of distributing the backups, so that the WUWT/CA would only need to make one or two backups, which would then propagate around all the volunteers…wouldn’t matter if some failed or disappeared. Could be encrypted if Anthony and Steve are concerned about it being misused…
A couple of cron jobs on the WUWT/CA servers and all the valuable data is safe…
The offer is there in principle, anyway.
REPLY: Thanks Once it is back up and running we’ll explore the options. – Anthony
I volunteer!
Another volunteer here, plenty of room on my hosting account.
Mr Pete: “Now, with respect to evolution of life — from non-life to life, the question of origins.”
At the moment, as far I understand it, the origin of life is somewhat speculative, although there’s no particular reason why life could not have arisen from inanimate matter – after all, living things are composed of matter.
Once again, science operates on a basis of methodological naturalism. If you start including speculations about non-naturalistic influences, you’re not doing science, you’re doing something else such as theology or philosophy.
“If you would like to read up on a surprising, truly scientific set of falsifiable hypotheses regarding naturalistic vs supernaturalistic origins, check out
http://www.amazon.com/Origins-Life-Biblical-Evolutionary-Models/dp/1576833445.”
Thanks for the link. I will check it out.
Climate Audit still down? Maybe they tried nested comments too, but ones that work the other way by deleting the previous level and replacing it with a blank.
“I anticipate it will be Monday evening before CA is back up and running”
It won’t be Monday, it’s going to be longer than that.
ClimateAudit tried nested comments a long time ago, but soon got rid of them. Whatever the disadvantages of referring to previous posts by number, it is much easier to catch up with a long flat thread.
There seems to be a spate of sites of conservative leaning with hardware issues. American Blogger has been down all week. I wonder if it is not time for certain sites to start hosting outside the US. With the EPA moving to regulate CO2, I don’t see sites such as WUWT being looked upon in a benign way. America is becoming the land of the less free.
If GWers are trembling, it’s because the denialists have succeeded so well at suspending the laws of physics when applying them to AGW that the chance to avoid serious tipping points is slipping away.
Congratulations!
Walt, you are projecting, just as you were in your comments at CA regarding Mann’s infantile outburst vs Lawrence Solomon [ = “Mr. Lawrence”, h/t Anthony], which you essentially then replicated by similarly attacking CA!
What do you ~ “think you are accomplishing” by bashing and blaming others, when they are really only a projected proxy for yourself and AGW’s weak science? All you are doing is talking about yourself and AGW’s weak science, and it’s really quite easy for others to see! Has a real tipping point been reached concerning your own personal psychodymics and “scientific” thinking, or, hopefully, can you walk them back toward reality?
Walt has just thrown in the towel. No loss to the AGW faction though. Walt does not use scientific discussion to persuade, but merely attempts to bludgeon us with the chimeric tipping point boogieman. Sorry Walt, go back to the dugout and send in some heavier hitters.
Yes, RAID your drives, and mirror them, too.
Also, I recommend you mirror your system on-site, and have an off-site hot site ready (Comdisco, or other).
May I first apologise to Anthony that the serious problem with Climate Audit has got rather entangled on this thread with a discussion about my column in the London Sunday Telegraph (which regularly cites WUWT for obvious reasons).
But may I also thank those readers who rushed eloquently to my defence against the garbage which has been inserted into my wikipedia entry and elsewhere on the net, grotesquely caricaturing my statements on various scientific issues. The origins of this sabotage lie in the fact that I have aroused the ire of those who make money out of various asbestos scams in the UK. They have now won the support of our leading journalistic AGW fanatic, George Monbiot (aka Moonbat) of the Guardian, who has swallowed whole the scientific confusions behind the asbestos scare just as he has those used to promote climate alarmism. The one thing I have to hand to these people, however, is that they are very clever in rigging both wikipedia and Google, which is why their scientific illiteracy (and malevolence) gets a much wider circulation than it deserves.
Meanwhile, we all send our best wishes to Steve in getting CA back on line. CA and WUWT are both doing an invaluable job for the planet in this new dark age we are living in and we are immensely grateful to them.
Mr Booker
I appreciate your comments on climate change. There is a lack of balance in the coverage of the debate, and that leaves an under-represented body of opinion, in my view.
Your sentence is nearly correct and I would not go so far as to call it a howler. I could also excuse the BBC if it finds this all a little bit too confusing.
From an engineering/mathematical perspective, negative feedback provides a countervailing force to an imput force. Pull on a spring and the spring extends. But it also develops a force proportional to extension. The spring will therefore reach an equilibrium length when the reaction balances the input force.
The crucual deremining characteristic is that negative feedback is based on the difference between two signals (or forces, or whatever). Positive feedback adds them.
You said “negative feedback” would lower temperatures rather than raise them. Almost correct. In the context of a force which tends to raise temperature (after all, is that not what global warming is all about) negative feedback must mean a reaction which tends to cool and therefore counteract the warming force. It would have been better to have said “negative feedback would *tend to* lower temperatures”, but that’s not really such a big deal in my opinion.
On the above thread, a poster claimed that negative feedback stabilises temperature.
To an engineer (or mathematician) that statement is incorrect. It comes to the point I have made a couple of times. For some reason (heaven knows why) Earth Science seems to have adopted a strange classification of feedback. Not a very useful classification either – because I don’t think the Earth Sciences classification of positive feedback really destabilise (in a mathematical sense)
I should just say it to be clear – according to the mathematical/engineering convention, stability is never guaranteed. Both positive and negative feednack can be stable or unstable.
I can promise you that anybody who wishes to attempt a proper analytical study of feedback in the climate system will need to forget the Earth Science classification and adopt the mathematical convention.
So is there confusion? You bet!
And I have my own view about where things started to go wrong.
ps sorry about the typos (like “imput”)
J. Peden,
It doesn’t matter any more. You and the other obstructionists “win.”
Your children and their children will reap the “reward.”
Further discussion is utterly futile.
Walt,
“Your children and their children will reap the “reward.””Since the AGW alarmists and you refuse to voluntarily reduce their/your Carbon footprint by 90% or more immediately, my kids will just have to “suffer” with a MWP.
Christopher:
I have enjoyed your efforts to counter-balance the catastrophic AGW drumbeat but I hasten to say that CA’s main contribution has simply been to push for access to data and code used in a certain set of climate related studies. IMHO he and other contributors have done a tremendous job shedding light on some rather poor practices in this particualr research area, viz., the HS and the current Nature article on the warming of Antarctica. If that is what you mean by “dark age” then I agree. I would hesitate, however, to characterize SM’s view of AGW in general.
Was I right, or what?
Joe Black,
Enjoy your time to make light of this crisis which will redefine man’s relationship with his only planet in ways that future generations will be helpless to avoid. The thanks will go to those who were alive today and chose to suspend laws of physics only where AGW was concerned, who were capable of rational thought except where AGW was concerned, and who learned and loved science except where AGW was concerned.
When you take carbon which nature needed MILLIONS of years to store, and you “un-store” it in a century, you are taking over the major influences on the climate. As Hansen puts it, man is now in charge of climate. The only problem is, he has no idea what he is doing. Thus he is utterly blind to the consequences of his own actions.
There is zero scientific basis to assume that CO2 is not an enormous forcing on climate. Paleological evidence is conclusive that it always has been. The same evidence assures us that the planet can be ice-free, that sea levels can be hundreds of feet higher than they are today, and that abrupt sea level rise of a meter or more per century can happen.
Your only hope is that some unknown mechanism will prevent that from happening again. It is a fool’s hope.
Whatever does happen in the end, it is somewhere close to insanity to pretend that you know it will all be alright. Yes, the planet will be alright. It was alright in an ice free state before and it will be again.
It’s life as we know it that must undergo dramatic change in a paleological eye-blink.
To you and the other obstructionists, again I say: Congratulations. You have successfully assured that it will happen.
Based on which, man quite deserves to be rendered extinct, and surely will be.
Pogo: “We have met the enemy, and he is us.”
Well, great.
If it’s such an emergency, you have obviously rearranged your life to minimize your impact (or gone off to starve yourself to save Gaia). Since you would have done so to both save Gaia and to be a leader, lighting the way to enlightenment, please advise as to the steps you have personally taken.
What you are saying here is a contradiction.
You are saying the physics behind CO2 forcing is invalid and the people promoting it are irrational and are using something other than science to promote it. Yet in the next paragraph you say it is a major influence on the climate! Obviously there is no science to back this up.
And, from your first paragraph, there is zero scientific basis to assume that it is, just an irrational belief system! So, if there is no scientific basis one way or the other then the effect on the planet could be good, bad or nil.
“If everybody walked backwards it would decrease CO2 emissions.”
There is zero scientific basis to assume that the above statement is not correct.
RE:
**There is zero scientific basis to assume that CO2 is not an enormous forcing on climate**
Can you give me the name of one scientific study which MEASURES the amount or percentage of the heating or cooling caused by CO2?
All you have is a graph which shows temperatures rising and CO2 increasing. Note that TV sets and cell phone are also increasing in number.
Note that models are just that – models.
Gerald,
Do you understand these terms?
1. Persistence
2. Acceleration
3. Forcings
4. Feedbacks
5. Tipping Points
Get back to me when you THOROUGHLY understand them and how they relate to your question.
Drop me a line at http://realskeptic.blogspot.com
Walt, a hysterical, fatuous rant.
Trying to fix something you don’t understand properly is ridiculous and futile. Pretending you do understand it when you don’t is stupid. Wasting vast amounts of money with no achievable objective is criminal.
Supporting poorly established conclusions in the face of contrary and conflicting evidence is unscientific.
Take your certainties to your place of worship or your political soapbox. They have no place in science which must always celebrate and preserve skepticism, questioning and further careful exploration and examination.
Alan,
I think you must be learning-impaired.
Perhaps YOU do not understand the power of CO2. Others do.
Your solution for things you don’t understand is to keep doing them?
OK then. Chalk up one more piece of evidence why man will surely go extinct. Once more, he is just too dumb to recognize the consequences of his actions.