In the attached picture you’ll be able to see what’s happening to one of my most important servers containing some irreplaceable climate data. I’m at a loss and understand what’s going on because Linux is not my specialty and my Linux expert has since moved on to other ventures. It’s very important I get this server working again, so I’m asking the WUWT community for help.
The server had been offline for a few weeks, and was properly shut down. Upon powering it back up, I got a “no bootable disk found” message. I determined the RAID (Hardware RAID1 – 2 mirrored drives) had been degraded, and it seemed one disk had failed. So I purchased two new identical HD’s cloned the good one, and rebuilt the RAID1. The RAID is administered by an on-board Adaptec RAID controller, and it reports the RAID as healthy.
What happens now is that it attempts to boot, but gets stuck in a loop on the last messages “Init ID c1, c2…etc” and repeats those error messages. I get the same partial boot and error sequence if I take out the RAID in BIOS, and try booting a single drive in straight SATA mode.
This machine was built circa 2007, and has Slackware Linux of that era installed, I don’t see a version number coming up on boot, so can’t provide it.
Any and all help appreciated. – Anthony
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Virtual console issue. I don’t have time to look at it right this second (dentist appt) but there’s a bunch of stuff on google about this error message. Might be having trouble finding the virtual console device or the console fonts. Might have lost one or more files somewhere. Might have to boot off a live CD, mount the raid device and go hunting.
yes. I agree. this should not stop you booting tho.
Try http://www.linuxquestions.org/questions/slackware-14/slack-12-2-upgrade-and-init-open-dev-console-no-such-file-or-directory-693365/
Is the highest priority to recover the data or to get the server running again? If data recovery is most important would suggest offline recovery activities.
+1
First rule of disaster recovery: sit on your hands and not make a bad situation worse.
Before you do anything else, image those drives somewhere safe.
After that, I would personally start a disaster recovery using copies of those images and a known good OS in VMWare, but that’s me.
And remember it’s the restore that counts.. not the backup. Make sure you don’t restore from a bad backup.
My reading of the story is that a new RAID was created using the one good original onto two new replacements. If so, the original is now the Golden Master backup of the data.
IMHO, best path to recovery is to build a new Linux on a dedicated single drive, then mount the RAID to it and commence to validate the data. Slackware is a decent build, so I see no reason to change (though generally I lean to Debian, or since the systemd infestation, Devuan.
I’m available to do it if needed. 3 hours drive away.
Drive 3 hours to help? That’s awesome. Voluntary communities are the best.
I think what you have is a RAID puncture. It usually begins with a faulty disk. You replace the disk, the RAID is re-stripped, but the array still refuses to boot. I went through this a few years ago. A punctured RAID array is pretty much lights out. I don’t know if you use a PERC card or not. This seemed to plague those cards about a decade or so ago. I don’t work in the Linux world that much, so I don’t have any Linux tools that come to mind. Here’s a link that may be of help. When I went through this very same problem, I did an emergency P2V (physical to Virtual machine) conversion. I was lucky.
http://www.theprojectbot.com/what-is-a-punctured-raid-array/
JP, if you look at the original post, it’s a RAID 1, not a RAID 5. The failure mode you described doesn’t seem possible for RAID 1.
You can have a punctured RAID 1 array. Especially if using Perc cards
https://community.spiceworks.com/topic/279704-punctured-raid-1
“raid 5 measuring checksum speed” ?? If this is supposed to be a RAID1 is the bios on the controller / motherboard somehow set to RAID5?
Like others suggested get a backup of the good disk. Plug it into another computer and manually mount it, back it up. Remember Captain AHAB (Always Have A Backup) so Moby Disk doesn’t eat your data.
RAID 1, mirrors, can be re-cloned from the last good disk. Anthony reports buying TWO new disks and cloning the old disk. With luck the old disk still exists. Using a live bootable image, you can clone the disk, partition table and everything with “dd” (disk to disk) but this is a thing terribly easy to get wrong and erase the source disk. But it is easy, reliable and every Linux has it built-in.
I’d install a new linux and then copy the valued data from the old disk.
If the RAID is in fact healthy, another thing to do is to create a bootable Linux flash drive or CD, boot from that drive, see if you can mount the volume in question. If you can mount it, copy the data to another drive. I don’t know how much you can afford, but you may want to begin thinking about taking your site to the cloud. Let the Cloud providers worry about the hardware.
Yes. this would be the best course. Boot from a CD or other media and fix the configuration. You can also try bringing up the system in single user mode. The messages indicate that processes forked by init are crashing, likely related to tty’s being misconfigured. Why this would occur after recovering a disk is unknown, but could be because the hard drive failed during an update which left the configuration in a bad state.
Totally agree. I recommend System Rescue CD (also works on a USB stick, despite its name.)
Best guess: your /sbin/agetty binary is corrupt or missing.
.
Yeah, right. They worry about the hardware then all you have to worry about is who, what and where, the “cloud” is and whether that may evaportate at some unknown point in the further. You do realise clouds can evaporate, right?
Firstly this is not about the WUWT site it was some climate data that he had/has backed up off line. I would imagine that the point of that is be in possession of a physical copy in case the orginal provider decides to remove it from public view. If this is a “valuable” backup then that has probably already happened.
While making more copies “on the cloud” would not do any harm it does replace being physical possession of your own data.
Whenever someone writes “in the cloud” you should read “somebody else’s computer.” And there are cloud failures all the time. Bottom line: as long as your bill is paid, Amazon/Azure doesn’t give a damn if you lose your data. Read any cloud TOS & ask yourself if they have any liability whatsoever. (Hint: they don’t.)
Anthony…the advice to get a backup right now is sound. P2v if you can. You should focus on saving your data at this point; the hardware isn’t going anywhere.
In future…if a RAID1 goes down, just break the mirror, mount the good disk, then boot to it. Backup yr data & THEN rebuild the mirror. Trying to mirror a faulty drive can easily result in 2 bad HDs.
Cloud failures happen, but local failures happen a lot more often.
Amazon cares a very great deal whether you lose your data, it’s critical to their whole business model.
Eventually it may make no more sense to own your own data storage than your own water storage. Now, in some situations you need your own water, but for the vast majority…
“You do realise clouds can evaporate, right?”
Which is why all systems, local or cloud, must have remote offsite backup and restore capabilities for ultimate safety. The only substitute for this capability is supreme confidence that nothing can seriously go wrong. Ever.
Someone once said experience is the hardest teacher; it first administers the punishment, then the lesson.
The hard lesson learned by most admins is that, eventually, *something* will go badly wrong.
Lessosn from experience are often both simple and obvious, as in this case. Purely static data can be reliably transferred to a hard copy non-volatile medium which is easily stored in your bank safe-deposit box (good). Or with a trusted friend in a location geographically distant (better). Or in a salt mine vault (best).
This simple step to provide ultimate salvage capability for critical data usually becomes obvious only after a major calamity, rather than before. Speaking from bone-headed experience, of course.
Whatever is supposed to run on the virtual console (probably /sbin/login or something like it) is crashing at startup, possibly due to bit errors that the RAID controller didn’t detect. The “respawning too fast” message means that the supervisor process for them (called init, because it’s the program that the OS automatically runs at startup) is waiting in hopes that somebody will fix the problem — it isn’t smart enough to figure out that nobody can fix it in the current state.
I’d recommend making a “LiveCD” disc or thumb drive, and boot off that. You should be able to check the remaining data, and copy it elsewhere if need be. Installing a clean OS without wiping the existing data can be done, but should be done by someone with a moderate amount of Linux experience.
If you can’t feasibly make a live CD, you should be able to bit into single user mode by editing the kernel’s command line. Details of how to do that will depend on the bootloader that is set up.
I carry two bootable “thumb drives” just for this purpose. Emergency booting of a computer and then copy the disk that won’t boot onto a portable USB disk. Do it twice on different destinations; verify success of the copy. Then you can start rebuilding your system.
Gee… similar to my kit. A few bootable USBs of different releases, an empty TB USB disk, a stack of “live CDs” and “installation CDs”, a portable laptop pre-built dual boot to work from… The more your experience, the bigger the kit grows…
If you look at that picture, it looks like the partitions are ok. So the data is most likely safe. I’d boot that computer with a usb-based live-Linux, mount the drives and copy the data to a safe place. Your best bet is to find some knowledgeable Linux-user nearby to do that. When your data is safe, you can then concentrate on fixing the problem.
Very sound advice
Yes.
Before you do ANYTHING on this machine:
From a different machine, hunt down instructions to create a bootable Slackware USB flash drive.
Create the boot flash drive on that machine.
Use your new USB boot drive on this server, mount the RAID, copy the data somewhere else.
It actually ends up only being a few commands in total for this process (including possibly changing BIOS config to boot from USB), but commands will be specific to your situation.
Only once these steps are done should you make any attempt a rectifying the issue with your server.
For future reference, RAID is for availability, not backup.
That should be in bold and all-caps: RAID is for availability not backup.
Server failure, theft, or destruction should never endanger your “irreplaceable climate data”. Although I am not a fan of “cloud storage” it makes a reasonable off-site backup of last resort.
What you have here is the problem of mixing hardware “fake raid” and software raid.
Those adaptec raid controllers are primarily setup to be used on M$ based systems.
The linux software raid is the way to go.
Saying that, your data is not lost, it just will take many many hours to recover. Especially if there are multiple TB of data.
The mdadm tools are there to recover the data, I have to do this for clients once in a while, thank goodness this is not a common problem.
There are many many forums on the interwebs to help recover this data, it is just that they are very cryptic.
I’m sorry that I’m a little far from Chico, or I’d be over there right away.
The most important thing to do is get a brand new Hitachi or WD HDD and dd your best drive to it, that way you are not making unrecoverable mistakes.
For a storage server use software raid 1 with raid certified drives ( not all drives are friendly for raid ).
The best server environment I have used is the Koozali SME server found at contribs.org.
For just data storage use a ultra low wattage system and a name brand power supply.
I use a ZOTAC H87-ITX board with the lowest wattage laptop processor available and an Antec power supply.
Anything I can do from here in BC I will.
If it’s a mirror raid then there should be two identical copies of the data.
I have run many varieties of raid configurations… and lost a lot of data, but never with mirroring.
My worst data loss was with a 3 drive configuration that should have been able to rebuild except I did it with a virtual machine with soft raid and somehow lost 2 drives in quick succession.
I spanned 2 drives once, and lost 500GB of tv shows I recorded. I have yet to find out how Battle Star Gallactica turned out (not the original series but the remake).
Don’t know anything about RAID drives or Linux, but I think you should feel happy you never saw how they ended Battlestar Galactica. Just keep your imagination about how they might have ended it.
Agree w/Timo — an awful/confusing ending to Battlestar second edition.
Don’t know if this helps but I saw this on line:
>This happens when you are using the serial console. To get rid of those
>messages, comment out all lines under “# TERMINALS” in /etc/inittab and
>execute “telinit q”. There should be six lines from c1 to c6.
This is partially right. Here are a few sequential lines on the
/etc/inittab:
…
…
# SERIAL CONSOLE
c0:12345:respawn:/sbin/agetty 9600 ttyS0 vt100
# TERMINALS
c1:12345:respawn:/sbin/agetty 38400 tty1 linux
c2:12345:respawn:/sbin/agetty 38400 tty2 linux
c3:12345:respawn:/sbin/agetty 38400 tty3 linux
c4:12345:respawn:/sbin/agetty 38400 tty4 linux
c5:12345:respawn:/sbin/agetty 38400 tty5 linux
c6:12345:respawn:/sbin/agetty 38400 tty6 linux
…
…
So, if I change the above to this,
# SERIAL CONSOLE
#c0:12345:respawn:/sbin/agetty 9600 ttyS0 vt100
# TERMINALS
c1:12345:respawn:/sbin/agetty 38400 tty1 linux
c2:12345:respawn:/sbin/agetty 38400 tty2 linux
c3:12345:respawn:/sbin/agetty 38400 tty3 linux
c4:12345:respawn:/sbin/agetty 38400 tty4 linux
c5:12345:respawn:/sbin/agetty 38400 tty5 linux
c6:12345:respawn:/sbin/agetty 38400 tty6 linux
it works. But if I change it to this,
# SERIAL CONSOLE
c0:12345:respawn:/sbin/agetty 9600 ttyS0 vt100
# TERMINALS
#c1:12345:respawn:/sbin/agetty 38400 tty1 linux
#c2:12345:respawn:/sbin/agetty 38400 tty2 linux
#c3:12345:respawn:/sbin/agetty 38400 tty3 linux
#c4:12345:respawn:/sbin/agetty 38400 tty4 linux
#c5:12345:respawn:/sbin/agetty 38400 tty5 linux
#c6:12345:respawn:/sbin/agetty 38400 tty6 linux
as you suggested, it does not work. I still get the same message and I get
more error messages. So, the answer, at least in my sparc64, is to comment
the serial console that is giving you problems. Ie.
# SERIAL CONSOLE
#c0:12345:respawn:/sbin/agetty 9600 ttyS0 vt100
At the end of the day get Centos installed, it’s a production grade OS that while behind the curve of the latest releases it is bullet proof for production and they do timely security updates.
FYI: I had a mirrored RAID setup on a production server once(Novell), when the controller got flaky both drives were hosed. After spending three days rebuilding I discovered the issue and had to restore for the day before backup, lost 8 hours of 26 editors work. Moved to RAID 5 with 6 hot-plug and play drives in external box.
Now I use USB Thumb drives for backup and long term storage, all my tape units and DVD optical crap is history. Hint: Tape backup is just that, don’t try to restore it two years down the road! DVD’s are close behind due to mechanical and media issues.
Do not use thumb drives for backup or archiving. Or SSDs for that matter. Best for low end are hard disks, refreshed every 2 years and replaced after 10.
Why not? 7 years ago I would agree with you but the latest USB 3.x drives are bulletproof. By running read/write diagnostics for a few hours to test the new drives before using them for long term archival storage. The MTF for mechanical drives is several times lower than NAND drives.
The electric charge on SSD, SD cards and Thumb Drives very slowly leaks away. You don’t notice in normal use since simply plugging them in refreshes the charge. BUT, put one in a safe deposit box for a few years and you can come back to find the data simply leaked away, or that in the early stages some bits have changed from 1 to 0.
IFF using those NAND storage systems (or similar) for backups, plug them in to a live system once a year to refresh the charge.
Oracle also gives their RedHat variant away, along with their VM server. All commercial grade. You just have to pay for support if you need it. But the software is all free.
Like others have said, use a live CD to boot and then copy the data to a different device. After that, why not just reinstall a modern Linux version (or other OS) from scratch – you don’t want to tell the world that you are running an ancient Linux version that likely doesn’t get any security fixes any more.
If you don’t get enough of an answer here, I might suggest contacting frequent contributor E. M. Smith, “Chiefio”. He has a clue or two about Linux.
It looks like you’ve already been steered in the right direction.
Thanks for the props.
Yes, I do Linux for a living and have since day one (I.e. started *nix on BSD pre Linux).
Anthony, I’m available for free if desired and can bring a bag of kit including spare USB disk, thumb drives of several flavors, laptop with Linux and more.
Hehe, I popped my cherry on Sys III, but spent a big chunk of the last 20 year using windows, I now have a VM Server that’s all on linux, and a bunch of linux vm’s I run as needed.
Did you ever have to drill coax to put an ethernet tap in? I hated that, it seemed so cheesy lol. I hope he guy who put it all in a box got very wealthy, he did a great service for the world.
“Did you ever have to drill coax to put an ethernet tap in?”
I came into thicknet right at the tail end. We had one wall covered in figure-8’s of thicknet in order to attach the server farm. You can only attach at equidistant marked spots on the coax to avoid standing waves. These taps were called “AUI” if I remember right.
They were at qtr wavelength for 100Mhz, so about 3 m, never had to wire a server room, what a mess that had to be, but I see how you got to figure 8’s lol
Most of mine were just on coils, or around a few cubicles. They were on First generation electronics design workstations, 68020 running Bsd. The story went around the founders knew the Sun guys from Berkley I presume, but gave them the design which they turned into the Sun3. Which Valid switched to instead of continuing to build their own workstations. Fun times.
Yes. Don’t remind me. (Raised ceiling over engineering desktops… DEC gear… At Apple in the ’80s…)
Nothing like panel dust in your face and shirt, balanced on a ladder, drilling an unsupported round thing (using a jig) while folks ask why you aren’t done yet to make you wonder why folks think computer work is all desk work and comfortable…
You need to comment out entries 1-6 in /etc/inittab as Bear said above. Generally this error is caused by keymappings in rc.conf passing unwanted keystrokes to a virtual terminal. (That’s the “console” error you see in the middle). Editing the init file should remove those possible console connections. Was this machine originally setup without a monitor and only logged in remotely? I’ve seen this when older machine are decommissioned and login is attempted locally instead of remotely. You may need to boot to a live instance to edit the files to get it to boot, or…. just save you data and set up something more modern 😉
If all else fails, try loading Microsoft Word and Word for Windows!
Is this a troll?
This is: “try loading Microsoft Word and Word for Windows” – but use the Mac versions!
I am no linux specialist, just a user 🙁 ….
Can you boot from a linux CD or USB and mount the drive?
Eventually then show the inittab?
Some more troubleshooting help:
https://www.lifewire.com/text-terminals-on-linux-2205461
Are you able to boot in single user mode?
By the way, you should be able to boot with a recent version of linux and mount the drives to copy/save the data. Do you need the old version running for any particular reason?
What is the status now?
Quick question: How much data and is it compressed? Just looked at a 1TB SSD drive for online read storage, long term bullet proof stuff, not so good for daily production but for READ-ONLY great stuff.
I’m not the Linux expert you’re looking for, but let me pretty much agree with everyone else. Yes, the symptom (not necessarily the underlying problem) is with your virtual consoles. It’s quite possible that most, maybe all, your data is intact, so your top priority probably should be not to inadvertantly destroy it. What I’d probably do is buy yet more drives, clone the ones you have, verify that the clones do the same thing as the current disks, then try to recover your data from them using a single “disk” linux system — usb stick, cdrom, linux installed in a spare partition if you have one.
Don’t overlook the possibility that your RAID controller, CPU, or memory might be failing in some weird fashion. That’s probably not the case, but if it is, it’s possible that nothing you try will quite work or make sense.
I also agree with PSU-EMS-Alum that all things being equal Slackware might be a good choice for the single disk system. It tends to be the most “plain vanilla” unix system around which means various cures/analytic procedures you find on the internet may well work. And my experience in the past is that the is intelligent life at http://www.linuxquestions.org/questions/forumdisplay.php?forumid=14 — something that is not true of all Linux distribution support forums.
I’ve used most versions of Linux. Above someone suggested Centos. It is good, but stodgy, and is now systemd for init so LOTS of conversion work for any customizations on the Slackware (that is BSD like rc.d init).
For ease of update, I’d likely keep it Slackware and update to current as a fresh install, then copy over any customizations and last reinstall the data disks.
On a second machine I’d be doing the disk cleaning and prooving up. Any Linux ought to do for that, as the file system is likely ext3. (Says ext2 for root but the RAID might vary)
Last on my list would be a full on conversion to a different OS with a new init system and all… and only if there was some good reason to do it. Arguing over your favorite flavor of Linux is not a good reason…
puppy linux on a usb drive to enable it to boot?
Another thing to do is run the dmesg command as root and post the output here, as it is, we can’t tell too much of what’s going on, with dmesg output, we could possibly trace down the errors.
I’m a Linux sysadmin. If I were you, I wouldn’t trust half of the answers you’re getting.
And for that matter, there’s no reason for you to trust me either: ANY jackass can claim to be a Linux expert.
Call Tony Heller. He’s someone you know, and I’m betting he knows his Linux.
It’s a lot like climate science, ‘eh?
There are at least 4 main types of Linux, and countless variations. The details of how to do things vary widely, few people have experience outside their favored flavor, and a given error message can have several causes so folks will recommend the one fix they did in their case while YMMV.
Then there are huge personal style issues…
I learned Unix on my own, so I learned what I learned, but I always liked working with other admins, because many did they same, but learned different things that I did, I tried to snatch up all I could.
And yes, this is measure 64 times, then measure another couple, then cut situation.
I would set those drives to the side, build a new system, get it all working. Then mount a copy of your drive on that, and figure out what’s wrong.
I use Linux but my Linux expert keeps it healthy for me. His suggestion follows:
“Looks like he lost and|or corrupted data when his RAID failed. His trouble is with setting up the virtual terminals. Needs to look at /etc/inittab (not too familiar, his system is running very obsolete version of Linux) He will probably need to use a LiveCD session to fix. May have to comment out the offending init lines to get it to boot and then reinstall the associated visual terminal package that is broken some getty package needed for login.”
Good luck, I hope you can save your data!
PMK
Praying for you, Anthony.
Try what Bear and Dougmanxx suggest. Of all the suggestions, that is simplest, with no possible adverse effects on the data. However, it assumes you can boot into a level allowing you to edit that /etc/inittab file. Perhaps someone can coach that aspect?
Maybe this forum posting will help?
https://forums.gentoo.org/viewtopic-t-1051286-start-0.html
Googled “run level 3 Id “c1″ respawning too fast”
There may be a misconfiguration of your /etc/inittab file
You are going to need a specialist. I was a linux user for about 6 years, got sick of the unfriendly command line, and lack of hardware support, dumped linux and Went to windows 10 late last year.