Broken fakeraid

[donkeyass]

So I've learned my lesson, the hard way unfortunately, to never use a fakeraid or NTFS every again. I had an Intel fakeraid 10 setup across 4 disks with a single NTFS partition and one of the disks had a physical failure. I thought it was no big deal because I could just replace the failed device and it would rebuild itself. Turned out to not be that easy.

I replaced the failed disk and went into the Intel setup at boot time to configure the new disk. From there I booted into Fedora and waited for it to rebuild itself but it didn't. I could still access the disk at this point though. I booted into Windows to use the Intel RAID utility to rebuild the RAID and it started to. Said it was gonna take like 60 hours or something ridiculous. After a couple hours I went to go and look at the disk through Windows but it said the partition was RAW. This is when my hopes and dreams were smashed to the floor.

I stopped the Intel RAID utility and booted into Fedora to view the partition with TestDisk. I created an image of the disk and tried to get some files off of it but couldn't. Went back to Windows to use Zero Assumption Recovery to copy off some files. Some of the files aren't damaged but a lot are, says that there's a lot of "missing" files and directories.

So my quest for advice at this point is, should I:

Run chkdsk on the partition.

Continue to rebuild the RAID with the Intel tool.

Try to repair the RAID with Zero Assumption Recovery or some other tool.

Any advice would be greatly appreciated. Thank you very much!

[jono-ats]

Always image RAID drives and work with the copies -- not the originals. That way, you can always go back to the initial failure mode.

[donkeyass]

Image each device? Or the partition?

[sathyan]

Try to bring up the failed drive if that is easier.
What is the physical failure to the drive you mentioned?
Trying to bring that dirve up and clone it to another working new drive would be one option.
Else you will have to try the rebuilding of the raid.
But make copies of the working disks and try with them,avoid using the original disks,
so that even if if the process fails or corrupt the working disks, you still have the original disks in its original state to start the process again or seek pro help on them.

[donkeyass]

The failed drive just started making a weird noise one night, the next morning it stopped making the noise but I couldn't access my RAID. So I rebooted and the system would hang and wouldn't finish booting. I figured it was a failed disk so I shutdown my computer, disconnected one disk from the RAID and rebooted. Continued this for all my RAID disks until they system didn't hang and actually booted.

Ya think that the Intel RAID tool would be the better option? Or RAID recovery software?

Exactly, that's why I made the disk image as soon as this happened.

[lcoughey]

It has been my experience that if one drive in a RAID fails, the rebuild with the replacement will bring forward the bad sectors on the other drives, causing the RAID to completely crash. So, always confirm that the data is 100% backed up before replacing a failed drive in an array. If the array went offline due to the failure, you know that it has already reported 2 or more drives as offline and you need to get a full mirror of each drive before you go any further.

[donkeyass]

It has been my experience that if one drive in a RAID fails, the rebuild with the replacement will bring forward the bad sectors on the other drives, causing the RAID to completely crash. So, always confirm that the data is 100% backed up before replacing a failed drive in an array.

Ouch, I definitely will then! Just ordered 4 3TB disks off Newegg to help get through this as successfully as possible. Since this happened I've been hearing that it can be very difficult to rebuild fault tolerant RAIDs and recover the data when a disk fails, isn't the whole point of a fault tolerant RAID to avoid catastrophic data loss? That's why I started using it in the first place.

If the array went offline due to the failure, you know that it has already reported 2 or more drives as offline and you need to get a full mirror of each drive before you go any further.

Well the RAID never broke until I tried rebuilding it with the Intel tool ironically enough, when the first disk went down it just said that the RAID was degraded which was expected. I could still access the partition no problem.

EDIT: As I read over the first part of your response again, it makes more sense in my situation. The rebuilding of my RAID probably did forward the bad sectors to the new disk and screwed up the filesystem.

[Vulcan]

EDIT: As I read over the first part of your response again, it makes more sense in my situation. The rebuilding of my RAID probably did forward the bad sectors to the new disk and screwed up the filesystem.

That is not what I interpret Luke was saying - he didn't say that the rebuild would forward (i.e. copy) bad sectors to the new disk. He said that the rebuild would bring forward bad sectors (i.e. reveal what was previously hidden) on the other drives. Since I work with large (enterprise quality) RAID systems every day, this is a good hypothesis (though not yet proven in this case) and that is why it is important to do whatever can be done, to always be confident in the health of all the disks. One simple scheme is called "background scrubbing", although that's not perfect.

In short, RAID is never "install and forget", if you want to rely on it working as you hope in the future. There is extra sys admin overhead, if you want to minimise unhappy surprises.

I've been hearing that it can be very difficult to rebuild fault tolerant RAIDs and recover the data when a disk fails

Of course there can be problems, but good-quality, well-designed RAID systems, without a pre-existing hardware fault (this can be difficult to check!), and where the necessary sys admin is being done, will only rarely fail to cope with typical disk failures, in my experience. Very often, I find (as Luke implies) that there was a pre-existing problem e.g. unreadable blocks, sometimes from years ago(!), when I see a RAID reconstruction failure.

isn't the whole point of a fault tolerant RAID to avoid catastrophic data loss?

There are limits... Depending on the exact fault-tolerance in a given RAID system, there are specific faults which will allow continued data access (i.e. they can be coped with). In general terms, the cheaper and less well-tested / well-designed / well-maintained the RAID system, the greater the number of failure types which it cannot cope with.

For this and other reasons, (redundant) RAID is never a substitute for backups. Unfortunately the advertisng of consumer RAID systems doesn't make that point, as well as those systems being designed to be low-cost, and something has to be sacrificed for that.

[donkeyass]

He said that the rebuild would bring forward bad sectors (i.e. reveal what was previously hidden) on the other drives.

Ahhhh, kk gotcha. Ya these disks were in okay shape, didn't have any problems with them but I definitely didn't do regular maintenance.

In short, RAID is never "install and forget", if you want to rely on it working as you hope in the future.

That's something I've also learned during this, cause that was the impression I was under for a very long time.

So it looks like I'll have to wait for my new disks to get here and make those images. I'm thinking that the first plan of attack would be to try and repair the RAID using the images? Are there any super 1337 tools anyone can suggest to rebuild the RAID or help me in this situation?

[Vulcan]

Ya these disks were in okay shape, didn't have any problems with them but I definitely didn't do regular maintenance.

If you weren't doing regular checks, then I don't know what evidence there is that the disks were OK. For example, if there was no background scrubbing in operation, then you are exposed to an increased possibility that there were undetected unreadable sectors on other disks. In those cases, RAID reconstruction may fail (depending on RAID level etc.)

So it looks like I'll have to wait for my new disks to get here and make those images.

If you are accepting responsibility for the risks of DIY (e.g. you could cause totally unrecoverable problems if you make a mistake or are just unlucky), then IMHO making image files (or raw clones - whatever your plan is) from the existing disks would be step 3.

I suggest that before doing that, then in order to reduce the risks:

- Step 1 would be to thoroughly test your new disks, and setup that new hardware ready to perform the imaging / cloning.
- Step 2 would be to choose, and then practice with, suitable imaging / cloning software, which can cope with the possibility of some unreadable sectors on a source disk. You should do that practice with other disks, not the "live" ones which you will eventually use in step 3.

The subject of cloning, and discussions about various utilities, has been covered here in many previous threads. Your choice depends on several factors, which I've mentioned in some of those threads.

[donkeyass]

The subject of cloning, and discussions about various utilities, has been covered here in many previous threads. Your choice depends on several factors, which I've mentioned in some of those threads.

Okay I'll browse through those threads, thank you!

Well I used TestDisk to create an image of the entire partition and each of the disks. Right now I'm running Zero Assumption Recovery on the partition and it seems to be finding some stuff. I'll see how much I can get back like that then maybe try to rebuild the RAID with the images.

[Vulcan]

Well I used TestDisk to create an image of the entire partition and each of the disks.

I've never used TestDisk for cloning, nor seen it recommended in previous discussions on the forum, perhaps due to it's limited functionality e.g. (as far as I can see from a quick look at the code) its lack of logging. You just get a message at the end of the clone - which differs depending on whether read errors occurred or not, but no report of which sectors were affected, nor how many. Also it provides no control over retries etc.

It's interesting if you managed to clone all the disks successfully because, if you did that and if TestDisk reported no read errors, then why would the RAID controller have had any problems rebuilding the array, after you replaced a drive? If that Intel (fake) RAID controller / driver has any logging capability, I suggest that you review what the logs show.

[donkeyass]

Ya it seemed to create the image, I think it just used the dd Linux command.

then why would the RAID controller have had any problems rebuilding the array, after you replaced a drive?

Well the thing is, when I was rebuilding the RAID with the Intel tool under Windows, It said it was gonna take something like 60 hours. That sounded a bit weird to me so I went to check the RAID through Windows explorer and it showed the device as RAW and inaccessible so I stopped it because that scared the hell out of me since I could access my data on the degraded RAID.

If that Intel (fake) RAID controller / driver has any logging capability, I suggest that you review what the logs show.

Good idea, I'll see if the Intel software under Windows logged anything.