Messed up RAID10 mdadm - Need help

[jduartedj]

Hi,

So I realized I did many things I shouldn't have and now I'm trying to save my RAID10 with 6 x 3TB disks + 1 Spare 3TB. (Initally it was 4 x 3TB with 2 spares but I grew it after.)

After a series of fails (i don't know if related to disk or not) some disks were missing from the sytem, I rebooted and at one point the RAID was "recovering" (According to /proc/mdstat). For some reason at that point I ran mdadm --stop on it and the raid stopped. After reboot the raid wouldn't start. After some failed mdadm --assemble commands I used mdadm --create to re-create the raid, but it wouldn't mount (uknown fs or bad supoerblock). Realizing I didn't take into account the disk order I did it again, now in the order I thought was correct. It went into resync, so I waited. After the resync it still wouldn't mount. SO I'm using testdisk to scan for partitions in /dev/md0 in the hopes to find something. still 4% (it'll take a day) but no good news so far.

I don't have backups (yeah, I know...)
I also don't have any free 9TB space to create disk images.

I'm not a 100% sure of the disks order since the disks were initially removed and added a few times and even the spare was "switched" with another drive.
I'm not a 100% sure of other data such as offset or chunk size, I didn't use anything specific so I'm assuming default values for current version of mdadm and ext4 264192 and 512K.
I did try to find a valid superblock to run a fsck (e2fsck.ext4) but without luck, all the backup superblocks gave errors as "Bad magic number", after exausting the full list of backups I turned to testdisk.

I hope I don't have to turn to RAW data recovery (Assuming I can) and even so, only to recover the most important files.

Please, give me any kind of help on how to recover this:
How can I know the right disk order? can I trial and error it without disk writes?
Do you know any other tool that can read and/or detect the raid metadata or filesystem to recover it?
What should I do?

[Arch Stanton]

http://www.pclab.com.pt

[jduartedj]

Thank you for your suggestion, but I'm avoiding the use of such services.

I'm looking for solutions I can implement here, locally, without spending money on professional services.

[Arch Stanton]

ok. good luck.

[crashpcberlin]

Thank you for your suggestion, but I'm avoiding the use of such services.

I'm looking for solutions I can implement here, locally, without spending money on professional services.

If you really have messed up the disk order while doing some rebuild-action several hours or days, you can be happy if you get any functional data AT ALL.

I think, without recovery software there isn't anything you could do anymore.

Mount your Drive in a Windows-System and use raid recovery programs like reclaime raid recovery or ufs explorer, r-studio, etc. They only reading the drives and do not any writing.

Every writing attempt to the drives like you did already several times while you working on the original raid members on linux would cause more and more mess.

I would normally advise to do copys of all members first, but apparently it's already too late for this.

[jduartedj]

Thanks crashpcberlin.

Currently I'm running R-Studio, it works on Linux and apparently it has RAID10 parameters recovery/discovery. I know that I'll have to pay up if it does find anything.
Still every scan takes a day or two. Only good thing is I was able to find a faulty drive that was only reading at 4MB/s, this was probably one of the initial issues.

Also, I realize there was a full resync once, after the bad array was created so not sure what happened to the data there...
I believe that currently all RAID/array information is long gone but the data is possibly recoverable as long as I get the right order of disks, offset and chunk size. R-Studio also shows a lot of superblock sectors found, in places that I didn't find before. But that can lead up to over 40k combinations, if I include missing drives + all 7 drives.

I also have a last resort plan to run a script and create every combination of the array in readonly/assume-clean mode, that can be more than 40k combinations.

[lifeguarddubai]

i think its beyond DIY now better get expert help,

Good Luck jduartedj.....

[jduartedj]

That is just the thing, not sure it is worth it! The data's value could be more sentimental than anything, mostly archives and some media. It pains me to let it go, but I don't think it pains me 300$ or more worth of repair services.

[Arch Stanton]

That is just the thing, not sure it is worth it! The data's value could be more sentimental than anything, mostly archives and some media. It pains me to let it go, but I don't think it pains me 300$ or more worth of repair services.

$300 barely covers time required for disk-jockeying the drives to create disk-images/clones. Then there's the chance all your previous attempts screwed up any chances at recovery you had, so any pro taking the case runs the risk of getting paid nothing if the deal is no cure - no pay.

[jduartedj]

Well, that was just the price tag I saw online at a service, someone gave me a budget that is way more expensieve. It just makes my point, not really worth it!

[DRUG]

The simple fact pclab quoted your for 300€ is amazing.
I wouldn't touch that job for less then 1200€+VAT

[DRUG]

Then there's the chance all your previous attempts screwed up any chances at recovery

1200+500€ random RW for days fee.

[jduartedj]

Oh! No, that's wasn't a quote from PCLab, in fact it was a price off the pricelist in another website. The PCLab quote was 1000€ + VAT.