Recovering data from RAID (and possibly LVM)

[acoder]

First time poster, not sure what protocol is for posting. Hope I'm not giving too much information with my issue.

-------------

I inherited a problem and am hoping to get some guidance. The situation is fubar'ed but no writes have been done to the disks in question.

The system has two arrays of drives. One is for the OS (CentOS 6), the other for Data. Here is the physical disk count on the machine:

Code:

#   Description                                     Total Gigs
2   HARD DRIVE, 300GB, SAS6, 10, 2.5, H-CE, E/C      600
6   HARD DRIVE, 600G, SAS6, 10, 2.5, W-SIR, E/C     3600


The two smaller drives are mirrored as 300g - this is where the OS lives.
The rest of the drives were used for data storage and were also mirrored... I think. The RAID config was set up by someone who I don't have contact with anymore.

Here's how the drives are seen by linux right now:

Code:

[root@ursula ~]# pvscan
  PV /dev/sda5   VG vg_ursula   lvm2 [276.34 GiB / 0    free]
  PV /dev/sdb1                  lvm2 [1.80 TiB]
  PV /dev/sdc1                  lvm2 [948.67 GiB]
  Total: 3 [3.00 TiB] / in use: 1 [276.34 GiB] / in no VG: 2 [2.73 TiB]

Going by the sizes listed right above, could my data still exist on the data drives.. in some configuration or another?

The painful part is that there aren't any backups of the previous Volume config - just the web application and database (and those are stale since I didn't catch the original network issue until several weeks had passed).

The other problem is that I am unsure whether sdb/c were previously joined as one volume under LVM, or if they were used as separate volumes by the OS. The most recent fstab I had from the previous installation showed that /dev/sdb1 was mounted at /www. I am *think* the total drive space for /www was 1.8T. (As an aside I now know of vgcfgbackup and will make good use of it going forward.)

Back to today. I used dd to read what I could from these two volumes. Identical output was generated when reading from sdb1 and sdc1:

dd if=/dev/sdb1 bs=1M count=1 | strings -n 16 > https://gist.github.com/anonymous/d3de8a57c477e62c8eeb
In the dd output, "ursula2012" was the servername when the machine was originally build. "ursula2015" is the servername given yesterday. I'm unsure whether the dd output will be helpful or not.

I grabbed a screenshot just before confirming partition settings during the CentOS (re)install. Note that the Format option was not checked for the data drives.



After rebooting the OS loaded fine. But, I can't mount the data volume.

Code:

# mount -o ro /dev/sdb1 /www
mount: unknown filesystem type 'LVM2_member'

vgscan only shows the OS drive volume group, 'vg_ursula'.

Code:

  Reading all physical volumes.  This may take a while...
  Found volume group "vg_ursula" using metadata type lvm2

lvscan only shows the OS drive logical volumes.

Code:

  ACTIVE            '/dev/vg_ursula/LogVol05' [29.30 GiB] inherit
  ACTIVE            '/dev/vg_ursula/LogVol04' [48.83 GiB] inherit
  ACTIVE            '/dev/vg_ursula/LogVol03' [48.83 GiB] inherit
  ACTIVE            '/dev/vg_ursula/lv_root' [111.74 GiB] inherit
  ACTIVE            '/dev/vg_ursula/lv_home' [9.77 GiB] inherit
  ACTIVE            '/dev/vg_ursula/lv_swap' [27.89 GiB] inherit

file -s /dev/sdb1

Code:

/dev/sdb1: LVM2 (Linux Logical Volume Manager) , UUID: B1bLeFveeDcnfZ2i0tuqWtHgSd6UAgM

file -s /dev/sdc1

Code:

/dev/sdc1: LVM2 (Linux Logical Volume Manager) , UUID: SMMVLUKEuBPHuTeoarMkDAlJDDY1Gm2

Here's full output from vgck -vvv: https://gist.github.com/anonymous/076cd514c42ec1d0d356

parted:

Code:

[root@ursula ~]# parted /dev/sdb 'print'
Model: DELL PERC H710 (scsi)
Disk /dev/sdb: 1979GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
1      1049kB  1979GB  1979GB  primary               lvm

[root@ursula ~]# parted /dev/sdc 'print'
Model: DELL PERC H710 (scsi)
Disk /dev/sdc: 1019GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
1      1049kB  1019GB  1019GB  primary               lvm


fsck:

Code:

[root@ursula ~]# fsck /dev/sdb1
fsck from util-linux-ng 2.17.2
fsck: fsck.LVM2_member: not found
fsck: Error 2 while executing fsck.LVM2_member for /dev/sdb1
[root@ursula ~]# fsck /dev/sdc1
fsck from util-linux-ng 2.17.2
fsck: fsck.LVM2_member: not found
fsck: Error 2 while executing fsck.LVM2_member for /dev/sdc1

I spun up testdrive last night to see what it showed. No luck seeing any files. The only useful information it gave was that sdc's size "seemed too small". Could the difference account for the data on the drive (several websites, databases, and a cache of videos)?



TL;DR: System has multiple drive arrays. OS was reinstalled on separate array. Data storage drives were not formatted or otherwise written to - but may have been previously joined under LVM. Linux sees them as empty drives, though the capacities indicate my data is still there.

Is the data recoverable?

[acoder]

Bit of an update.

I have what looks like pieces of previous volume configs. I'll post an example below.

When I went through the installer, I failed to set mount points for sdb and sdc. It so happens that the old Volume names carried over to the new installation on sda. I'm wondering whether that was what screwed the pooch.

Here's a snippet from one of the config texts I pulled from sdb using dd:

Code:

LogVol05 {
id = "ZdFwA8-lj4Y-XwD0-hFMB-iHf8-kZdo-VfJy89"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_host = "ursula2012"
creation_time = 1378846949
segment_count = 2

segment1 {
start_extent = 0
extent_count = 471858

type = "striped"
stripe_count = 1    # linear

stripes = [
"pv1", 0
]
}
segment2 {
start_extent = 471858
extent_count = 225240

type = "striped"
stripe_count = 1    # linear

stripes = [
"pv2", 0
]
}
}
id = "Ex8O6x-y9bt-nDDZ-tqfM-L5hG-GKPM-O1YcsS"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_host = "ursula2012"
creation_time = 1378847075
segment_count = 1

segment1 {
start_extent = 0
extent_count = 15000

type = "striped"
stripe_count = 1    # linear

stripes = [
"pv0", 0
]
}
}

LogVol03 {
id = "o2RB9c-Idd9-WGCT-54o4-e2ZZ-EtBn-q8s9Vo"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_host = "ursula2012"
creation_time = 1378847083
segment_count = 1

segment1 {
start_extent = 0
extent_count = 15000

type = "striped"
stripe_count = 1    # linear

stripes = [
"pv0", 15000
]
}
}
LogVol02 {
id = "Szb39M-Kp37-j5di-kY9I-g2VV-SOkk-oMj1d0"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_host = "ursula2012"
creation_time = 1378847092
segment_count = 1

segment1 {
start_extent = 0
extent_count = 37500

type = "striped"
stripe_count = 1    # linear

stripes = [
"pv0", 30000
]
}
}
}
}
# Generated by LVM2 version 2.02.98(2)-RHEL6 (2012-10-15): Tue Sep 10 17:04:52 2013

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "ursula2012"    # Linux ursula2012 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64
creation_time = 1378847092  # Tue Sep 10 17:04:52 2013

[acoder]

Would rstudio be useful in automating any of the recovery here? testdisk wasn't very helpful unfortunately.

[lcoughey]

Have you cloned the drives yet? The more you mess around with the originals, the higher the chances that you will be the reason that the data will never be recoverable.

On this post, it seems you have removed the part about it being very important to recover the data, as you stated on Reddit. Does this mean that the data is no longer important to recover?

[acoder]

Have you cloned the drives yet? The more you mess around with the originals, the higher the chances that you will be the reason that the data will never be recoverable.

On this post, it seems you have removed the part about it being very important to recover the data, as you stated on Reddit. Does this mean that the data is no longer important to recover?

Images are being made now (sdb finished early this AM, sdc cloning now). I am not doing any writes or changes to VG configs until I have images of both sdb and sdc. The data is still important to recover.

Last night I went through 12 config files recovered using dd. Here are those files:

All in one:
https://gist.github.com/anonymous/aff98eba7b06db1834db

Separated:
lvm-6144: https://gist.github.com/anonymous/078e6fc021b63eee9c1c

lvm-8192: https://gist.github.com/anonymous/c2b9d1b4966b7c5aba02

lvm-10240: https://gist.github.com/anonymous/81393b9da32c8cae83f1

lvm-12800: https://gist.github.com/anonymous/c8a0e0b44c9d7601d23c

lvm-15360: https://gist.github.com/anonymous/c780af853b0b99ce893a

lvm-18432: https://gist.github.com/anonymous/9e5f6575f8f240b6745c

For the rest of these note that the creation_host and creation_time changed to Sept 23 - the day of the reinstall. These disks were not included in the partitioning and weren't formatted (and haven't been written to in any way since). I grabbed as screenshot of the partitioning screen right before clicking Next to finalize that part of install. Still, I don't know why I'm seeing dates of Sept 23 2015 in these configs.

lvm-22016 https://gist.github.com/anonymous/08f818c5799140f3f255

lvm-25088 https://gist.github.com/anonymous/4ea37727706e041d1a29

lvm-28160 https://gist.github.com/anonymous/59b1e39c7be2cceeec31

lvm-30720 https://gist.github.com/anonymous/9da2610672d0c72a1e77

lvm-32768 https://gist.github.com/anonymous/725b7626da94c96ca711

lvm-34816 https://gist.github.com/anonymous/c59fba434c4269a43e65

[MegaBrutal]

What kind of RAID were you using? Hardware RAID (where you have a dedicated RAID controller card), or software RAID?
In case of software RAID, there are two popular solutions: LVM over mdadm RAID, or RAID LVs over LVM. As I see, you were using the latter, but it's better to ask. In case you were using the mdadm way, however, you could possibly overwrite the mdadm header on the partition. Now it's clear that you have LVM metadata there, but what if there was mdadm metadata previously, what the CentOS installer overwritten?

The LVM metadata you posted were extracted from the LVM volumes you want to recover? Maybe a vgimport could do the job? (And yes, make an image of your disks before you try it!)

P.S.: there is no use to try to mount LVM PVs directly, like you did „mount -o ro /dev/sdb1 /www”. Only the LVM metadata is there, which can not be interpreted as a file system – only the LVM subsystem can use it.

[veedar]

acoder,
LVM recovery can be a bit tricky. If you still need help let me know. I do a lot of remote RAID recovery work.

http://freedatarecovery.us

-Stephen