First time poster, not sure what protocol is for posting. Hope I'm not giving too much information with my issue.

-------------

I inherited a problem and am hoping to get some guidance. The situation is fubar'ed but no writes have been done to the disks in question.

The system has two arrays of drives. One is for the OS (CentOS 6), the other for Data. Here is the physical disk count on the machine:



The two smaller drives are mirrored as 300g - this is where the OS lives.
The rest of the drives were used for data storage and were also mirrored... I think. The RAID config was set up by someone who I don't have contact with anymore.

Here's how the drives are seen by linux right now:



Going by the sizes listed right above, could my data still exist on the data drives.. in some configuration or another?

The painful part is that there aren't any backups of the previous Volume config - just the web application and database (and those are stale since I didn't catch the original network issue until several weeks had passed).

The other problem is that I am unsure whether sdb/c were previously joined as one volume under LVM, or if they were used as separate volumes by the OS. The most recent fstab I had from the previous installation showed that /dev/sdb1 was mounted at /www. I am *think* the total drive space for /www was 1.8T. (As an aside I now know of vgcfgbackup and will make good use of it going forward.)

Back to today. I used dd to read what I could from these two volumes. Identical output was generated when reading from sdb1 and sdc1:

dd if=/dev/sdb1 bs=1M count=1 | strings -n 16 > https://gist.github.com/anonymous/d3de8a57c477e62c8eeb
In the dd output, "ursula2012" was the servername when the machine was originally build. "ursula2015" is the servername given yesterday. I'm unsure whether the dd output will be helpful or not.

I grabbed a screenshot just before confirming partition settings during the CentOS (re)install. Note that the Format option was not checked for the data drives.



After rebooting the OS loaded fine. But, I can't mount the data volume.



vgscan only shows the OS drive volume group, 'vg_ursula'.



lvscan only shows the OS drive logical volumes.


file -s /dev/sdb1

file -s /dev/sdc1


Here's full output from vgck -vvv: https://gist.github.com/anonymous/076cd514c42ec1d0d356

parted:


fsck:



I spun up testdrive last night to see what it showed. No luck seeing any files. The only useful information it gave was that sdc's size "seemed too small". Could the difference account for the data on the drive (several websites, databases, and a cache of videos)?



TL;DR: System has multiple drive arrays. OS was reinstalled on separate array. Data storage drives were not formatted or otherwise written to - but may have been previously joined under LVM. Linux sees them as empty drives, though the capacities indicate my data is still there.

Is the data recoverable?

Bit of an update.

I have what looks like pieces of previous volume configs. I'll post an example below.

When I went through the installer, I failed to set mount points for sdb and sdc. It so happens that the old Volume names carried over to the new installation on sda. I'm wondering whether that was what screwed the pooch.

Here's a snippet from one of the config texts I pulled from sdb using dd:

Would rstudio be useful in automating any of the recovery here? testdisk wasn't very helpful unfortunately.

Have you cloned the drives yet? The more you mess around with the originals, the higher the chances that you will be the reason that the data will never be recoverable.

On this post, it seems you have removed the part about it being very important to recover the data, as you stated on Reddit. Does this mean that the data is no longer important to recover?

_________________
Luke
Recovery Force Data Recovery

Images are being made now (sdb finished early this AM, sdc cloning now). I am not doing any writes or changes to VG configs until I have images of both sdb and sdc. The data is still important to recover.

Last night I went through 12 config files recovered using dd. Here are those files:

All in one:
https://gist.github.com/anonymous/aff98eba7b06db1834db

Separated:
lvm-6144: https://gist.github.com/anonymous/078e6fc021b63eee9c1c

lvm-8192: https://gist.github.com/anonymous/c2b9d1b4966b7c5aba02

lvm-10240: https://gist.github.com/anonymous/81393b9da32c8cae83f1

lvm-12800: https://gist.github.com/anonymous/c8a0e0b44c9d7601d23c

lvm-15360: https://gist.github.com/anonymous/c780af853b0b99ce893a

lvm-18432: https://gist.github.com/anonymous/9e5f6575f8f240b6745c

For the rest of these note that the creation_host and creation_time changed to Sept 23 - the day of the reinstall. These disks were not included in the partitioning and weren't formatted (and haven't been written to in any way since). I grabbed as screenshot of the partitioning screen right before clicking Next to finalize that part of install. Still, I don't know why I'm seeing dates of Sept 23 2015 in these configs.

lvm-22016 https://gist.github.com/anonymous/08f818c5799140f3f255

lvm-25088 https://gist.github.com/anonymous/4ea37727706e041d1a29

lvm-28160 https://gist.github.com/anonymous/59b1e39c7be2cceeec31

lvm-30720 https://gist.github.com/anonymous/9da2610672d0c72a1e77

lvm-32768 https://gist.github.com/anonymous/725b7626da94c96ca711

lvm-34816 https://gist.github.com/anonymous/c59fba434c4269a43e65

What kind of RAID were you using? Hardware RAID (where you have a dedicated RAID controller card), or software RAID?
In case of software RAID, there are two popular solutions: LVM over mdadm RAID, or RAID LVs over LVM. As I see, you were using the latter, but it's better to ask. In case you were using the mdadm way, however, you could possibly overwrite the mdadm header on the partition. Now it's clear that you have LVM metadata there, but what if there was mdadm metadata previously, what the CentOS installer overwritten?

The LVM metadata you posted were extracted from the LVM volumes you want to recover? Maybe a vgimport could do the job? (And yes, make an image of your disks before you try it!)

P.S.: there is no use to try to mount LVM PVs directly, like you did „mount -o ro /dev/sdb1 /www”. Only the LVM metadata is there, which can not be interpreted as a file system – only the LVM subsystem can use it.

acoder,
LVM recovery can be a bit tricky. If you still need help let me know. I do a lot of remote RAID recovery work.

http://freedatarecovery.us

-Stephen