Combining/Merging HDD Images

[crytoy]

Hi there,
I have created 2 disk images of my HDD with different cloning settings in each one.
In the first HDD Image: I have set the the program cloning the HDD to a "skip factor of 100", which means every time it encounters a bad sector while imaging, it should skip 100 sectors ahead.
In the second HDD Image: I have set the program cloning the HDD to a "skip factor of 10", which means every time it encounters a bad sector while imaging, it should skip 10 sectors ahead.

My question, is there a way to merge those 2 HDD Images together?
So that maximum data is extracted from a HDD with many bad sectors.
What HDD image extensions can I merge, VHD RAW IMG IMA DD .....?






Please Help,
Thank you

[fzabkar]

Which tool(s) did you use, and did these tools maintain a log?

How do bad sectors appear in the image? Are they filled with zeros, or do they contain some other identifying characters (eg "BAD" or "DEAD")?

[lcoughey]

Although I would have just made one clone using professional data recovery tools, here is how I'd deal with your situation.

Step 1 - Clone or copy first image to a new location (this copy will be the first part of the merge)
Step 2 - use ddrescue to generate a log file against the this copy (assuming that the skipped sectors contain zeroes)
Step 3 - use ddrescue with the log file to copy the second image clone to the image copy created in step 1 (this should only copy the sectors that contain zeros on the first copy

If you need to ask for more details about how to do this, I suggest that you are over your head and might want to seek the assistance of a pro.

[fzabkar]

Although I would have just made one clone using professional data recovery tools, here is how I'd deal with your situation.

Step 1 - Clone or copy first image to a new location (this copy will be the first part of the merge)
Step 2 - use ddrescue to generate a log file against the this copy (assuming that the skipped sectors contain zeroes)

How will ddrescue be able to differentiate between those sectors in the image which contain recovered data from those that contain zero-fills?

AFAICS, ddrescue will just copy the entire image as is, and the log file will register 100% success.

[mr_spokk]

Clone 50% of image 1 and then the rest of 50% from image 2.

[fzabkar]

Clone 50% of image 1 and then the rest of 50% from image 2.

I suspect that the OP's diagram may not reflect the actual damage. Instead I'm assuming the diagram is just for illustrative purposes, and I suspect that the damage is spread out over the entire drive.

Assuming that bad sectors are zero-filled, I would try to find some way to OR both images before writing them to a new destination.

[digitalferret]

Hi crytoy,

just to note your diagram is not a great representation although i get your idea.
If image 1 got so far, say 70%, without discovering bad sectors, image 2 would also get to that stage.
The 100 is not a granularity. its a skip size at an error.
So, image 1 would skip forward 100 only when it happens upon an error, and image 2 would crawl along by 10s from that point.
see reworked image.

Any overlap where image 1 would have data and image 2 would not, would surely be a matter of a few single sectors on the boundary of one to the other only.
Further, the skipped areas of size 10 sectors would remain unless you drop down to a skip of 1 (in reverse) without going into details of disabling look-ahead caching.
You should then be left with the smallest of errors which should be the bad blocks themselves.

Anyways, back to the questions:
can you merge 2 images? yes, with conditions.

echoing & acknowledging fzabkar,
which tool(s)have you used? & did it/you specify to make logs?
& (If you had blank targets you could try a logical OR operation to merge the 2)
& (a merge via a copy of the copy via ddrescue would fail AIUI as it skips errors not zeros)

Usually a single image is created as multi-pass onto a blanked target drive.

first pass uses larger skip size in order to bypass larger areas of corruption / damage.
second pass if needed would use smaller steps to fill in the gaps, and maybe do a reverse pass too.
If the quick(er) passes allow you to retrieve the required files, then imaging the rest of the drive is academic and a waste of time.

If you have successfully made an image with smaller skip sizes, wouldn't that make the one with larger skips mainly redundant anyway?

One would not normally make 2 different images unless they were from different devices with identical content. ie CD copies

can i ask, is this a technical exercise or do you actually have files to recover?

K

Attachments

reworked

mlg6a.jpg (50.03 KiB) Viewed 17863 times

[HaQue]

better off taking ta a DR shop and asking for them to take an image with a hardware imager?

[crytoy]

Please bear in mind that this is a Technical Exercise.

In the situation of a failing HDD with many bad sectors, if trying to image the HDD we must stress it gradually.
The need of multiple imaging pass rises from fear of the HDD's death, because: the faster a failing HDD is stressed, the faster it dies.
That's why first we need to extract the good and easy data quickly without putting any stress on the HDD, and then secondly re-read the sectors and try to extract as much data a possible.
The second image retries with the skipped sectors and tries to extract data from the bad sectors.





So Finally my repeated question rises: if I image an hdd using ddrescue with high bad sector skip factor in the first image, and low bad sector skip factor in the second image, can I compare both hdd images and merge/combine them together.
Can I integrate both High & Low bad sector skip factor

Which tool(s) did you use, and did these tools maintain a log?

I am using "DDrescue" , and I also have "Diskpatch" and "zero assumption recovery", all with the capability to modify the skip factor for bad sectors.
DDrescue and DiskPatch both have logs.

digitalferret you are a genius, lcoughey you are intelligent, fzabkar you are smart

[labtech]

Assuming that bad sectors are zero-filled, I would try to find some way to OR both images before writing them to a new destination.

OR function works as long as the zero filled assumption is true.

[digitalferret]

crytoy, thanks for clarifications but to ask:
when you say you have taken 2 images, are you meaning 2 images one laid over the other on the same target drive
or
2 images written on 2 separate disks?

If it's the first, then ddrescue, with a log will be doing the merge for you.

If it's that latter, just use ddrescue as it was designed: 1 source drive, one target drive and it will lay multiple images over each other on the same target drive as intended.

It will image the disk using a pass with the large skip first and lay the second pass using smaller skips down on top of that using its own log system. read the link page for detailed description.
It isn't necessary to make 2 separate target images in fact it actually makes the process more difficult.

Despite that if now you have 2 distinct separate images you can also use ddrescue as folows:

read this
https://www.gnu.org/software/ddrescue/m ... anual.html
and also scroll down to

Example 5: Merge the partially recovered images of 3 identical DVDs using their logfiles as domain logfiles.

ddrescue -m logfile1 dvdimage1 dvdimage logfile
ddrescue -m logfile2 dvdimage2 dvdimage logfile
ddrescue -m logfile3 dvdimage3 dvdimage logfile
(if errsize is zero, dvdimage now contains a complete image of the DVD
and you can write it to a blank DVD)

you could aslo try
http://www.deepspar.com/pdf/DeepSparDis ... paper3.pdf
for info

zero-filled:
ddrescue does not write zero's to the target drive in rescue mode in substitution for bad sectors (It leaves them blank to be written on the next pass)
A pre blanked target drive will thus show zeros where no data have been written.

It can however be used in fill-mode to pre load a target drive with a predetermined pattern such that bad areas can easily be identified in hex ed after rescue mode runs (ANSI/OEM columns)
ie compare
"00000000BADBADBADBAD " zeros and bads - identifiable.
with
"0000000000000000..." zeros and bads - cant tell

anyways, easier to read the links i pasted.
good luck and congrats for an active interest. be as nosey as you can
hth
K

[lcoughey]

How will ddrescue be able to differentiate between those sectors in the image which contain recovered data from those that contain zero-fills?
It doesn't know if the zero file is read vs unread. But, it shouldn't matter because if it is zeros that were read, overwriting it zeros from the other drive won't matter and if the other drive can't read those sectors, they will still be zeros.

Drive 1 - 123456789000098765xxxx
Drive 2 - xxxx567890000987654321

Log of 1 - 123456789xxxx98765xxxx
Merge with Drive 2 -1234567890000987654321

x = unread sector
green = sectors read a second time

AFAICS, ddrescue will just copy the entire image as is, and the log file will register 100% success.

Does the above help clarify?

[lcoughey]

So Finally my repeated question rises: if I image an hdd using ddrescue with high bad sector skip factor in the first image, and low bad sector skip factor in the second image, can I compare both hdd images and merge/combine them together.
Can I integrate both High & Low bad sector skip factor

If you are using ddrescue, why are you creating two images with the intent to merge. With the use of a log file, you can keep focusing on the unread sectors, continually tweaking the settings until you get the cleanest image possible. Just change the number of retries, direction and block size until and you should be able to get the majority of sectors with a single image file.

Which tool(s) did you use, and did these tools maintain a log?

I am using "DDrescue" , and I also have "Diskpatch" and "zero assumption recovery", all with the capability to modify the skip factor for bad sectors.

digitalferret you are a genius, lcoughey you are intelligent, fzabkar you are smart

[digitalferret]

Luke:
i think i get where you are coming from but if i understand correctly the logfile isn't as extensive
as that.
looking here:
https://www.gnu.org/software/ddrescue/m ... -structure

imaging a good drive even with a partial image with zeros or gibberish for blocks
wouldn't it return a + in the log. ie it was read correctly regardless of content?

got an itch i can't scratch now lol. will boot a 'nix box this even and do a run if i can.

cheers
Kern

[fzabkar]

How will ddrescue be able to differentiate between those sectors in the image which contain recovered data from those that contain zero-fills?
It doesn't know if the zero file is read vs unread. But, it shouldn't matter because if it is zeros that were read, overwriting it zeros from the other drive won't matter and if the other drive can't read those sectors, they will still be zeros.

Drive 1 - 123456789000098765xxxx
Drive 2 - xxxx567890000987654321

Log of 1 - 123456789xxxx98765xxxx
Merge with Drive 2 -1234567890000987654321

x = unread sector
green = sectors read a second time

AFAICS, ddrescue will just copy the entire image as is, and the log file will register 100% success.

Does the above help clarify?

All that is fine ... as long as ddrescue is able to refer to a log, and assuming that the log is in a format that ddrescue is able to understand. The OP's post made no mention of any log, or even which tool was used, hence my questions. In the absence of same, your method would be useless. In fact the OP's question was of a general nature, referring to image files in several different formats. All that was known was that there were two files that needed to be merged. In the absence of a log file, the only possible solution would be to OR the two files, and that would only be possible if the bad sectors were zero filled.

In fact here is an example in a recent thread where a person imaged the same drive twice, but ended up without a log on each occasion:
viewtopic.php?f=7&t=28015

[lcoughey]

In the original case stated, I'm assuming that the image files were created with something other than ddrescue where no log is present...even if it were created with ddrescue with a log, I'm suggesting that the user use ddrescue to generate a fresh log, based on the partial image and then use that log to complete the image using the other partial image. In theory, it should produce the exact same result as using the OR algorithm, but in both cases, it is necessary that the unread sectors are zeros in the image files. Otherwise, there is no way to differentiate between data from the original and data that was already there.

The command to generate the log file is as follows:

ddrescue --generate-mode infile outfile logfile

[fzabkar]

@lcoughey, in the absence of a log file, and in the absence of a file system analysis, there would be no way for you or any tool to determine whether a particular sector in an image file contained recovered data or otherwise. Assuming the image file were written to a good drive, then it would contain no unreadable sectors, in which case ddrescue would make a perfect copy of it on a third destination. Without a log file, ddrescue would have no way of knowing if sector X in the image file were zero filled, or contained a "BAD" signature, or if it simply contained the stale data that existed on the drive before the image file was written. All readable sectors in the file would be imaged to the third destination as is, regardless of their contents. Therefore the new log file would register 100% success, ie it would be a pointless exercise.

My suggestion would be to compare the two image files and examine the differences. This may help to identify the fill patterns, if any. If neither image contains fill patterns, and if unrecovered sectors simply contain the stale data that existed on the destination, then there would be no simple way to merge the two images.

[crytoy]

Thanks all for your help, my question was answered
by the way I have created a windows version of DDrescue using cygwin if anyone wants.

Attachments

ddrescue-1.18-pre7.rar

DDrescue windows version

(1.14 MiB) Downloaded 477 times

[lcoughey]

@lcoughey, in the absence of a log file, and in the absence of a file system analysis, there would be no way for you or any tool to determine whether a particular sector in an image file contained recovered data or otherwise. Assuming the image file were written to a good drive, then it would contain no unreadable sectors, in which case ddrescue would make a perfect copy of it on a third destination. Without a log file, ddrescue would have no way of knowing if sector X in the image file were zero filled, or contained a "BAD" signature, or if it simply contained the stale data that existed on the drive before the image file was written. All readable sectors in the file would be imaged to the third destination as is, regardless of their contents. Therefore the new log file would register 100% success, ie it would be a pointless exercise.

I've used the auto generate to create a log file many times over...it has always worked. But, I always write to a blank drive and don't write any patterns to the unread sectors. If you wrote a pattern to the unread sectors (ie, unread), you would need to add another step of searching those sectors our and overwriting them with zeros....then generate the log.

My suggestion would be to compare the two image files and examine the differences. This may help to identify the fill patterns, if any.

I think the whole point of the exercise is to automate the process, rather than manually do it.

If neither image contains fill patterns, and if unrecovered sectors simply contain the stale data that existed on the destination, then there would be no simple way to merge the two images.

This is true.

[lcoughey]

Thanks all for your help, my question was answered
by the way I have created a windows version of DDrescue using cygwin if anyone wants.

ddrescue is an installable app within the cygwin setup application now.