Corrupted File Finder

[DRUG]

Dear DR expert,

Ever done a recovery to find out some files were corrupted? Even though MFT displays their name and size correctly? I did.

And it was frustrating to look file by file, just to have an estimate of good vs bad files to report to client.

So I decided to create a small software that will look for a determined percentage of 0's in the file hex and append a _corrupted label to the file.

Please let me know if this is something that you would use on your daily work.

[fzabkar]

Here is a command line example that recurses through the subdirectories of the Recovery folder and performs an integrity check of all ZIP files. The results are written to zip_tests.txt.

for /r C:\Recovery_Root_Dir %f in (*.zip) do "c:\program files\7-zip\7z" t %f >> C:\zip_tests.txt

[LarrySabo]

Hi @DRUG,

Such a program would be helpful to me, but not on a daily basis. I assume you are selling this program. if so, what is the cost, please?

I've asked DFL (whose tools I use) to write such a program or to incorporate the function into their existing DE program. What I had asked, was to append ".bad" to the filename of any recovered file that contains one or more sectors full of nothing but the fill character (Bs rather than 0s, in DFL's case).

That fill character is written by DFL's imaging/cloning program when a sector on the patient drive is unreadable. The image or clone, from which I want to extract recovered data, will not give a read error of course, because it's a healthy clone drive/image file. Therefore, there is no way to flag files that encountered read errors during the imaging/cloning process without such a program or function. Your program sounds like it would be perfect for this.

Larry

[fzabkar]

@LarrySabo, would this work for you?

grep -r -l -U -e 'B{512,}' /recovery_root_dir > /badfiles.txt

I haven't tested it -- I just based it on my reading of the grep man page.

https://www.gnu.org/s/grep/manual/grep.html

[DRUG]

@LarrySabo @fzabkar

I'm still improving the inner features, but as most of the forensic software the importing and parsing of files takes some time.

If this is to be sold and not distributed freely (only will be sold if I have development costs it will be on a simbolic price)

This is how it works:

Attachments

[LarrySabo]

@LarrySabo, would this work for you?...

I'll try it and report back. Thanks!

[LarrySabo]

Thanks, @DRUG. Looks good so far. I would suggest allowing the "fill character" to be a user-definable variable, e.g., "B" instead of "0", and the tag to be an added extension of ".bad" rather than "_corrupted." An option to move all "bad" files to a separate folder would also be handy.

[DRUG]

Thanks, @DRUG. Looks good so far. I would suggest allowing the "fill character" to be a user-definable variable, e.g., "B" instead of "0", and the tag to be an added extension of ".bad" rather than "_corrupted." An option to move all "bad" files to a separate folder would also be handy.

I'm already working on the fill character and the added extension. Moving the files may be a bit risky, but I'll check it out also.

Thank you for the feedback.

[DRUG]

@LarrySabo

I've ended up compiling and coding your sugestions.

Attachments

[LarrySabo]

@LarrySabo
I've ended up compiling and coding your sugestions.

Looks great. When and how can we get a copy to trial?

[DRUG]

@Pclab is testing the software ao i can fix eventual bugs and then i'll make a public release.

Perhaps I'll add a module to wipe disks and to make bye to byte copies on the same software to make it a more complete tool.

[LarrySabo]

I would welcome an opportunity to test it, too. No need to add other features, in my opinion. If it performs it's primary function properly, it will be complete enough for most users, I would expect.

@Pclab is testing the software ao i can fix eventual bugs and then i'll make a public release.

Perhaps I'll add a module to wipe disks and to make bye to byte copies on the same software to make it a more complete tool.

[pclab]

Looks very promising.
Still have to glitches, but he will solve them for sure!

[fzabkar]

That fill character is written by DFL's imaging/cloning program when a sector on the patient drive is unreadable.

Does DFL maintain a log of bad sectors (like ddrescue)? If not, then does this mean that DFL cannot resume an interrupted cloning session?

[LarrySabo]

That fill character is written by DFL's imaging/cloning program when a sector on the patient drive is unreadable.

Does DFL maintain a log of bad sectors (like ddrescue)? If not, then does this mean that DFL cannot resume an interrupted cloning session?

Yes, it does @fzabkar. I often use the ability to restart a cloning/imaging session; it picks up right where it left off. I have also used that ability to correct an oversight where I have started imaging to a smaller drive rather than my intended RAID storage array. I just stop the imaging job, copy the image/clone (a VHD) to the proper destination, edit the task's destination spec to refer to the correct (RAID) location, then restart the job, and select the existing task. Works a treat!

I'm in the middle of a recovery job right now where I did exactly that, followed by a selective file extraction job to extract the files from the clone. I could use @DRUG's program tomorrow when that file extraction job finishes, because there were lots of read errors sprinkled throughout the drive during cloning, even after three progressively more aggressive passes. Extracting files from the clone produces no read errors, of course.

[fzabkar]

If DFL logs the bad sectors, then you should be able to determine which file they belong to. AIUI, that's one of the features offered by complementary utilities for ddrescue (or HDDSuperClone).

https://sourceforge.net/p/ddrutility/wiki/

ddru_ntfsfindbad is a utility for NTFS partitions to find which files are related to bad sectors in a ddrescue log file.

ISTR that Microsoft or diskinternals has a utility for identifying the file which owns a particular NTFS cluster or sector. If you could make use of something like that, then it should be much faster than scanning the entire file system. Of course that would only work if the FS metadata and folder structure were intact.

[LarrySabo]

I have not attempted to analyze the map produced by DFL during the cloning process and have no idea how compatible it would be with those utilities/programs. There are many bad sectors scattered throughout the drive as well as skipped sectors between some of them (resulting in 8-bytes of inaccessible data each time). I would be happy to have a program such as @DRUG has developed scan the entire file system and flag those that are corrupted. All it takes is time, and little to no effort on my part.

The drive in question is a 2TB drive from a My Book containing TV show recordings, videos, music and pictures. Opening a random assortment of videos yielded no errors so I'm hopeful overall results will be "good enough." There's a limit to how much time and effort I'm able to dedicate to this job, given it's cost to the client. Plus, I'm lazy.

[fzabkar]

Would you indulge my curiosity by uploading your map file, or excerpts from it, privacy concerns notwithstanding?

[LarrySabo]

I would if I could find anything that makes sense (and I could sanitize it for privacy) but the task map for the cloning task is a 3.64GB file containing blocks of numbers where huge sections will all contain the same number, more or less -- (e.g., hundreds of 03 followed by hundreds or 35, or whatever). No text data at all, save for passport data of the drive. I've looked for something more meaningful but am not finding anything that makes sense. I'll look some more tomorrow. It's time for bed here (10:51 PM).

[fzabkar]

I was thinking that @DRUG might be able to make sense of DFL's map file, in which case he could write code specifically for you. AISI, all one would need to do would be to convert DFL's maps to ddrescue's log format, then run ddrutility against the log. BTW, the author of ddrutility is @maximus.