All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 48 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: Corrupted File Finder
PostPosted: November 7th, 2018, 16:42 
Offline
User avatar

Joined: April 22nd, 2015, 20:32
Posts: 199
Location: Portugal
Dear DR expert,

Ever done a recovery to find out some files were corrupted? Even though MFT displays their name and size correctly? I did.

And it was frustrating to look file by file, just to have an estimate of good vs bad files to report to client.

So I decided to create a small software that will look for a determined percentage of 0's in the file hex and append a _corrupted label to the file.

Please let me know if this is something that you would use on your daily work.

_________________
BTC Wallet - 16S1yq41ehJr9Kh8GeA2cgSYLt397XKU4Q


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 7th, 2018, 17:39 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 11070
Location: Australia
Here is a command line example that recurses through the subdirectories of the Recovery folder and performs an integrity check of all ZIP files. The results are written to zip_tests.txt.

    for /r C:\Recovery_Root_Dir %f in (*.zip) do "c:\program files\7-zip\7z" t %f >> C:\zip_tests.txt

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 7th, 2018, 21:04 
Offline

Joined: December 8th, 2010, 11:37
Posts: 610
Location: Ottawa, Canada
Hi @DRUG,

Such a program would be helpful to me, but not on a daily basis. I assume you are selling this program. if so, what is the cost, please?

I've asked DFL (whose tools I use) to write such a program or to incorporate the function into their existing DE program. What I had asked, was to append ".bad" to the filename of any recovered file that contains one or more sectors full of nothing but the fill character (Bs rather than 0s, in DFL's case).

That fill character is written by DFL's imaging/cloning program when a sector on the patient drive is unreadable. The image or clone, from which I want to extract recovered data, will not give a read error of course, because it's a healthy clone drive/image file. Therefore, there is no way to flag files that encountered read errors during the imaging/cloning process without such a program or function. Your program sounds like it would be perfect for this.

Larry

_________________
Sabo Computer Repairs & Data Recovery


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 8th, 2018, 0:32 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 11070
Location: Australia
@LarrySabo, would this work for you?

    grep -r -l -U -e 'B{512,}' /recovery_root_dir > /badfiles.txt

I haven't tested it -- I just based it on my reading of the grep man page.

https://www.gnu.org/s/grep/manual/grep.html

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 8th, 2018, 8:34 
Offline
User avatar

Joined: April 22nd, 2015, 20:32
Posts: 199
Location: Portugal
@LarrySabo @fzabkar

I'm still improving the inner features, but as most of the forensic software the importing and parsing of files takes some time.

If this is to be sold and not distributed freely (only will be sold if I have development costs it will be on a simbolic price)

This is how it works:


Attachments:
2018-11-08_12-10-20.gif
2018-11-08_12-10-20.gif [ 13.57 MiB | Viewed 1790 times ]

_________________
BTC Wallet - 16S1yq41ehJr9Kh8GeA2cgSYLt397XKU4Q
Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 8th, 2018, 9:23 
Offline

Joined: December 8th, 2010, 11:37
Posts: 610
Location: Ottawa, Canada
fzabkar wrote:
@LarrySabo, would this work for you?...
I'll try it and report back. Thanks!

_________________
Sabo Computer Repairs & Data Recovery


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 8th, 2018, 9:29 
Offline

Joined: December 8th, 2010, 11:37
Posts: 610
Location: Ottawa, Canada
Thanks, @DRUG. Looks good so far. I would suggest allowing the "fill character" to be a user-definable variable, e.g., "B" instead of "0", and the tag to be an added extension of ".bad" rather than "_corrupted." An option to move all "bad" files to a separate folder would also be handy.

_________________
Sabo Computer Repairs & Data Recovery


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 8th, 2018, 11:44 
Offline
User avatar

Joined: April 22nd, 2015, 20:32
Posts: 199
Location: Portugal
LarrySabo wrote:
Thanks, @DRUG. Looks good so far. I would suggest allowing the "fill character" to be a user-definable variable, e.g., "B" instead of "0", and the tag to be an added extension of ".bad" rather than "_corrupted." An option to move all "bad" files to a separate folder would also be handy.



I'm already working on the fill character and the added extension. Moving the files may be a bit risky, but I'll check it out also.

Thank you for the feedback.

_________________
BTC Wallet - 16S1yq41ehJr9Kh8GeA2cgSYLt397XKU4Q


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 16th, 2018, 13:48 
Offline
User avatar

Joined: April 22nd, 2015, 20:32
Posts: 199
Location: Portugal
@LarrySabo

I've ended up compiling and coding your sugestions.


Attachments:
1.png
1.png [ 233.04 KiB | Viewed 1582 times ]
2.png
2.png [ 522.62 KiB | Viewed 1582 times ]

_________________
BTC Wallet - 16S1yq41ehJr9Kh8GeA2cgSYLt397XKU4Q
Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 16th, 2018, 20:59 
Offline

Joined: December 8th, 2010, 11:37
Posts: 610
Location: Ottawa, Canada
DRUG wrote:
@LarrySabo
I've ended up compiling and coding your sugestions.

Looks great. When and how can we get a copy to trial?

_________________
Sabo Computer Repairs & Data Recovery


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 17th, 2018, 9:45 
Offline
User avatar

Joined: April 22nd, 2015, 20:32
Posts: 199
Location: Portugal
@Pclab is testing the software ao i can fix eventual bugs and then i'll make a public release.

Perhaps I'll add a module to wipe disks and to make bye to byte copies on the same software to make it a more complete tool.

_________________
BTC Wallet - 16S1yq41ehJr9Kh8GeA2cgSYLt397XKU4Q


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 17th, 2018, 10:16 
Online
User avatar

Joined: December 19th, 2006, 8:49
Posts: 10265
Location: Portugal
Looks great !

_________________
1Q9xrDTzTddUXeJAFRn37aqh1Yr6buDCdw - (Bitcoin Donations)
paypal.me/Spildit - (PayPal Donations)
The HDD Oracle - Platform for OPEN research on Data Recovery.


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 19th, 2018, 16:58 
Offline

Joined: December 8th, 2010, 11:37
Posts: 610
Location: Ottawa, Canada
I would welcome an opportunity to test it, too. :) No need to add other features, in my opinion. If it performs it's primary function properly, it will be complete enough for most users, I would expect.

DRUG wrote:
@Pclab is testing the software ao i can fix eventual bugs and then i'll make a public release.

Perhaps I'll add a module to wipe disks and to make bye to byte copies on the same software to make it a more complete tool.

_________________
Sabo Computer Repairs & Data Recovery


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 19th, 2018, 19:07 
Offline
User avatar

Joined: July 12th, 2010, 4:38
Posts: 1109
Location: Portugal
Looks very promising.
Still have to glitches, but he will solve them for sure!

_________________
http://www.facebook.com/PCLAB.Assistencia.Tecnica
ACELab partner


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 19th, 2018, 19:29 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 11070
Location: Australia
LarrySabo wrote:
That fill character is written by DFL's imaging/cloning program when a sector on the patient drive is unreadable.

Does DFL maintain a log of bad sectors (like ddrescue)? If not, then does this mean that DFL cannot resume an interrupted cloning session?

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 19th, 2018, 21:14 
Offline

Joined: December 8th, 2010, 11:37
Posts: 610
Location: Ottawa, Canada
fzabkar wrote:
LarrySabo wrote:
That fill character is written by DFL's imaging/cloning program when a sector on the patient drive is unreadable.

Does DFL maintain a log of bad sectors (like ddrescue)? If not, then does this mean that DFL cannot resume an interrupted cloning session?

Yes, it does @fzabkar. I often use the ability to restart a cloning/imaging session; it picks up right where it left off. I have also used that ability to correct an oversight where I have started imaging to a smaller drive rather than my intended RAID storage array. I just stop the imaging job, copy the image/clone (a VHD) to the proper destination, edit the task's destination spec to refer to the correct (RAID) location, then restart the job, and select the existing task. Works a treat!

I'm in the middle of a recovery job right now where I did exactly that, followed by a selective file extraction job to extract the files from the clone. I could use @DRUG's program tomorrow when that file extraction job finishes, because there were lots of read errors sprinkled throughout the drive during cloning, even after three progressively more aggressive passes. Extracting files from the clone produces no read errors, of course.

_________________
Sabo Computer Repairs & Data Recovery


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 19th, 2018, 21:34 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 11070
Location: Australia
If DFL logs the bad sectors, then you should be able to determine which file they belong to. AIUI, that's one of the features offered by complementary utilities for ddrescue (or HDDSuperClone).

https://sourceforge.net/p/ddrutility/wiki/

Quote:
ddru_ntfsfindbad is a utility for NTFS partitions to find which files are related to bad sectors in a ddrescue log file.

ISTR that Microsoft or diskinternals has a utility for identifying the file which owns a particular NTFS cluster or sector. If you could make use of something like that, then it should be much faster than scanning the entire file system. Of course that would only work if the FS metadata and folder structure were intact.

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 19th, 2018, 22:09 
Offline

Joined: December 8th, 2010, 11:37
Posts: 610
Location: Ottawa, Canada
I have not attempted to analyze the map produced by DFL during the cloning process and have no idea how compatible it would be with those utilities/programs. There are many bad sectors scattered throughout the drive as well as skipped sectors between some of them (resulting in 8-bytes of inaccessible data each time). I would be happy to have a program such as @DRUG has developed scan the entire file system and flag those that are corrupted. All it takes is time, and little to no effort on my part.

The drive in question is a 2TB drive from a My Book containing TV show recordings, videos, music and pictures. Opening a random assortment of videos yielded no errors so I'm hopeful overall results will be "good enough." There's a limit to how much time and effort I'm able to dedicate to this job, given it's cost to the client. Plus, I'm lazy. :shock:

_________________
Sabo Computer Repairs & Data Recovery


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 19th, 2018, 22:26 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 11070
Location: Australia
Would you indulge my curiosity by uploading your map file, or excerpts from it, privacy concerns notwithstanding?

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: Corrupted File Finder
PostPosted: November 19th, 2018, 23:55 
Offline

Joined: December 8th, 2010, 11:37
Posts: 610
Location: Ottawa, Canada
I would if I could find anything that makes sense (and I could sanitize it for privacy) but the task map for the cloning task is a 3.64GB file containing blocks of numbers where huge sections will all contain the same number, more or less -- (e.g., hundreds of 03 followed by hundreds or 35, or whatever). No text data at all, save for passport data of the drive. I've looked for something more meaningful but am not finding anything that makes sense. I'll look some more tomorrow. It's time for bed here (10:51 PM).

_________________
Sabo Computer Repairs & Data Recovery


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 48 posts ]  Go to page 1, 2, 3  Next

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group