Any interest in a ddrescue script to find bad files?

[maximus]

I am working on a linux bash script for ddrescue that will use the log file from a whole drive recovery and, assuming the file system is intact enough, will list all files with bad sectors for both NTFS and Linux EXT2/3/4 partitions on the drive. I have an alpha/beta version that appears to actually work. I am just doing this for myself, but was wondering if there might be interest in it. Interest or lack thereof might influence the direction I take from here...

[fzabkar]

I may not use it personally, but I'll make others aware of it. I participate in several storage forums, so I can see the need for something like this. Many thanks in advance.

[cat1x]

Do you know ddrescue-checkrescuedfiles? A bash script writted by a ubuntu user for checking checking for damaged files after rescuing

http://ubuntuforums.org/showthread.php?t=1860559

I don't tested it but it can help you. This is the code:

Code:

#!/bin/sh
# ddrescue-checkrescuedfiles

# Scans the results of a ddrescue rescue to find which files might be damaged
#  because of having bad-sector blocks in them which weren't able to be rescued.
# This script follows the algorithm described in the GNU ddrescue manual,
#  Section 8 "Fill Menu".
# http://www.gnu.org/s/ddrescue/manual/ddrescue_manual.html#Fill-Mode
# Note that the files which this script discovers as having bad-sectors will
#  in many cases still be perfectly usable, they may only have one or two bad
#  sectors in them. In the case of MP3s, JPGs, videos, etc, you may not even
#  notice. This script will just give you some information on which ones may
#  have errors in them.

# Before using this script, you must have rescued as much as you can using
#  ddrescue. Make sure when doing these initial run/s of ddrescue to use a log
#  file and not to use sparse writes.

# This script needs to know the path to the device or image containing rescued
#  data (RESCUED_DEVICE), the path to the ddrescue log file (DDRESCUE_LOGFILE),
#  as well as the type of filesystem the rescued device was using.
# It also needs an empty directory to use as a mount point for the device/image
#  containing rescued data (completely arbitrary, just needs to exist and be
#  writable).
# These parameters must be set within this file, by editing
#  SCRIPT PARAMETERS, below.

# Example usage, rescue from /dev/sdb1 to ~/RESCUED_IMAGE:
# First rescue as much data from /dev/sdb1 as you can, using ddrescue:
# $ ddrescue -n /dev/sdb1 ~/RESCUED_IMAGE ~/DDRESCUE_LOGFILE
# $ ddrescue -d -r3 /dev/sdb1 ~/RESCUED_IMAGE ~/DDRESCUE_LOGFILE
# Then, edit this script to set RESCUED_DEVICE to "~/RESCUED_IMAGE"
#  and DDRESCUE_LOGFILE to "~/DDRESCUE_LOGFILE" (as well as setting correct
#  filesystem and mount point) and run the script.

# Results will be printed to stdout and will look like, e.g.
#  /path/to/file1: OK
#  /path/to/file2: FAILED
# Note that FAILED does not necessarily mean file is bad, it just means there
#  are some bad blocks in it. OK means that there are no faulty blocks in the
#  file.

# Results, as described above, will be output to stdout, while script status
#  messages and errors will be output on stderr.
# This means you can capture the results without the status messages and errors
# e.g. run 'ddrescue-checkrescuedfiles > output'
# and you will still see " * Mounting rescued filesystem..." messages, etc. on
# terminal, but "/path/to/file1: OK" results will go to a file called "output".

# SCRIPT PARAMETERS
# *** SET THESE! **************************************************************
FILESYSTEM="ext3" # Type of filesystem we are working with (for rescued medium)
RESCUED_DEVICE="/path/to/rescued-device" # Path to device or image
DDRESCUE_LOGFILE="/path/to/ddrescue-logfile" # Path to ddrescue log file
RESCUED_MOUNT_POINT="/mount/point" # Directory to mount device or image in
    # (must exist and be writable but could be anywhere)
#******************************************************************************

# Check script was run as root
test "`id -u`" != 0 && { echo "This script must be run as root" >&2; exit 1; }

# Arrange a temp dir to work in
# (filename is "<script base name><script PID>" plus optionally "-" and a number
    # if dir already exists)
TMPDIR="/tmp/`basename "$0"`$$"
TMPDIRBASE="$TMPDIR"; i=1
until test ! -e "$TMPDIR"; do TMPDIR="$TMPDIRBASE-$i"; i=`expr $i + 1`; done
mkdir "$TMPDIR"

# Mount rescued file system
echo " * Mounting rescued filesystem..." >&2
mount -t "$FILESYSTEM" "$RESCUED_DEVICE" "$RESCUED_MOUNT_POINT" || exit 2
echo "Done." >&2

# Recurse into rescued filesystem, building MD5 sum list, saving into temp dir
echo " * Building MD5 checksum list..." >&2
r () { for f in "$1"/*
    do if test -d "$f"
        then r "$f"
        else md5sum "$f"
    fi
    done
}; r "$RESCUED_MOUNT_POINT" > "$TMPDIR/rescued-md5sums"
echo "Done." >&2

# Unmount filesystem so ddrescue can work on the device/image
echo "Unmounting rescued filesystem..." >&2
umount "$RESCUED_DEVICE" || exit 2
echo "Done." >&2

# Fill the bad sector blocks with ASCII '1' (0x31) byte value
#  (this is an arbitrary, non-zero bytevalue and will cause md5sums of files
#  with bad-sector blocks to change)
echo " * Filling bad-sector blocks with non-zero bytevalue..." >&2
# Create a tmp file containing the arbitrary bytevalue for ddrescue to fill with
#  (ddrescue takes this filename as an argument when in "fill mode")
echo -n 1 > "$TMPDIR/bytevalue" || exit 3
ddrescue --fill=- "$TMPDIR/bytevalue" "$RESCUED_DEVICE" "$DDRESCUE_LOGFILE" 2>&1\
    || exit $?
echo "Done." >&2

# Re-mount rescued device/image
echo " * Mounting rescued filesystem once more..." >&2
mount -t "$FILESYSTEM" "$RESCUED_DEVICE" "$RESCUED_MOUNT_POINT" || exit 2
echo "Done."
# Verify checksums using file created before, saving results into tmpdir
echo " * Verifying checksums..." >&2
md5sum -c "$TMPDIR/rescued-md5sums" > "$TMPDIR/results" 2>&1
echo "Done." >&2
# Unmount rescued filesystem for the last time
echo " * Unmounting rescued filesystem..." >&2
umount "$RESCUED_DEVICE"
echo "Done." >&2
# Fill bad sectors with zero (0x00) bytevalue once more to restore image/device
echo " * Restoring bad-sector blocks to zero bytevalue..." >&2
printf \\000 > "$TMPDIR/bytevalue" || exit 3
ddrescue --fill=- "$TMPDIR/bytevalue" "$RESCUED_DEVICE" "$DDRESCUE_LOGFILE" 2>&1\
    || exit $?
echo "Done." >&2

# Print blank line and some text to stderr to make reading results easier
# NOTE: results can be captured from stdout, leaving stderr on display
echo >&2
echo "  RESULTS:" >&2
echo "(writing to stdout)" >&2
echo "----------" >&2
# Print results onto stdout
cat "$TMPDIR/results"
# Clean up temporary files
rm -rf "$TMPDIR"

[lcoughey]

Maximus...I tend to only use ddrescue for devices where DDI really or DE are not viable options. So, I don't use it every day. But, I certainly would be happy to test it with a few cases to see if it is helpful and somewhat accurate.

[maximus]

Do you know ddrescue-checkrescuedfiles? A bash script writted by a ubuntu user for checking checking for damaged files after rescuing

http://ubuntuforums.org/showthread.php?t=1860559

I did not know about that one. It never showed up in my many searches. If I put a section in to check fat and other file systems I would do something like that.

My current method takes a different approach. It uses ntfscluster for finding bad files in NTFS partitions, and debugfs for finding the bad files in ext2/3/4 partitions. This makes it much faster then the md5sum method or the grep method. But it is limited to NTFS and Linux partitions.

I will clean the code up some and post it soon. If anyone has other places that this would be good to post feel free to let me know.

[maximus]

Ok, here is what I have so far. For what it does, it is a working beta. To me it is an alpha version just because I have many more ideas for it (including being menu driven). But time is precious, and I make no promises.

If I do continue to actively work on it, where would the best place be to keep posting the code? I was thinking of putting it in the ubuntu forum.

Code:

#!/bin/bash
#########################################################
# Name: ddrescue-findbadfiles
# Author: Scott Dwyer
# License: This file is licensed under the GPL v2
# Latest change: 2012.03.17
# Version: 0.6a
#########################################################
#########################################################
# The following are sources that this code was created from
#########################################################
# Raphael Hoegger
# Useful code-snippets in datarecovery
# http://pfuender.net/?p=80
#########################################################
# Bruce Allen and Douglas Gilbert
# Bad block HOWTO for smartmontools
# http://smartmontools.sourceforge.net/badblockhowto.html
#########################################################
#########################################################
# This script is meant to use a ddrescue logfile and find
# which files are related to the bad sectors. It is a work
# in Progress. It currently only works on a whole drive
# recovery, and the target must be a whole drive and not
# a partition or an image. It only works on NTFS and EXT2/3/4
# filesystems. It does not work on and will skip over other
# filesystems such as FAT32. It uses ntfscluster for NTFS,
# and debugfs for EXT2/3/4. It also requires a version of
# ddrescue that has ddrescuelog installed. It has been noted
# that certain installation packages do not contain ddresculog.
# The best method is to install ddrescue from source.
# This script will create several temporary files, but
# it deletes them when done. It leaves results in two files,
# a short sorted results file, and a longer version if you
# need more information. This script does not knowingly
# write or alter any data on the target drive and does
# not require the target filesystems to be mounted.
#########################################################
# Usage:
# You must edit the lines below for your needs.
# The TARGET is the rescued copy of the drive. It must be
# a whole drive. The LOGFILE is simply the logfile from
# ddrescue that contains a list of the bad sectors. The OUTPUT
# and OUTPUTLONG are where the results will end up.
#########################################################

# Edit the following lines to suit your needs
TARGET=/dev/sdb ## the target drive
LOGFILE=testbad.log ## the one from ddrescue
OUTPUT=results.txt ## where you want your results stored
OUTPUTLONG=resultslong.txt ## where the long version is stored

## cleanup any files that could be left over from a forced exit
rm -f $OUTPUT
rm -f $OUTPUTLONG
rm -f tempfdisk
rm -f tempfdiskb
rm -f tempdev
rm -f tempstart
rm -f tempend
rm -f tempout
rm -f tempoutb
rm -f tempoutlong
rm -f tempcmdi
rm -f tempcmdn
rm -f tempbad
rm -f tempblocks
rm -f tempcmdi
rm -f tempcmdn
rm -f tempicheck
rm -f tempicheckb
rm -f tempicheckc
rm -f tempinode
rm -f tempncheck
rm -f tempncheckb

ddrescuelog -l- $LOGFILE >>tempbad
ddrescuelog -l/ $LOGFILE >>tempbad
ddrescuelog -l* $LOGFILE >>tempbad
ddrescuelog -l? $LOGFILE >>tempbad

fdisk -lu $TARGET >>$OUTPUTLONG

######################## NTFS ###############################

fdisk -lu | grep $TARGET | grep NTFS >tempfdisk

for PARTITION in $(cat tempfdisk | awk ' { print $1 } ') ; do
   grep $PARTITION tempfdisk >>tempdev
   grep -v "*" tempdev | awk ' { print $2 } ' >>tempstart
   grep "*" tempdev | awk ' { print $3 } ' >>tempstart
   grep -v "*" tempdev | awk ' { print $3 } ' >>tempend
   grep "*" tempdev | awk ' { print $4 } ' >>tempend
   grep -v "*" tempdev | awk ' { print $4 } ' >>tempblocks
   grep "*" tempdev | awk ' { print $5 } ' >>tempblocks
   TEMPSTART=$(cat tempstart)
   TEMPEND=$(cat tempend)
   TEMPBLOCKS=$(( ($TEMPEND-$TEMPSTART) ))
   echo $PARTITION >>tempout

   for failingSector in $(cat tempbad | awk ' { print $1 } ') ; do
      NTFSsector=$(( ($failingSector-$TEMPSTART) ))
      if [ "$NTFSsector" -ge 0 ] && [ "$NTFSsector" -le "$TEMPBLOCKS" ]; then
         echo Checking NTFS partition $PARTITION Sector $NTFSsector
         echo .......... >>tempout
         ntfscluster -f -s $NTFSsector $PARTITION 2>/dev/null >>tempout
      fi
   done

   sort -u tempout >>tempoutb
   echo ................................................................. >>$OUTPUT
   echo ................................................................. >>$OUTPUT
   echo $PARTITION >>$OUTPUT
   grep Inode tempoutb >>$OUTPUT

   echo ................................................................. >>$OUTPUTLONG
   echo ................................................................. >>$OUTPUTLONG
   echo $PARTITION >>$OUTPUTLONG
   cat tempout >>$OUTPUTLONG

   rm -f tempdev
   rm -f tempstart
   rm -f tempend
   rm -f tempblocks
   rm -f tempout
   rm -f tempoutb
   
done
####################### LINUX ###############################

rm -f tempfdisk
fdisk -lu | grep $TARGET | grep Linux >tempfdisk
grep -v swap tempfdisk >tempfdiskb

for PARTITION in $(cat tempfdiskb | awk ' { print $1 } ') ; do
   grep $PARTITION tempfdiskb >>tempdev
   grep -v "*" tempdev | awk ' { print $2 } ' >>tempstart
   grep "*" tempdev | awk ' { print $3 } ' >>tempstart
   grep -v "*" tempdev | awk ' { print $3 } ' >>tempend
   grep "*" tempdev | awk ' { print $4 } ' >>tempend
   grep -v "*" tempdev | awk ' { print $4 } ' >>tempblocks
   grep "*" tempdev | awk ' { print $5 } ' >>tempblocks
   TEMPSTART=$(cat tempstart)
   TEMPEND=$(cat tempend)
   TEMPBLOCKS=$(( ($TEMPEND-$TEMPSTART) ))
   BLOCKSIZE=$(tune2fs -l $PARTITION | grep "Block size" | awk ' { print $3 } ')

   for failingSector in $(cat tempbad | awk ' { print $1 } ') ; do
      LINUXsector=$(( ($failingSector-$TEMPSTART) ))
      if [ "$LINUXsector" -ge 0 ] && [ "$LINUXsector" -le "$TEMPBLOCKS" ]; then
                  
         LINUXblock=$(( ($LINUXsector*512)/4096 ))

         echo Checking Linux partition $PARTITION Sector $LINUXsector

         echo open $PARTITION >> tempcmdi
         echo testb "$LINUXblock" >> tempcmdi
         echo icheck "$LINUXblock" >> tempcmdi

         debugfs -f tempcmdi > tempicheck
         echo .......... >>tempoutlong
         echo Sector $LINUXsector >> tempoutlong
         cat tempicheck >>tempoutlong
         grep -v debugfs tempicheck > tempicheckb
         grep -v -i block tempicheckb > tempicheck
         TEMPINODE=$(cat tempicheck | awk ' { print $2 } ')

         echo open $PARTITION >> tempcmdn
         echo ncheck "$TEMPINODE" >> tempcmdn

         debugfs -f tempcmdn > tempncheck
         cat tempncheck >>tempoutlong
         grep -v debugfs tempncheck > tempncheckb
         grep -v -i inode tempncheckb >> tempout

         rm -f tempcmdi
         rm -f tempcmdn

      fi
   done

   sort -u tempout >>tempoutb
   echo ................................................................. >>$OUTPUT
   echo ................................................................. >>$OUTPUT
   echo $PARTITION >>$OUTPUT
   cat tempoutb >>$OUTPUT

   echo ................................................................. >>$OUTPUTLONG
   echo ................................................................. >>$OUTPUTLONG
   echo $PARTITION >>$OUTPUTLONG
   cat tempoutlong >>$OUTPUTLONG

   rm -f tempdev
   rm -f tempstart
   rm -f tempend
   rm -f tempblocks
   rm -f tempout
   rm -f tempoutb
   rm -f tempoutlong
   rm -f tempicheck
   rm -f tempicheckb
   rm -f tempicheckc
   rm -f tempinode
   rm -f tempncheck
   rm -f tempncheckb

done

rm -f tempfdisk
rm -f tempfdiskb
rm -f tempbad
cat $OUTPUT


[maximus]

It took over a day for my post to be approved here. I have already made changes and updated my script to work with fat32. I think I will be posting it on the Ubuntu forum soon and link to it here. Unless there is a better place to post it...

[maximus]

Ok, the latest code is posted in the Ubuntu forum as a gnuddrescue utility script. It is much improved over what I have posted here. I encourage people to try it out and spread the word. Feedback is welcome.

http://ubuntuforums.org/showthread.php?p=11779251

[maximus]

I just thought I would mention that I added a new feature that might make this more useful to others. It can now also use a text file containing sector numbers. So you could take output from some other source and create a text file with one sector number per line and no other text, and it will process it and find what files are related to those sectors. Latest version is on the Ubuntu forum.

[maximus]

I really don't know where else to ask this. If anybody gets the opportunity to test the latest script (posted at the Ubuntu forum link in previous post), I would greatly appreciate it. It is being considered to be included in a future release of a live CD, and so far as I can tell I am the only one that has ever tested it.

[lcoughey]

I was going to give it a run on Friday, but when I realized that the FS was HFS+, I quickly changed my mind.

[maximus]

I was going to give it a run on Friday, but when I realized that the FS was HFS+, I quickly changed my mind.

Bummer. I haven't looked too much into HFS filesystem support. I don't have any Macs so I would only be able to test it in Linux anyway. But if I happen to come across a way to do it, i will try to add support for it.

[maximus]

I would just like to say that my program (ddrutility) has made it onto the new release of Ubuntu Rescue Remix. I think I almost spent more time and effort trying to get it packaged in time than I did writing it.

I would also like to say that now that I can work on it again, I believe I have found a way to support HFS/HFS+. It will take some time for me to make it work and test it, but it does appear possible.

[maximus]

New version released. Now has HFS+ support

[glorifyday]

Hi Maximus,

I've been looking long for the tool like yours. I am really astonished how low the interest was
I am rescuing data from 1TB drive and will use your tool packaged in the latest Ubuntu Rescue Remix.

How sure are the results of your utility? Does the utility tend to under or overestimate? What happens if the restored filesystem lacks consistency?