Recovered audio from formatted SD card - ends prematurely

[admadm]

Inadvertently formatted a SD card and used the newly formatted card to record once.
Realized I needed an audio file recorded before the formatting.

I'm able to recover the wav file. The file seems to be the proper length (6:57:00, e.g.). But on playing the file there are periods of static every 10-15 seconds that seem to be inserted between rather than over the audio. (In other words, I hear a sentence that should sound "Now is the time for all good men to come to the aid of the party" as "Now is the time for all good men [STATIC NOISE FOR A SECOND OR SO] to come to the aid of the party"

The effect seems to be that all of the words are there but the timecoding is thrown off. So the recording stops playback at 6:57:00 but only 5:59:00 of the actual audio has played.

My thought is that the remaining 58 minutes of recorded audio is there in the recovered file but won't play because the metadata is telling the media player that the file is ended.

I would welcome alternate explanations, suggestions, corrections; but most importantly, I would hope someone could tell me how to get that last 58 minutes of audio to play.

Thank you.

adm

[HaQue]

maybe see if you can open the file in Audacity and edit the static out. Or open in a hex editor and go to the end of file markers and change the values to HEX 0xFF or something like that

[admadm]

maybe see if you can open the file in Audacity and edit the static out.

Tried that. But that just makes the file shorter/smaller

Or open in a hex editor and go to the end of file markers and change the values to HEX 0xFF or something like that

That sounds like the sort of jiggery-pokery I'm looking for. But I would need some guidance on how to do that.

Thanks!
adm

[fzabkar]

IMO one plausible explanation for what has happened could be that the original file was heavily fragmented, but the recovery software reassembled the recovered data on the assumption that the original file was contiguous. If this is the case, then it would be highly likely that the remainder of the data is contained within those sectors immediately after the end of the file.

In a FAT32 file system each file has a directory entry that identifies it by its long and short name. The directory entry also holds the date/time stamps, attributes, starting cluster number, and file size. The OS locates all the remaining clusters within the file by consulting the File Allocation Table (FAT). A contiguous file will have a single unbroken chain of clusters, but a fragmented file may consist of two or more cluster chains. When you format a drive, the FATs are cleared, resulting in the loss of the cluster information.

If the data recovery software can locate the original directory entry, it will know where the file begins and how large it is, but it will have no idea whether it is fragmented or contiguous, so it will proceed on the assumption that the file is contiguous (unless it uses some intelligent algorithm, eg entropy analysis).

One approach that I would try would be to locate the file's directory entry (eg by searching the cloned image for its file name), and then edit the file size within the directory. Choose a size that would be equivalent to 10 minutes of audio, say. Then run your data recovery software against the modified clone. Hopefully you will now have a 10 minute audio file that contains all your audio data, albeit interspersed with noise fragments, plus some superfluous stuff at the end.

[fzabkar]

Another way might be to examine the first sector of the file with a hex editor, see if there is anything unique about it (eg text information), or simply cut the first few dozen bytes and search the cloned image for this same pattern. Then carve this sector and the next X sectors from the image (where X is equivalent to 10 minutes of audio), and copy these sectors to a file. Hopefully the carved file will contain all your audio data.

[HaQue]

Franc, when a Flash device writes to a card, there are a few fundamental differences to conventional Hard Disks that are going to make this difficult. due to Wear Levelling algorithm you have no way of knowing where on the flash chip the actual DATA is going to be written. Well that is not exactly true. You can know EXACTLY where the DATA is written. If you know the controller, the wear levelling and whatever hokey pokey the manufacturer does such as change block numbers, shift sectors around in the block, invert etc, you could work from a known state forwards to current state. It would take some work.

So your suggestion is based around the image, not what is on the NAND.



To get back an image from what is physically stored, you might need to:
1. get NAND contents (might be 1 - 8 separate dumps of data from a number of chips)
2. cut the contents into say 50 chunks
3. Pair up all the chunks
4. look at some attribute, and on some blocks, rotate some sectors around
5. take a number unrelated to actual sector size of 512, say 2112 or 9216 and join the data by a page from each
6. strip out all the Drive service data and create an image by now mutating the previous block and page sizes and creating 512 sector sizes
7. go through the image looking for parts that are in conflict (2 blocks having the same block number - one is valid one is not) and correct it
8. other operations are possible, along with many variations.

basically you end up writing to a NAND(physically) like this: red squares are the last write:

1.jpg (8.86 KiB) Viewed 11572 times

If you get data before anything is written, then maybe there will be no corruption.

Once you write again, the controller can do a number of things:
1. read some blocks and decide if it should merge your data with another block
2. decide a block is end of life and copy to another one
3. write new data to anywhere based on what I described earlier

now the NAND may look like:

2.jpg (16.23 KiB) Viewed 11572 times

Once a block is written, thats it. The data that was there is GONE

on a Conventional disk, LBA 0x18923234 is usually not moving around. On Flash, the location 0x18923234 in the image might change physical locations many times. might be block 0x092 then it might be block 0x330 after a write. generally doing recoveries, if you are missing something or have something in the wrong place, the size of the error is a block. a significant amount of data

Video and audio don't handle missing parts well.






1. How did you recover the file? what software? maybe a better program may get a better result? I recommend take an image of the device with DMDE and running getDataBack or R-Studio on the image.
2. have you tried:
http://www.dslrfilmnoob.com/2013/08/06/ ... files-vlc/
http://www.delback.co.uk/wavrep/

or similar?

[fzabkar]

Haque, when I refer to an image, I'm referring to a logical clone, not a physical NAND dump. Are we at cross purposes?

[HaQue]

I'm not sure, maybe

But I was getting at once the drive was written on again, the whole image(once you take a logical clone) is going to be "peppered" with new data or moved blocks/pages/sectors, and likely be corrupting Wanted DATA.

I just re-read your post and I think where you said "be highly likely that the remainder of the data is contained within those sectors immediately after the end of the file." .. You are talking about the reported end of the file, or the location that software now thinks is the end of the file. This cold be the case, but I think a 100% clean file back is very optimistic. ..

[fzabkar]

Let's say we start with a bunch of small files.

file #1
file #2
file #3
file #4
file #5
file #6
file #7
file #8

Now let's delete several files. This leaves us with holes.

hole
file #2
file #3
hole
file #5
hole
hole
file #8

Let's assume that the drive is very full, and that the next audio file is a large one that is laid down as several cluster chains, each filling the next available free space. As you can see, the file is severely fragmented.

audio data cluster chain #1
file #2
file #3
audio data cluster chain #2
file #5
audio data cluster chain #3
file #8
audio data cluster chain #4

The file's size and the location of its starting cluster is stored within the file's directory entry. The location of each cluster chain is stored within each copy of the FAT. Formatting a drive clears the FATs, but it leaves the data area intact (apart from the root directory, which is cleared).

When the data recovery software scans the data area, it locates the directory entry. This enables it to identify the file's starting cluster and the file's size. However, the software has no way of knowing that the file consists of several fragments, so it assumes that the file is contiguous.

This means that the recovered file might be assembled from the following consecutive fragments:

audio data cluster chain #1
file #2
file #3
audio data cluster chain #2
file #5

As far as the software knows, these fragments might just as well be a single cluster chain constituting a single file with the correct size. If you now fool the software by increasing the file's size, then it will incorporate the succeeding fragments into a larger recovered file.

The recovered file will now look like this:

audio data cluster chain #1
file #2
file #3
audio data cluster chain #2
file #5
audio data cluster chain #3
file #8
audio data cluster chain #4
extra stuff

There will be additional "noise" at the end of the file.

I have attached two screenshots from DMDE. They show a file, IsoBuster.exe, and a subdirectory, FAQ, within a FAT32 directory named IsoBuster.

Here is Microsoft's FAT file system specification:
http://staff.washington.edu/dittrich/misc/fatgen103.pdf

Attachments

[admadm]

You guys are way over my head! (But thank you for engaging.) I do get the forest if not the trees and leaves and ferns and moss of your discussion.

I've tried some things with the recovered file with some interesting results.

Converting to Audio-CD in VLC "corrected" the file length from 6:56 down to about 5:56.

Opening it in Adobe Premiere Elements 9 shows the length to be 6:56 -- but only shows waveform for 5:56.

So I've come to believe that the missing data is not in the recovered file but has been separated from it but may still be on the card (somewhere).

The question now is, is there any hope of recovering the severed last hour of audio?

adm

[admadm]

IMO one plausible explanation for what has happened could be that the original file was heavily fragmented, but the recovery software reassembled the recovered data on the assumption that the original file was contiguous. If this is the case, then it would be highly likely that the remainder of the data is contained within those sectors immediately after the end of the file.

I would think it would not be highly fragmented because of the way it's used. This is an SD card that's used as a backup. It would be formatted, then used to record about 6 or 7 hours of audio, steno shorthand notes, and a couple other related files each day. But it is never accessed or edited or anything that would fragment the files.

When the card gets near full, I get a message to that effect and re-format and use again.
So this card had 83 files on it. Then I re-formatted and used once for a couple hours.

Unfortunately, I didn't find out the the main audio recording (for which the SD card file is a backup) had been turned off before the end of the proceeding.

Don't know if that's helpful info, but...

thanks
adm

[fzabkar]

Can you use a hex editor to show us the first sector of your recovered file?

HxD - Freeware Hex Editor and Disk Editor:
http://mh-nexus.de/en/hxd

We could then search your card (or its cloned image) for this sector. Once we find this sector, we can then use the editor to extract the following 10 minutes' worth of data. It's a lot easier than it sounds.

If this doesn't find your missing data, then you could try alternative tools, as per Haque's suggestion.

BTW, what is the size of the recovered file in bytes? This will give us an idea as to the number of bytes per minute.

[admadm]

Hey, thanks again for taking the time to help with this.

Good news! There was a backup to the backup, and I was able to get the intact file from there.

adm

[hhddrec]

I think this case maybe recoverable with dump extractors tools like softcerntere tools or ACE tool.

what you think?

[HaQue]

I think this case maybe recoverable with dump extractors tools like softcerntere tools or ACE tool.

what you think?

It is solved, but no, I don't think in this case the tools will produce anything different.

[bithound]

I could imagine a special scan technique for this particular case, assuming that we are looking for a WAV file that was written in sequence to fragmented free space on a FAT-formatted device.

AFAIK a WAV file has a small header, followed by uncompressed audio samples. For example: in 16-bit stereo format each sample is 4 bytes long, 2 bytes per channel. Each value is a signed integer, probably encoded as LSB-MSB. In a HEX editor, you can easily recognize this pattern. Even bytes look almost random and odd bytes are less random. Odd bytes also alternate regularly between positive and negative values.

It shouldn't be too hard to write a small scanner program that looks at each sector of the SD card image and decides whether it contains WAV data or not. Next step is to concatenate only the sectors that look like WAV data and postprocess the resulting file as a raw 16-bit stereo sample file with a tool like Audacity.

NOTE: This method will yield bad results if the card also contained other WAV files, or if the requested file was not written in one continuous write action.

Regards,

Erik

[fzabkar]

It shouldn't be too hard to write a small scanner program that looks at each sector of the SD card image and decides whether it contains WAV data or not. Next step is to concatenate only the sectors that look like WAV data and postprocess the resulting file as a raw 16-bit stereo sample file with a tool like Audacity.

I like it. :idea: