This is related to the "Firmware modding to gain raw access to data from read head?" thread I started a couple days ago, but it's more generalized to the point that I think a separate thread is called for.

I had an IBM Deskstar go bad in January 2002, and it spawned an ongoing data recovery project. At first all I did was clone the partition (with WinHex which logged bad sectors), fix the cloned copy so it was accessable, and write a program to generate a list of affected files from the list of bad sector LBAs. Later I mapped the geometry of the bad sectors and worked on recovering the data from the bad sectors themselves. The project was a big success; I fully recovered many of the files most important to me, and learned a heck of a lot along the way.

Talking about this in the StorageReview forum was a tad disappointing towards the end of the thread, when I was describing the technical details of what I had figured out and done, and nobody had much to say about it. Now I've discovered the HDD Guru forums. Perhaps I have found a place where I can actually discuss these things with others?

Many of you on the forum are extremely experienced data recovery engineers, and I would like to hear what you think may have actually happened to my drive. What kind of event would create the bad sector stripe in this graph? (Also see the polar graph.) Is it possible a physical scratch was created in that path, or did the head simply write while seeking (and why would it do that)? I have a vague hypothesis, but I'm a newbie at this compared to you.

And the steps I used to fully recover some bad sectors... how many of you have done these things, or even do them regularly?

1. XOR-descrambling and inverse-RLL'ing multiple reads of a sector, and lining up the multiple reads so that intact parts can be identified, and some of the marginal parts can be recovered
2. filling in data in a sector by context; how about in compressed files, where (with the aid of a custom-written program) backreferences can be filled in one by one, each one giving more and more context to do more and more filling-in?
3. using Galois field matrix division (automated by a program, of course) to correct any chosen set of N bytes using the ECC (where the ECC is N bytes long, and 512-N or more data bytes are known to be correct — brought past the all-or-nothing threshold with help from the previous two steps)

For me, this was painstaking work, and doing #2 absolutely required an intimate knowledge of the nature of the data that had been corrupted. (Most of the bad sectors I recovered were pieces of game replay logs in a format I know only because I'm a developer of the game in question.) It makes me wonder, if I had sent this drive to a high-end data recovery center, would they have simply called these bad sectors a loss? Or is there some magic means of data recovery that could have worked on these same sectors without requiring intimate knowledge of the data — perhaps a hardware technique that trumps my software-only technique? Steps #1 and #3 could be automated to recover a subset of sectors (and recover the rest with zeroed gaps inside)... does anyone do that?

About #1 and #3 — I would imagine the XOR scrambling, RLL (*PRML) algorithm and ECC polynomial&matrix would have to be reverse-engineered, like I did. Or is this information shared at all?
About #2 — what if there are "bad sector gaps" in a data file that is, let's say for the sake of argument, a story the client wrote. What if the client would know how to fill in some of the gaps, because he/she remembers writing it — and this extra amount of filling-in could bring it to the critical threshold of being ECC-correctable — do you bring in the client, or do you call the missing data a loss?

Perhaps I should get into data recovery as a career. How many of you got started recovering your own data?

GAH! The title of this thread was supposed to be "How similar was my recovery project to what you do as a career?"
The truncation makes it look like something very sarcastic and rude, like "what do you care"
Moderators, can you fix this please?
If it's not possible to exceed that length, please use this one:
How similar is my recovery project to professional recovery?

Title has been fixed. Sorry for the confusion

_________________
Dmitry

Hi Qubit,

Hell of a job you did!

I'm quite sure not 1% of the data-recovery engineers go as far as you did.
It still has to be affordable for the client, and if we had to do a reverse engineering job as you did, costs would be enourmous.
This is only feasible if we could use it for a lot of recovery-jobs.
Like they say in french 'Chapeau'

I'm curious how you did this; what programming language did you use, where did you get the information on the galois fields etc... This is already advanced stuff i had in the uni.

If you would like to share your knowledge, count me in

As far as recovering data from bad sectors concerns, there is a program called 'HDD regenerator' which does very similar things. It reads the bad sector many times, uses advanced math to calculate the most probable value for the sector, and then remappes that sector to a good one with the calculated value.

Best regards,

Dobre

_________________
Murphy was an optimist

Datarecovery in Belgium, Holland, France and Germany
Datarecoverytools http://www.drtools.eu

Hello qubit

Like dobrevjester, on first welcome to the forum , im think and many of the people wich we post here,we doesn´t have u advanced knowledgment, so u are not a newbie for us , congratulations for u research and for share with us, about u question and u photo, yes on IBM there are some tendence to head crash, over the platters, so could be physical scratch.its some common,on IBM/Hitachi. id like the reverse engineering too on my experience on data recovery i begin´s to researchabout hard disks,and specially low level data recovery, because on my country many people have the idea "if u cannot access to hdd its died, and unrecoverable ", but i always, think on my mind , maybe could be another way, to access HDD even was not detect by BIOS ; and check whats happenning! like syntomph, then discover the damaged, fix it and get data .

I am interesting on u research´s too if u want u can share with me

Best Regards

Sinceraly from Mexico

Alberto

Hi Qubit
Welcome To Forum ,
You have done a great Job. I am sure you have done a great research kind of work instead a data recovery. We will be happy to share knowledge with you.
Thanks and "Wish You All the Best"

Hi,

I have also dealt with such things a couple of years ago, but i found these things are far too complicated for the most recovery jobs. And also there are some brands of HDDs supplying too little information (not all the ECC information, or no data at all from detected bad sectors).
I also started to create an algorythm to find out the most probable values of a sector, but this takes huge amounts of time for an entire drive, so such things must only be applied on really important areas of files.
This is my oppinion.

pepe

_________________
Adatmentés - Data recovery

maysoft, thanks for the quick title fix.

dobrevjetser, beto, hddbug, thank you!

Dobrevjetser, I programmed exclusively in C for this project. To gain access to the ATA READ LONG and WRITE LONG commands, I modified the IDE taskfile driver in a Linux kernel. For most of the other things, I programmed under Windows.

The information I could find on the web about Reed-Solomon correction and Galois fields was too abstractly mathematical, with utterly no concrete examples, and I didn't understand it. I ended up teaching myself how it worked by experimenting on the hard drive — writing a sector of data and reading it back with ECC to see what effect changes in the data had on the ECC.

I would indeed like to share what I've learned. I just read the description of the Hard Disk Drive Technology forum, and it seems like the right place to discuss this research — perhaps I should start a thread there?

Fine for me Qubit. If Maysoft agrees


Best regards,


Dobre

_________________
Murphy was an optimist

Datarecovery in Belgium, Holland, France and Germany
Datarecoverytools http://www.drtools.eu

qubit u are welcome to share with us what are u learned and yes its good idea put some over the Hard Disk Drive Technology forum and we are going to send replys

best regards

Sinceraly

Alberto

qubit,
You did a great job.
You will be disappointed but the truth is that data recovery technicians normally do not go so deeply into analyzing the problem and trying to solve it. They try a few tricks/tools they know and if it doesn't work - call it quits. Even if they would love doing this don't forget they have managers above them and all they care is man-hours per drive == $$$.
Your field is probably R&D for data recovery. I know Actionfront had their R&D division. Not sure what happened to them after Seagate took over.

Hello friends

starling u are right, unfortunely some´s techinican on DR doesnt like to analyze problems, and they try to solve quicly solutions and if they dont get solution they said its "unrecoverable" , i get on my country some cases from companies like ondata, drive savers, and another´s where´s they said, there are not, solution, on this cases i late, some time but finally get data, one case toshiba and another wd, and another samsung case, where the people who post i can do the data recovery and doesnt know nothing about hdd enigneering opent the hdd´s and they did a head swap, and when put the hdd on start up, they create a beatiful head crash over the platters, , then the client send to us, and well on this case we can get like 70% of data , but the point its the data recovery field its not a easy solution job, i had a case wich i post here and with u help´s WD400BB some people could rember this case , on this case we late like two month´s for get data, 100 of data. can u believe that.

So im agree with u point , many people shoud be understand need to study , im not a hddguru like yours but i always learn from u posts

Best Regards

Sinceraly

Alberto

Hi qubit,
How easy is it to modify the IDE taskfile drivers to allow READ LONG commands? When I try to run sg_read_long (from sgutils tools), it returns "command not supported". Could you show me how to fix that?

THanks!

This thread goes back to 2007...

Yeah, I know. But there is scarce information on what I'm trying to find out. So I'm desperately hoping that I can find out how qubit accomplished the LONG READ, even if it happened years ago.

Wow, are you kidding me? The Linux kernel still doesn't support these commands? Back when I did that project I was sure it would only take a couple years at most before it did. And I didn't submit my own modification because it was specific to my needs.

I think I still have the source I originally modified, somewhere, so I suppose I could just find it and diff it with the unmodified source of that kernel version for you.

(Oh, it just occurred to me before submitting this reply — perhaps the Linux kernel does support these commands now, but your drive doesn't... on the other hand, taking the more optimistic approach, maybe it supports only the SCSI READ LONG and not the IDE/ATA one.)

Also I'd like to mention, Windows supports doing arbitrary ATA commands, so this can be done in userspace — and I used exactly this to do READ LONG to recover all the data from a friend's hard drive (though that only worked on the first 128 GiB of course; for the rest I had to use normal reads, though I did them through the same interface). Due to that project I now have the actual ECC algorithms used by two drives, my Deskstar 75GXP and his Maxtor. (Though I didn't manage to figure out the PRML algorithm used by his drive, and didn't need to.)

Okay, here's the patch. It's against Linux 2.4.19-pre9-ac2, so some modification may be needed to adapt it newer kernels.

Also, this is exactly how I left it in May 2005. Find the part that says and change the 40 to the correct number of ECC bytes for your drive.

If I can find the usermode part of this (how to use the driver from a C program) I'll post it too.

Should be pretty doable, as sources for virtually everything are available. So with some level of experience, I believe it should take an evening or two even if done from the scratch.
Considering that qubit posted his patch and you can also get some support at Linux forums, nothing should hold you back.

But keep in mind that without a clear purpose, this development may turn into something "just for fun", as globally READ LONG works on some old drives only.
E.g., a Samsung drive (independently of generation) on LONG will return junk, undeciphered data from the read channel "as is", which is useless.


Also there can be a kernel-level lock for some commands in Linux. I don't work with Linux much and not 100% sure about this, so better recheck somewhere else.

And speaking generally, as an observation, GCC++ compiler is likely to appear whimsical if you used to code in Visual Studio.

_________________
• Remote RAID, NAS, SAN, VMware, DVR (CCTV), flash and tape recovery. Data recovery support.