Creating a Linux Tool, Links to Technical Details Please

[Falkoner]

So I'm planning on creating a suite of Linux-based, or ideally, OS-independent tools for analysis of Windows-based systems from an offline environment.

So, what Linux currently seems to have is BadBlocks, however, it's fairly limited for my purposes. What I'd like to know is if anyone can link me to the details of differences between performing low-level hdd diagnostics on various OSes.

For example, I've heard that it's fairly limited under Windows, however, if anyone can link me to the lowest-level code implementation available, I'd add it as a Windows version for compilation.

So, the areas where I'd love to be linked to tutorials, books, and technical details would be:

Low-Level HDD Scanning on Various OSs
Which OSes will give me truly low-level scan results(comparable to those of MHDD or HDAT2), without hiding errors? What functions/libraries in C/C++ would be useful to create these scans?

SMART Data
Similar to the above question, how do I programmatically obtain this information, and does anyone have any resources that would help me understand the output perfectly, in order to simplify it for end-users?

Bad Sector Repair
With programs such as HDD Regenerator or SpinRite claiming to be able to repair discs, do these claims hold any water? Can they be duplicated? Also, I assume most bad sector repairs are really just the HDD using its collection of spares, similar to how HDAT2 does. How can I force these repairs?

Filesystem Specifics
Is there a good resource to try to understand the various filesystems deeply? Perhaps most important to me, is finding out if there's a way to detect which file is located on a specific sector in an NTFS filesystem.

Partition Table Repair
Is there any go-to resource that explains how to programatically detect and attempt to repair messed up partition tables, whether they use MBR or GPT?

Thanks in advance for any tips you guys can link me to!

[HaQue]

How about downloading Linux source code of hard disk tools and drivers from whatever your favorite distro is
Do a search on sourceforge and got for tools to work with hdd smart and file systems and look at the source
As for ntfs you should be able to find the NTFS spec itself

[Falkoner]

Yeah, that's definitely the plan, I suppose I was just hoping someone would know of some pre-chewed information about implemention that could help accelerate my understanding of the process.

[labtech]

For example, I've heard that it's fairly limited under Windows.

The majority of data recovery tools, software and hardware, are/have been developed for Windows based platforms as it has been the most popular.
However, other, especially Linux is making some strides.

Bad Sector Repair
With programs such as HDD Regenerator or SpinRite claiming to be able to repair discs, do these claims hold any water?

No. What these tools do is highly misinterpreted. These tools are not meant for data recovery. I guess they can be considered refurbishing tools, at least that is my interpretation of their goal, but they are very limited. In my opinion, due to such limitations, they are more detrimental than useful when used in "repairing" applications, especially when the ultimate goal is data recovery. Why? Because data recovery is about preservation or preventing (eliminating is really the goal) any further alteration of the failed drive with whatever software and hardware tools engaged. These tools do not serve that purpose.

One other critical thing about why these tools are highly misinterpreted is also directly due to the fact that the user misinterprets the damage and the extent of damage a drive has sustained. In lame terms, when a person has a specific health issue, the person should not use the wrong medicine for that.

[fzabkar]

The smartmontools project is open source and written for Linux.

http://smartmontools.sourceforge.net/

[louis]

@fzabkar. you'll have to study quite a while to start understanding what's behind the scene. windows it's just a layer. there are drives between hardware...and OS; your application will talk to the OS which will call the driver which will access the device. basically with a windows application you cannot access the device the way you need (raw ata commands) unless you write your own driver. to read smart data you must send ata commands directly to the hardware...so you must have your driver which communicate with the device. windows has some API which lets you communicate to the device like DeviceIOCtrl...without having a driver. to write a driver it's a pain in the ass, and the later win OS the driver must be signed so you must register a trusted certificate from a trusted authority on the web...and then windows accept your signed driver. a third way it's to use a driver which let's you access the I/O ports from ring3. there are few of them. one it's inside wdr folder...and it's working only on xp ; then if you can access the I/O ports..you must start finding where's your device attached..by learning the PCI bus structure...and how ide device stay plugged there; then finally you have the i/o ports and you can talk with the drive.

I don't know what you mean with low level scanning. I assume you override the OS? then you must do all I said above. else you can use some windows API..a little lower than ReadFile; perhaps one which let's you read the drive LBA by LBA. but nothing compares to VSC stuff...which is the direct communication with the device firmware.

for multi-platform support you should use c++ std lib.

those appz claiming to "repair" drives I suppose that uses the ATA commands which runs smart test on firmware, which knows what to do. there's nothing at user level to "repair" bad sectors on hard drive

for file system and partitions I can't help u, but the format it's public for most of the known systems.

[fzabkar]

@fzabkar. you'll have to study quite a while to start understanding what's behind the scene.

I fully understand what a driver is and does. I just don't know how to write one. My "smartmontools" suggestion was in relation to the OP's "SMART Data" question.

BTW, I have written assembly code in Data General machine language to address a particular disc controller at the register level, so I'm no stranger to programming. In fact that was my second data recovery job. :-)

[louis]

don't know how smartmontools works..but if it doesn't have a driver for windows...then it uses the DeviceIOCtrl to pass-through the ata identify command to the dev.

basically what you want to do, an image analyzer for a windows disk?

[fzabkar]

The OP asked the following question:

So I'm planning on creating a suite of Linux-based, or ideally, OS-independent tools for analysis of Windows-based systems from an offline environment.

...

SMART Data
Similar to the above question, how do I programmatically obtain this information, and does anyone have any resources that would help me understand the output perfectly, in order to simplify it for end-users?

To obtain the SMART data from a drive in a Linux-based environment, the OP would only need to use Linux tools. Whether the drive contains a "Windows-based system", or any other system, would be completely irrelevant.

[louis]

ok. then you learn how to access the hard drive in linux. if you want to make a windows version of the program, then take in consideration what I wrote. the hard drive access part of the application could be OS-dependent(linux, windows, etc. depending on what OS you run the app). the part of the program which sends ata/vsc commands could be encapsulated as a library, to be used on all OS. also the FILE SYSTEM analyzer part it's also OS independent.

[Falkoner]

Sorry for the delayed response, life got busy.

Thanks so far for the suggestions, louis answered my question about the low-level scanning reasonably well, since that is what I had heard, that Windows requires you to go through an API for access, and I've heard that occasionally that API will hide errors that it actually found. Does anyone know how HDDScan goes about it in Windows? That would probably be the route I'd like to go, with a BadBlocks-style implementation for Linux. I'd likely just write a function for HDD scanning, and then have the software call it, and it'll automatically choose which actual function to use based on the OS(or for which it's compiled).

With regards to the comments on HDD Regenerator and SpinRite, I'm particularly interested in how HDAT2 does its claimed repairs, since I've actually refurbed quite a few drives using it. I'd assume there's a read/write pattern that allows you to force the drive to attempt to replace the bad sector with a backup good sector. This would of course be different than the HDD Regenerator or SpinRite methods that claim to use magnetic magic to restore the sectors physically.

Part of why I need to understand this is for the software to alert users when potential damage could be done, or if they're scanning under an OS that will give less than true results.

I'm still open to any other links or information available!

[fzabkar]

Simply writing to a bad sector causes the drive to retest it and either return it to service if good, or replace it with a spare if bad.

As for "a way to detect which file is located on a specific sector in an NTFS filesystem", Microsoft provides a free utility called NFI.EXE. I believe the ntfsprogs project (freeware) also has a utility (ntfscluster) that can do this. If you're looking for a commercial tool, then I believe WinHex can do it as well.

The following resource has very good information regarding the structure of boot sectors, MBRs, and partition tables:

http://thestarman.pcministry.com/asm/mbr/index.html

As for HDD Regenerator or SpinRite, you may find the following thread enlightening:
viewtopic.php?f=1&t=28780

[Sasha Sheremetov]

Fzabkar, thanks for the mbr/boot link, haven't seen so detailed description before!

[athena]

Hi
thanks fzabkar , from all your posts you have genuinely tried to share knowledge & help the other person.
I appreciate this. I was also looking for similar information & got it.
keep it up. I am sending you a PM related to this query .
regards.

[Falkoner]

Dang, that website is sexy! Definitely the sort of thing I was looking for.

I don't know if you guys know about the program "ntbootautofix" it appears to be of Chinese origin, but I've had success using it to get Windows systems booting again which had issues with the partition table, and failed to boot after using PartitionGenius to fix it. I'd love to know exactly what that program searches for and fixes.

[HaQue]

RE it then in experience Chinese software is full of debug symbols and although awfully coded, quite easy to RE as it does only what it is set out to do

[poogi]

You may want to take a look at my tool: diskscan

It's intended to scan the disk, report on latency behavior across it and it will try to fix areas of the disk that have high latency by rewriting them. It worked for me on one disk.

You can find it on github: https://github.com/baruch/diskscan/

[Falkoner]

Interesting, I was actually planning on creating a scanner that combined the GUIs of HDTune with HDDScan/MHDD with the repairing abilities of HDAT2, and make it Windows/Linux compatible. A fairly large project, I'm sure, but I'm really enjoying the resources being passed so far, the DiskScan tool definitely gave me quick insight into how to do the scanning, just one question, did it rewrite reads that were simply slow, or reads that requires multiple tries?

[poogi]

Diskscan rewrites locations that are slow.

If you are interested in porting this to windows and/or create a gui for diskscan I'd be happy to take contributions, it's something that's been on my plan but I just never got around to it. There is a stub that worked for a while to make it work on FreeBSD but it was a pain to maintain the FreeBSD VM when I don't normally use it, but it served its purpose as a proof-of-concept for the porting work.