All times are UTC - 5 hours [ DST ]




Post new topic Reply to topic  [ 104 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next
Author Message
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 21st, 2019, 21:31 
Offline
User avatar

Joined: February 24th, 2019, 11:07
Posts: 44
Location: Canada
LarrySabo wrote:
@fzabkar, it was connected directly to the copmuter's USB port. I must say, I'm impressed with how fast HDDClone was. A power reset wasn't required for these 300000 sectors, fortunately. I'm curious whether a simple inline switch pressed manually could serve as an alternative to using a power relay with HDDSuper Clone Pro for drives where there are limited times power resets are required.

Yes Larry, I've tested it connecting external power supply with switch. It worked very well. I had a drive that had to power cycle every few minutes, took me two days but data recovered.

_________________
https://cslpc.ca/


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 9:23 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
To understand how these tools can have such a large improvement in speed when dealing with drives with bad sectors, you need to first understand what the OS is doing that is slowing it down. And the simple answer is retries. The OS refuses to accept that there can be bad sectors, and performs a stupid amount of retry attempts. This is happening at the driver level, and also possibly at an additional intermediate level. Before I made HDDSuperClone, I experimented with ddrescue and even had a passthrough patch for it. In my testing with Linux I could easily calculate that the OS was performing 5 retries. In older Linux kernels, it was 15 retries, but there was a way to get it down to 5 (setting a flag to perform direct IO without buffering). I think 5 was at the driver level, and there was an additional multiplier of 3 retries at an intermediate OS level.

Looking at Larry’s numbers, it would appear the retries are between 4 and 8 (DFL-DE = 8, R-Studio = 4, and DMDE = 5.7). If I had to guess, the standard retries here is 4 with an intermediate multiplier of 2 (R-Studio may be using a direct flag as it does in Linux). I can’t explain the 5.7x of DMDE. Some of this could be retries done within the programs themselves.

So if the retries are not too much different than Linux, why is Windows so bad with some failing drives? The only answer that I have for that is that when the bad sectors contain important metadata (such as the MFT), Windows has a panic attack, and then a heart attack and stroke, and that intermediate multiplier goes up like high blood pressure, and the end effect is a locked up computer until you remove the offending device. I think some of it may be that the OS is trying to be fast and using multi-threading to perform multiple read operations at once, and when they all hang, you are all done. Linux on the other hand, does not seem to get this panic attack, and just gives up after the normal retries, allowing quicker access to the disk. It also helps to have auto mounting turned off so the OS doesn’t try to read as much, but even if it is turned on, Linux will usually not lock up like Windows will.

So why is HDDSuperClone better in Linux? It uses direct passthrough commands. For USB storage devices this is SCSI passthrough. If you examined raw USB communication data, you would see SCSI commands. This basically bypasses the normal driver retries completely, as it is directly (as possible) sending and receiving the actual commands to the device, so it is just a simple send the command and get the results from the command, and it is up to HDDSuperClone to determine the status of the command. So bad sectors are only tried once in this case, further attempts would be in done in future phases of the recovery if they were enabled.

So my theory on how the USB Stabilizer works is that when it encounters a bad sector, it logs it, and then when there is another request for that sector (a “retry”), instead of trying to read it again, it immediately returns with a failed read. And when you return that fast, even though it is still a read error, it is very much helping to prevent a system lockup. I see similar results with my Virtual Disk Driver in HDDSuperClone, which because it is using the system in a backwards way, is just a computer lockup waiting to happen. But when the failed reads are instant, it is much more stable. It can’t be compared directly to the USB Stabilizer, but there are similarities in the concept of operation that allow me to make that connection.

So after seeing the price (WOW!), and the results compared to the free version of HDDSuperClone, which runs on a free OS, why would someone buy it? Simple, because it works with Windows and all of the recovery software that is made for Windows. As Larry indicated in his review, he was able to perform firmware commands through the stabilizer with his Windows based software, something that would not be possible with HDDSuperClone (you know, because it runs on that “other” OS that almost no one writes tools for). And it would also allow for a file level recovery as opposed to having to clone the whole drive first. Get past the price tag, and I could definitely see it being very useful.

And all of this is even without talking about the ability to power cycle. And I am curious about these lines in the features:

Quote:
Customizable read timeout duration in milliseconds.
Read timeout enforced by software reset, hardware reset, controller reset, or drive repower.

Other than a power cycle, I am curious what other kind of effective resets can be done with USB. I know there is a command to reset the USB port, but does it also really cause a reset to be sent to the drive itself, particularly in the case of it basically being a SATA device with a USB board (not a flash drive)? Did the read timeout actually have an effect in Larry’s Test? Considering the results of the HDDSuperClone comparison test, I would say no, the read timeout did not have an effect (smoke and mirrors?). From the log Larry sent me, I can tell that the failed reads took between 3-4 seconds. If the 1000ms (1s) timeout on the Stabilizer was working, it should have won hands down, but it didn’t. And the time almost perfectly matched the fastest time of the Stabilizer, so I would basically consider that a tie, meaning they both had the same effect. So far I have only dabbled in USB communication. I can see possible future experiments bouncing around in my head, probably not any time soon though…

P.S.
The Virtual Disk Driver in the Pro version of HDDSuperClone can also allow for file level recovery, but even I have to admit it is a bit touchy, maybe a little clunky and complicated, and requires more steps to get the end result. And of course it uses the dreaded Linux, and you are limited to only a few recovery programs that will work with it. But master it, and get one of the very affordable relay boards (get more than one, really, they are cheap) to use with it (yes you do have to do a little wiring), and it is one hell of a price difference for what you can do :)

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 11:45 
Offline

Joined: December 8th, 2010, 11:37
Posts: 738
Location: Ottawa, Canada
Thanks for the follow-up and analysis, @maximus!

_________________
Sabo Computer Repairs & Data Recovery


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 16:52 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15462
Location: Australia
maximus wrote:
So after seeing the price (WOW!), and the results compared to the free version of HDDSuperClone, which runs on a free OS, why would someone buy it? Simple, because it works with Windows and all of the recovery software that is made for Windows.

Thanks for the confirmation and explanation. It would seem that your only real competitor is ddrescue. ;-)

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 18:08 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15462
Location: Australia
maximus wrote:
And all of this is even without talking about the ability to power cycle. And I am curious about these lines in the features:

Quote:
Customizable read timeout duration in milliseconds.
Read timeout enforced by software reset, hardware reset, controller reset, or drive repower.

Other than a power cycle, I am curious what other kind of effective resets can be done with USB. I know there is a command to reset the USB port, but does it also really cause a reset to be sent to the drive itself, particularly in the case of it basically being a SATA device with a USB board (not a flash drive)?

These two documents have conflicting information, probably because the standard was still under discussion:

http://t10.org/ftp/t10/document.04/04-262r1.pdf
http://ftp.t10.org/ftp/t10/document.04/04-262r8.pdf

However, it would appear that one should be able to send software (SRST) and hardware (HWRST) resets via ATA Command Pass-Through.

Quote:
When the HWRST bit is set to one, the SCSI device ignores all other fields with the exception of Off.Line. The host controller shall issue a hard reset to the device. For PATA this is a Pin 1 reset, for SATA this is a COMRESET.

Quote:
When the SRST bit is set to one, the bridge ignores all other fields with the exception of HWRST and Off.Line. The ATA device shall execute a soft reset (SRST) as defined in ATA/ATAPI-7 unless the HWRST but is also set to one. If the HWRST bit is set to one, SRST is not performed.

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 18:43 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
fzabkar wrote:
However, it would appear that one should be able to send software (SRST) and hardware (HWRST) resets via ATA Command Pass-Through.

First, I have not been able to get that to work in Linux, the passthrough always responds with invalid command. Trust me, I have tried.

Second, those resets can only be done at the hardware level of the driver, you can't just pass them through USB via SCSI commands.

There is a part if me that wants to call BS on the soft and hard reset capability that is claimed. But they MUST have some form of resets that do something at the USB level that could be considered similar in some way to a soft and hard reset, to be able to make that claim. But they make it sound the same as a hard drive soft and hard reset. Smoke and mirrors I say...

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 19:49 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15462
Location: Australia
I'm not sure I understand. If DeepSpar intercepts the USB connection between the host and device, why should it not be able to independently implement the SRST and HWRST features in the pass-through standard?

As for Linux's rejection of your attempts to implement SRST and HWRST, which words in the pass-through command packet can the user control? If some words/bytes are under your control, then why not all of them?


Attachments:
ATA_pass_through_protocols.GIF
ATA_pass_through_protocols.GIF [ 21.03 KiB | Viewed 21065 times ]
ATA_pass_through_16.GIF
ATA_pass_through_16.GIF [ 31.89 KiB | Viewed 21065 times ]
ATA_pass_through_12.GIF
ATA_pass_through_12.GIF [ 25.17 KiB | Viewed 21065 times ]

_________________
A backup a day keeps DR away.
Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 20:08 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
fzabkar wrote:
I'm not sure I understand. If DeepSpar intercepts the USB connection between the host and device, why should it not be able to independently implement the SRST and HWRST features in the pass-through standard?

As for Linux's rejection of your attempts to implement SRST and HWRST, which words in the pass-through command packet can the user control? If some words/bytes are under your control, then why not all of them?

To perform soft or hard resets on a SATA drive requires directly manipulating bits in the AHCI IO memory. How could you possibly do that through USB? There are no commands for that. The IO part of the passthrough is controlled by the OS at the driver level.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 20:13 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
To answer what I have control of in Linux for the passthrough, the answer is I can set all fields.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 20:28 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
To further answer about the lack of support of the passthrough resets, I asked myself why it was not supported like it should be in Linux. And after really thinking about it, along with my experiments, a possible answer became somewhat clear. Performing some sort of reset using the passthrough could possibly interrupt the normal operations of the OS, and cause a system crash. That is the best reason I could come up with for the resets not being supported.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 20:46 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15462
Location: Australia
I wonder if the developers/programmers were at all confused by the docs:

http://t10.org/ftp/t10/document.04/04-262r1.pdf


Attachments:
Short_ATA_CDB.GIF
Short_ATA_CDB.GIF [ 24.84 KiB | Viewed 21048 times ]
ATA_CDB.GIF
ATA_CDB.GIF [ 27.57 KiB | Viewed 21048 times ]
ATA_extended_CDB.GIF
ATA_extended_CDB.GIF [ 29.35 KiB | Viewed 21048 times ]
ATA_protocols.GIF
ATA_protocols.GIF [ 23.16 KiB | Viewed 21048 times ]

_________________
A backup a day keeps DR away.
Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 23rd, 2019, 21:35 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15462
Location: Australia
AIUI, CDBs transmitted via ATA pass-through are transparent to the OS. That is, the packet is delivered to the target without its contents being inspected by the host. I could be misreading the standard, though.

Let's say that the OS retries a standard Read command (via SCSI-ATA translation) up to 15 times. If that same unreadable sector is accessed via an ATA Read Sector command encapsulated within a CDB packet, I would think that the OS would be unaware that the drive is processing a read. Is it possible that pass-through errors are processed in a different, less aggressive manner?

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 24th, 2019, 8:03 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
In the OS, you have to wait for the current command to finish and return before you are even able to send the next command, so there is no way to send a reset on a timeout basis, even if the resets did work. There is also a timeout field that is about totally useless. I never tried the reset on a USB drive. So if it were to be possible, it would need to be done at the raw USB level to be able to do it on a timeout basis. I actually have some hidden USB commands in hddsupertool for experimentation, but to perform actual commands on a storage device, I have to copy/clone the raw data from sniffing USB communication, because I do not know what the fields are. I can perform an identify device command, but I copied it from sniffing. I wanted to take it further, but I got stuck when I couldn’t find any documentation that explained what I needed to know.

Right now I have more pressing things to work on in HDDSuperClone (a few reported issues). So until I get those things taken care of, you could help me by finding some good USB documentation. I am looking for something that shows all the fields and commands for raw communication, specifically for bulk read and write for a mass storage device. I looked in the past and came up short. I found places that looked like they used to have the documentation, but it seemed to be replaced with newer documentation that did not have that info. Google failed me :(

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 25th, 2019, 14:30 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15462
Location: Australia
maximus wrote:
In the OS, you have to wait for the current command to finish and return before you are even able to send the next command, so there is no way to send a reset on a timeout basis, even if the resets did work.

What about enabling NCQ with a queue depth of 2? The first command would be a read, while the second could be an SRST or HWRST. Would this abort any "hung" command in the queue?

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 25th, 2019, 14:48 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Nice try, but my attempt at NCQ commands with the passthrough also resulted in an invalid command, or invalid cdb, something was invalid, just like the resets, it doesn't work in Linux. If I am going to play with USB drives in an advanced way, it must be raw USB.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 25th, 2019, 14:49 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15462
Location: Australia
maximus wrote:
... you could help me by finding some good USB documentation. I am looking for something that shows all the fields and commands for raw communication, specifically for bulk read and write for a mass storage device. I looked in the past and came up short. I found places that looked like they used to have the documentation, but it seemed to be replaced with newer documentation that did not have that info. Google failed me :(

Can you elaborate on those particular documents, ie URLs? Did you try the Wayback Machine for earlier versions?

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 25th, 2019, 15:20 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
I don't have any specifics about my past search, I didn't keep track since I didn't find what I was looking for. I did not try the Wayback Machine either. I do know my search was for something along the lines of information for the USB Mass Storage Class. I don't remember my search criteria, but I was looking for the command format / protocol. I may have not stumbled on the proper search terms, but it was not for lack of trying at the time.

Here is an example of a raw identify device passthrough command. It is 31 bytes which is sent as a raw write, then a raw read of 512 bytes retrieves the results from the identify device command. I can see where the command (0xec) is and the ata passthrough (0xa0). But I have no idea about the rest. Find me documentation that explains that whole 31 byte command packet, or at least find me the proper search terms that get results.
Code:
0x55 0x53 0x42 0x43 0x2f 1 0 0
0 2 0 0 0x80 0 0x0c 0xa1
8 0x2e 0 0 0 0 0 0xa0
0xec 0 0 0 0 0 0

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 25th, 2019, 16:06 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15462
Location: Australia
ISTR seeing some threads, in other forums, that discussed these USB packets. I also think HaQue posted something in this forum, but I haven't been able to find it. I'll start searching earnest, but I'm almost certain that the following excerpt from the command packet is the taskfile. That is, 0xA0 is the Device/Head register.

    0 0 0 0 0 0xa0 0xec

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 25th, 2019, 16:24 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Opps, it is the 0xa1 that is the ata passthrough, 0x0c is the command size. It is the stuff leading up to that I need to know.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New DeepSpar USB Stabilizer
PostPosted: March 25th, 2019, 16:37 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
0x80 is likely either usb direction in or usb endpoint in.

Edit: I could be wrong about the 0x80...

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Last edited by maximus on March 25th, 2019, 16:41, edited 1 time in total.

Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 104 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 22 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group