All times are UTC - 5 hours [ DST ]


Forum rules


Please do not post questions about data recovery cases here (use this forum instead). This forum is for topics on finding new ways to recover data. Accessing firmware, writing programs, reading bits off the platter, recovering data from dust...



Post new topic Reply to topic  [ 53 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 2nd, 2014, 11:39 
Offline

Joined: March 1st, 2011, 8:51
Posts: 76
Location: Australia
I was doing some research and noticed (probably old news for some) there is a new "Rebuild Assist" feature in "Serial ATA Revision 3.2" which is intended to improve RAID rebuilds when a drive fails, but could also be beneficial for us in data recovery.

Although the "Serial ATA Revision 3.2" document is not published freely by Serial ATA International Organization, there is some interesting information around the place:

http://www.seagate.com/files/staticfile ... covery.pdf

"...Upon this command, the drive will eliminate any unnecessary background activity, determine if it contains any unusable heads, write-protect the media, and enter a special mode that includes limiting error recovery to the free retries. This mode will stay active until the drive is power-cycled..."

"...As a contributing member to leading industry standards bodies, Seagate has submitted an open-standards proposal of the RAID Rebuild™ functionality under the name Rebuild Assist to the T103 (SAS Proposal: 11-298) and SATA-IO4 (SATA Proposal: SATA31_TPR_D144) committees for inclusion in their published standards specification..."

http://www.hgst.com/tech/techlib.nsf/te ... Rev1.0.pdf

"...The Rebuild Assist mode provides a method for a host controlling the rebuild process to determine that logical sectors on the failed device are unreadable without having to read every LBA to determine the unreadable logical sectors (i.e., the read command is terminated with an error and the failed LBA is reported in the sense data)..."

I wonder if we will start to see hardware imagers supporting this feature.


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 8th, 2018, 17:39 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Cris wrote:
I wonder if we will start to see hardware imagers supporting this feature.

It has been 4 years since this post. So do any of the hardware images implement this yet? Might be on my bucket list now :wink:

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 9th, 2018, 19:22 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
It seems that there are a few HGST models that support this new feature. And as @fzabkar has pointed out to me, the new HGST Helium drives support this feature. I could not find any direct evidence that Seagate has actually produced any drives that support this feature, even though they appear to have been involved with the idea early on.

I have a HGST drive that should support this feature on its way to me to test with, so that I can try to implement this into hddsuperclone.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 9th, 2018, 20:19 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15440
Location: Australia
maximus wrote:
I could not find any direct evidence that Seagate has actually produced any drives that support this feature ...

Rebuild Assist appears to be a feature of some Seagate SAS drives. Seagate muddies the waters by calling it "RAID Rebuild", in which case it is also a feature of Exos X12 SATA models.

https://www.seagate.com/files/www-content/datasheets/pdfs/exos-x-12-DS1946-2-1712US-en_US.pdf

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 9th, 2018, 21:00 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
fzabkar wrote:
maximus wrote:
I could not find any direct evidence that Seagate has actually produced any drives that support this feature ...

Rebuild Assist appears to be a feature of some Seagate SAS drives. Seagate muddies the waters by calling it "RAID Rebuild", in which case it is also a feature of Exos X12 SATA models.

https://www.seagate.com/files/www-content/datasheets/pdfs/exos-x-12-DS1946-2-1712US-en_US.pdf

Go figure. And if you look at the bottom of the document, you can find where Seagate is calling the "RAID Rebuild" feature theirs...
Quote:
Seagate RAID Rebuild and Seagate Secure are either trademarks or registered trademarks of Seagate Technology LLC or one of its affiliated companies in the United States and/or other countries.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 13th, 2018, 19:14 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
The drive arrived, and according to the identify device results it does support the rebuild assist feature. The DOM is August 2015, so HGST has been making drives that support this feature for at least 3 years now. Now I just have to find the time to test, and this will take some work so I don't expect to have results right away.

The drive is HGST Ultrastar 7K6000 HUS726040ALE614 4TB SATA. The SMART is pretty clean, only 11 power on hours. It was sold as "New other, under 100 power on hours". Just curious, how easy and common is it for someone to clear the SMART on these and sell as newer than they really are? I am thinking of making it a backup drive when I am done playing with it.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 14th, 2018, 16:54 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15440
Location: Australia
maximus wrote:
The drive is HGST Ultrastar 7K6000 HUS726040ALE614 4TB SATA. The SMART is pretty clean, only 11 power on hours. It was sold as "New other, under 100 power on hours". Just curious, how easy and common is it for someone to clear the SMART on these and sell as newer than they really are?

AIUI, these models are not yet supported by any data recovery tool. Therefore I doubt that many people would know how to access the SA on these models.

One possible scenario is that your drive was purchased as a RAID spare.

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 16th, 2018, 19:14 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Quote:
Well ... Thos newer HGST drives still support the old IBM command to access the "SUPER SMART" so most likely the clear SMART command will work as well ... Its just a question of testing that out. Issue the super smart on and the clear S.M.A.R.T. check if it does work ... Most likely it will work even on the latest HGST He filled drives ...
That is interesting to know.

Quote:
Also can you please DUMP the ROM content of your flash chip ? I might figure out the Super On VSC from the ROM dump.
I don't have any firmware tools or chip reader to do that. So unless there is a free firmware tool or VSC you can point me to that can dump the ROM, it won't happen.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 17th, 2018, 20:24 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Quote:
Are you willing to do some testing on your drive ?
http://www.hddoracle.com/viewtopic.php?f=118&t=2171
On the link you have the commands to enable super SMART and then clear SMART. Can you test this and check if it does work with your (modern) HGST drive ?
The results do show that some smart data is cleared. But the power on hours did not clear. Both outputs are with super smart on.

Before clearing smart:
Code:
ID#   FLAG  VALUE WORST THRESH   RAW DATA          ATTRIBUTE NAME
  1  0x000b  100   100    16   0x00000000000001   Read Error Rate
  2  0x0005  135   100    54   0x00000000000070   Throughput Performance
  3  0x0007  146   100    24   0x00000801970142   Spin-Up Time
  4  0x0012  100   100     0   0x0000000000001f   Start/Stop Count
  5  0x0033  100   100     5   0x00000000000000   Reallocated Sectors Count
  7  0x000b  100   100    67   0x00000000000000   Seek Error Rate
  8  0x0005  128   100    20   0x00000000000012   Seek Time Performance
  9  0x0012  100   100     0   0x00000000000030   Power-On Hours Count
10  0x0013  100   100    60   0x00000000000000   Spin Retry Count
12  0x0032  100   100     0   0x0000000000001d   Power Cycle Count
191  0x000a  100   100     0   0x00000000000000   G-Sense Errors
192  0x0032  100   100     0   0x00000000000020   Power-Off Retract Cycles
193  0x0012  100   100     0   0x00000000000020   Load/Unload Cycles
194  0x0002  253   130     0   0x00003000140016   Temperature
196  0x0032  100   100     0   0x00000000000000   Reallocation Events
197  0x0022  100   100     0   0x00000000000000   Current Pending Sectors
198  0x0008  100   100     0   0x00000000000000   Off-line Uncorrectable
199  0x000a  200   200     0   0x00000000000000   UDMA CRC Error Rate
211  0x0002  100   100     0   0x00007800160195   Vibration During Write
222  0x0012  100   100     0   0x00000000000029   Loaded Hours
223  0x000b  100   100    50   0x00000000000000   Load/Unload Retries
226  0x0007  187   187    40   0x00000000410002   Load-in Time
230  0x0012   94    94     0   0x00000000100001   GMR Head Amplitude


After clearing smart:
Code:
ID#   FLAG  VALUE WORST THRESH   RAW DATA          ATTRIBUTE NAME
  1  0x000b  100   100    16   0x00000000000000   Read Error Rate
  2  0x0005  100   100    54   0x00000000000000   Throughput Performance
  3  0x0007  100   100    24   0x00000000000000   Spin-Up Time
  4  0x0012  100   100     0   0x0000000000001f   Start/Stop Count
  5  0x0033  100   100     5   0x00000000000000   Reallocated Sectors Count
  7  0x000b  100   100    67   0x00000000000000   Seek Error Rate
  8  0x0005  100   100    20   0x00000000000000   Seek Time Performance
  9  0x0012  100   100     0   0x00000000000030   Power-On Hours Count
10  0x0013  100   100    60   0x00000000000000   Spin Retry Count
12  0x0032  100   100     0   0x0000000000001d   Power Cycle Count
191  0x000a  100   100     0   0x00000000000000   G-Sense Errors
192  0x0032  100   100     0   0x00000000000020   Power-Off Retract Cycles
193  0x0012  100   100     0   0x00000000000020   Load/Unload Cycles
194  0x0002  250   250     0   0x00001900190018   Temperature
196  0x0032  100   100     0   0x00000000000000   Reallocation Events
197  0x0022  100   100     0   0x00000000000000   Current Pending Sectors
198  0x0008  100   100     0   0x00000000000000   Off-line Uncorrectable
199  0x000a  200   200     0   0x00000000000000   UDMA CRC Error Rate
211  0x0002  100   100     0   0x00007800180192   Vibration During Write
222  0x0012  100   100     0   0x00000000000000   Loaded Hours
223  0x000b  100   100    50   0x00000000000000   Load/Unload Retries
226  0x0007  100   100    40   0x00000000000000   Load-in Time
230  0x0012   95    95     0   0x000000000c0004   GMR Head Amplitude

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 17th, 2018, 20:33 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
I would just like to add that I don't want to stray too far off topic here. I see there is a potential that maybe the drive I have could have more power on hours than indicated if someone knew the proper commands. But that is not really important for the topic, so I think I need to get back to the real testing part.

The first step was to figure out how to properly perform a READ FPDMA QUEUED command (required for the rebuild assist). That took me about a full day. Next step is to handle a read error with that command, as it reacts differently.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 18th, 2018, 19:11 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Quote:
And asthis is a ATA standard even on modern drives that are fully locked and that tools like PC-3000 can't work with you might be able to "clone by heads" or skip bad heads using this feature ! Very cool.
I have been bouncing ideas around in my head on a few different ways to utilize this feature, likely offering options to choose from. And one of those is to image by head. And if errors are encountered, it can move on to the next head, or maybe use some sort of aggressive skipping algorithm to skip out of a potential bad spot. I think the aggressive skipping is the way to go at first, because there is always the possibility of a good head giving errors when one of the heads hits physical media damage. I have seen logs where it looks like one surface had a bad spot, and the other heads also showed signs of reading difficulty when in this area.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 25th, 2018, 20:44 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
So according to the ATA standard, this feature requires using the READ FPDMA QUEUED command for the reads when using the rebuild assist. It took me awhile, but I have managed to tame that read command to be able to utilize it. So then I actually tried testing by disabling a head, and it also worked using regular READ DMA commands. Although I would like to point out that to be able to get an LBA value where the "disabled head" ends and the next good read begins does require using READ FPDMA QUEUED. So the testing was looking good...

And then I tried to disable more than one head at a time, and it won't let me! WTF!!! The standard explains it as you just can't disable all read elements, but you can disable multiple read elements. But I can only disable one at a time on this drive, that kills the idea of reading by head :( :? :evil:

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 25th, 2018, 21:32 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
So maybe frustration is a good thing. Now I am thinking of an efficient way to map the heads while only being able to disable one at a time, assuming the return data from some commands follows the standard and gives proper results. The mad scientist in me will make this work one way or another :twisted:

And oh yeah, Merry Christmas! :D

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 25th, 2018, 21:53 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15440
Location: Australia
It sounds like you want to build a virtual translator. Perhaps you could use the following method to determine the size of the serpentine segments, at least for testing purposes.

http://www.hddoracle.com/viewtopic.php?f=59&t=650

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 25th, 2018, 22:15 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
If my idea works, I should be able to use the data from using the rebuild assist to get an exact head mapping, while disabling one at a time. It would be exact based on some return data which I have not verified yet, but if that data does work, the head mapping would be exact. Still a theory at this time, but will be tested soon.

The one thing that could cause headaches is the possibility of reallocated sectors that perhaps are reallocated to a different head, and how they are presented when disabling a head. Not sure if that would ever be an issue, but always a possibility that I have to be aware of.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 25th, 2018, 22:31 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Quote:
It would be exact based on some return data which I have not verified yet, but if that data does work, the head mapping would be exact. Still a theory at this time, but will be tested soon.

Just did a simple test, and the data seems valid and usable. :D
Still much more work and testing to do...

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 26th, 2018, 18:18 
Offline
User avatar

Joined: September 8th, 2009, 18:21
Posts: 15440
Location: Australia
AIUI, if the drive has determined that particular read elements are bad, then it will set the appropriate bits in the Rebuild Assist log (15h). One can then transparently clone the drive without having to skip bad areas. The user can additionally disable a marginally good head, ie one that the drive thinks is OK. However, the problem AIUI is that this particular Hitachi firmware deviates from the ATA standard in that it does not permit the user to disable more than one additional head at any one time.

_________________
A backup a day keeps DR away.


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 26th, 2018, 20:45 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
Spildit wrote:
Ok let's think ...

Assuming a drive have 4 heads and 2 are gone .... let's say head 0 and 3 are ok.

You use Rebuild Assist commands and READ the data. When the READ is send to a LBA that belongs to a head that is damaged the drive will not attempt to read and will send an error, correct ? So you skip that sector and move on ...

Now can't you just use the READ command and the drive itself will know that head 0 and 3 are ok and will return the error on head 1 and 2 ? If so you can read in sequence and get the data out of the good heads getting error on LBAs that belong to bad heads.

G-List should have few entries. Even if you can't get the same error when reading a re-located sector even if the drive does have re-located sectors there will be just a bunch of LBAs and it will be way better than using one enite bad head. Even if some eentries on G-List do end up leading the drive to read sectors on bad head it will be just a few of them as G-List can have just a bunch of entries as the drive does have very limited number of spare sectors ...

Yes, that is more or less how it would work, in the easy raw form.



fzabkar wrote:
AIUI, if the drive has determined that particular read elements are bad, then it will set the appropriate bits in the Rebuild Assist log (15h). One can then transparently clone the drive without having to skip bad areas. The user can additionally disable a marginally good head, ie one that the drive thinks is OK. However, the problem AIUI is that this particular Hitachi firmware deviates from the ATA standard in that it does not permit the user to disable more than one additional head at any one time.

Exactly. Another thing to realize is that if you want to disable a marginally good head yourself, the only way to know which head to disable is by trial and error. There is no direct way to know which head any bad sectors are in until you start switching off heads one at a time to see which one makes the read different.


Spildit wrote:
Ok ...
Let's see how this ends up implemented in hddsuperclone !
:D

Yes, let’s see how I can manage to do this in a beneficial way! My current idea is to be able to create a head map using the available functions. Because I can only disable one head at a time, I won’t be able to fully test it in a scenario where the drive itself disables one or more heads. There is also a possible timing issue, where every time you enable the rebuild assist to switch heads, there is a delay. I have not officially timed it, but I am guessing between ¼ and ½ second. That can add up, I did the math on my first idea, and it could easily take 30-45 minutes to map the heads that way on my test drive. So now I have another idea to speed things up, but it starts to get more complicated.

Today was not very productive for me in a programming sense. So I used it as a think day, running ideas around in my head when I had the chance. I think I might have a good idea how to map the heads in a reasonable amount of time, but I won’t know until I actually get the programming in place. My ultimate goal is to be able to map the heads, and then be able to perform read operations knowing exactly which areas belong to what head, giving the program the control to handle the operation of skipping areas and head(s) as needed.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 28th, 2018, 0:03 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
I would like to point out that the head mapping will only work in direct AHCI mode in hddsuperclone. I did a quick test of the READ FPDMA QUEUED command using the ata-passthrough, and it choked as I expected. Those NCQ commands are a different kind of animal at the driver level.

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
 Post subject: Re: New "Rebuild Assist" feature in Serial ATA Revision 3.2
PostPosted: December 29th, 2018, 21:13 
Offline

Joined: January 29th, 2012, 1:43
Posts: 982
Location: United States
So I have now run into another issue. My test drive indicates it has 9 heads that can be disabled (0-8). This is according to the results of reading the rebuild assist log. But if I disable head 8, the rebuild assist completely quits working until I start over. I thought maybe it could have 5 platters with 9 heads, but when I just skip trying to disable head 8, everything lines up using heads 0-7, just like it should for a 4 platter drive with 8 heads. This is getting more and more frustrating. I am thinking of putting this on hold for now. This drive is no good for testing. Maybe now I know why no other tools are implementing this feature. I can’t implement head mapping using a test drive that is non-conforming.

And just to be clear, the drive will seemingly allow me to disable the mystery head 8, just as it should according to the bitmap returned by the rebuild assist log. It won’t let me disable any head that is not in the bitmap, attempting that will return with an aborted command as it should. So it is reacting properly according to what heads are available to be disabled. It just seems to have a phantom head that messes everything up. It is possible that while this drive appears to support the rebuild assist, it may have been in a beta stage at the time of manufacture, and therefore has issues.

So I think it is mission aborted for now, and time for a few beers… maybe more than a few :)

_________________
http://www.hddsuperclone.com
Home of HDDSuperClone


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 53 posts ]  Go to page 1, 2, 3  Next

All times are UTC - 5 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group