g-list sector numbers and "hdparm --read-sector"

[linux-hdd]

I've got a Fujitsu HD whose bad sector number count has skyrocketed so I want to experiment with it a little, given that I don't intent to give it a normal usage again. I think it's a good oportunity to put to use some of those hdparm dangerous parameters one would never use otherwise.

With hdparm I'd like to put into practice parameters such as --read-sector, --repair-sector and --write-sector. Quoting from hdparm's manual page:

--read-sector
Reads from the specified sector number, and dumps the contents in hex to standard output. The sector number must be given (base10) after this flag. hdparm will issue a low-level read (completely bypassing the usual block layer read/write mechanisms) for the specified sector. This can be used to definitively check whether a given sector is bad (media error) or not (doing so through the usual mechanisms can sometimes give false positives).

1) I'd like to know to what extent this low-level read bypasses the g-list (Grown Defect List). Let's say that sector number X has been labeled as "bad", put into the g-list and remapped to sector Z. If I issue "hdparm --read-sector X", what data will the computer try to fetch: that from sector X or that from sector Z?

--make-bad-sector
Deliberately create a bad sector (aka. "media error") on the disk. EXCEPTIONALLY DANGEROUS. DO NOT USE THIS FLAG!! This can be useful for testing of device/RAID error recovery mechanisms. The sector number is given as a (base10) parameter after the flag. Depending on the device, hdparm will choose one of two possible ATA commands for corrupting the sector. The WRITE_LONG works on most drives, but only up to the 28-bit sector boundary. Some very recent drives (2008) may support the new WRITE_UNCORRECTABLE_EXT command, which works for any LBA48 sector. If available, hdparm will use that in preference to WRITE_LONG. The WRITE_UNCORRECTABLE_EXT command itself presents a choice of how the new bad sector should behave. By default, it will look like any other bad sector, and the drive may take some time to retry and fail on subsequent READs of the sector. However, if a single letter f is prepended immediately in front of the first digit of the sector number parameter, then hdparm will issue a "flagged" WRITE_UNCORRECTABLE_EXT, which causes the drive to merely flag the sector as bad (rather than genuinely corrupt it), and subsequent READs of the sector will fail immediately (rather than after several retries). Note also that the --repair-sector flag can be used to restore (any) bad sectors when they are no longer needed, including sectors that were genuinely bad (the drive will likely remap those to a fresh area on the media).

2) I take for granted that "hdparm --repair-sector" will try to remove the original physical sector (i.e. the sector you could normally access before it was labelled "bad", added to the g-list and remapped) from the g-list. If it was genuinelly bad, it will most likely be re-added to the g-list and remapped again, it it wasn't (for instance, if you intentionally added it to the g-list yourself), you will be able to access that area again. I would like to test this a little, but I need sector numbers to give as parameters: how can I get the sector numbers of my g-list sectors? Smartmontools gives me the bad sector count, but it doesn't provide me with the info I need (their sector numbers), can this information (i.e. the g-list reallocation table) be accessed? If so, how?

Thanks in advance.

[drc]

That's not how it works. You don't add things to the G-List yourself. Defect handling occurs in a manner that is transparent to the user, so when a sector is remapped all you know is that a sector is remapped (by noting increase in SMART values, for example). Not which one it was, etc. Total LBA is the same as before but the physical address (which the user is not privy to) of one LBA has changed. Once it has been remapped you don't get access to the original one anymore.

[linux-hdd]

That's not how it works. You don't add things to the G-List yourself. Defect handling occurs in a manner that is transparent to the user, so when a sector is remapped all you know is that a sector is remapped (by noting increase in SMART values, for example).

I know that it works just like you have just perfectly described. However, there's one exception: "hdparm --make-bad-sector" will add to the G-List the sector number you give as an argument, either by physically damaging it or by flagging it (which is somewhat tantamount to adding it to the G-List yourself). From hdparm manual pages (--make-bad-sector): "if a single letter f is prepended immediately in front of the first digit of the sector number parameter, then hdparm will issue a "flagged" WRITE_UNCORRECTABLE_EXT, which causes the drive to merely flag the sector as bad (rather than genuinely corrupt it),"

Once it has been remapped you don't get access to the original one anymore.

Most of the time you don't. Though, again, there are exceptions to the rule: if somehow you edit the G-list and remove the entry where such sector has been defined as "bad". That's what "hdparm --repair-sector" would do (quoted from hdparm's manpages: --repair-sector flag can be used to restore (any) bad sectors when they are no longer needed, including sectors that were genuinely bad (the drive will likely remap those to a fresh area on the media). Nonetheless, without giving the sector number as an argument, it cannot obviously do a damn thing, hence my original question: how can I get to know the physical address number of the sectors in my G-list? There must exist some way to query that information, otherwise these hdparm options wouldn't have been created this way (i.e. expecting a sector number as an argument). I know this could be done using PC-3000 in order to gain access to, and make sense of, the HD area where this information is stored. However, I'd like to know whether there's a more straightforward method (i.e. a program that not only queries the bad sector count but also the physical sector numbers).

[drc]

You are incorrect.

Users do not get to address sectors in any way other than LBA (without going into vendor specific modes etc)

If you corrupt ECC data with a write_long or equivalent (hdparm --make-bad-sector), then attempting to read that sector will produce a UNC error. If you then successfully overwrite the sector normally, attempting to read the sector will not produce a UNC error. If you have a UNC sector that you can not successfully overwrite normally (in the case of a genuine bad sector, not a fake bad sector produced by write_long etc) the drive will remap it (add to G-List). Simply corrupting ECC data on its own does not cause the drive to remap the sector. In no cases are sectors removed from G-List.

In any event this is handled by drive firmware, not the user.

Your quote from hdparm manpage (--repair-sector flag can be used to restore (any) bad sectors when they are no longer needed) is referring to sectors that you have made "bad" by corrupting ECC data, not any genuinely bad sectors. Since these sectors are only "bad" in the sense that they will read as UNC, they are not actually liable for remapping. "Repairing" them with hdparm only allows them to be read again because you have removed the invalid ECC data by overwriting the sector normally. Part 2 of the quote (including sectors that were genuinely bad (the drive will likely remap those to a fresh area on the media) indicates that any sectors that were genuinely bad will not be "repaired", but will be remapped (by the drive), which results in that LBA being readable again because it is really a new physical sector.

True, you can obtain access to original sectors that were genuinely bad and therefore remapped by clearing or editing the G-List, but this is outside the domain of user actions.

Last edited by drc on October 4th, 2010, 15:16, edited 1 time in total.

[linux-hdd]

drc,

I am not familiar with the meaning of "UNC error" and "ECC data". These concepts are new to me and I need to understand them before being able to make sense of your post, could you please give a brief explanation?

It seems that somehow I took for granted that hdparm was able to interact with the HD at a much deeper level than it truly does.

True, you can obtain access to original sectors by clearing or editing the G-List, but this is outside the domain of user actions.

And how can a user go about broadening his domain of actions to the extent of being able to edit the G-List? PC-3000?

[drc]

drc,

I am not familiar with the meaning of "UNC error" and "ECC data". These concepts are new to me and I need to understand them before being able to make sense of your post, could you please give a brief explanation?

It seems that somehow I took for granted that hdparm was able to interact with the HD at a much deeper level than it truly does.

Try this paper for info re: WRITE_UNCORRECTABLE_EXT that you were referring to: http://www.t13.org/documents/UploadedDo ... ctable.pdf

For info on drive errors and what they mean you should check the relevant sections of the ATA specification

And how can a user go about broadening his domain of actions to the extent of being able to edit the G-List? PC-3000?

Yes. Tools such as PC-3000 interact with the drives at a vendor-specific command level (i.e. outside of the ATA specification, which is what I was referring to as the normal domain of user actions)

[SuperCPU]

This is a very interesting thread, sorry for reviving it after so much time but I think the best place to ask my questions is here.

In no cases are sectors removed from G-List.

In any event this is handled by drive firmware, not the user.

I guess that applies for adding to G-List also, right?
When exactly the firmware is adding sectors to the G-List and remap them? At boot time or anytime while the computer is working?
Is the BIOS playing any role into handling the G-List?

How can I see the size of the G-List? I mean, I want to track it a bit, to check if the next month the G-List is bigger than today, that means the hard disk has more and more bad sectors and it might fail soon. And I'm also simply curious how big is the G-List of my hard drive.

If the bad sector handling and remapping is done by the firmware, then what is the role of chkdsk and fsck?

CHKDSK /R Locates bad sectors and recovers readable information

Can CHKDSK detect physically bad sectors? Or it detects only the sectors with corrupt ECC data (software error)?
Does CHKDSK communicate with the firmware, telling to the firmware that it found a bad sector?

Is it true that the bad sectors are remapped to a special reserve pool of sectors and that when the pool is depleted the disk should be replaced?
I imagined that a bad sector will be remapped to the last available sector on the disk, reducing the total capacity of the disk by one sector.

And a question about spying:
I have seen news about Microsoft closely working with the government (FBI, etc) to include specific software for monitoring the users in Windows (version 7 and above)
In theory, if FBI thinks you have some sensitive data on your disk and that the data can incriminate you (if you use Dropbox, Skype or many other client-server programs then they probably can see everything you have on your disks), it can ask Microsoft to mark the sectors where those files are located as bad sectors (adding them to the G-List) so you can't delete the files.
Do you think such things are practiced already?

Thanks

[HaQue]

In theory, if FBI thinks you have some sensitive data on your disk and that the data can incriminate you (if you use Dropbox, Skype or many other client-server programs then they probably can see everything you have on your disks), it can ask Microsoft to mark the sectors where those files are located as bad sectors (adding them to the G-List) so you can't delete the files.
Do you think such things are practiced already?

Thanks

Possible? of course.

I seriously doubt Microsoft would be involved in adding anything to the GList or anything similar. I imagine that creating this sort of malware is high end, for a certain purpose and high targets. State sponsored type thing, stuxnet, Duqu etcetera.. This type of thing would be developed by external contractors, such as the stuff metioned in the HBGary leaked emails I suppose.

if you want to see what is Possible, checkout this talk from Monk at Recon 2013 that shows how to theoretically hide data in "Bad Blocks" that are not really Bad Blocks of NAND Flash. Sponsored by the DARPA funding, so the gubment is interested in it I guess...
Title: "Hiding @ Depth: Exploring & Subverting NAND Flash memory. Josh 'm0nk' Thomas"
PDF: http://recon.cx/2013/slides/Recon2013-Josh%20Thomas-Hiding%20@%20Depth%20-%20Exploring%20&%20Subvertion%20NAND%20Flash%20memory.pdf
Video: https://archive.org/details/Hiding_at_Depth
Project: https://github.com/monk-dot/NandX

or this talk from Jeroen at OHM2013 https://ohm2013.org/site/ on Hard disk Firmware Shenanigans:
Video: http://achtbaan.nikhef.nl/events/OHM/video/d2-t1-13-20130801-2300-hard_disks_more_than_just_block_devices-sprite_tm.m4v
homepage: http://spritesmods.com/?art=hddhack
Preso page: https://program.ohm2013.org/event/67.html

IIRC this talk references us here at HDDGuru. (well one of the talks I recently watched did, i think it was this one!)