I've noticed a few (read: 2 or 3 of 2000+) video files on different HDDs becoming silently corrupt after some time. This happened on both a WD20EARS and ST4000DM000, so I believe the issue is at the software level. I'm using Windows 7 so the drives are NTFS, single volume per physical drive (no RAID).

I did a hex compare between good and corrupt versions of the files and I found blocks of data missing and instead replaced with hex that represent the strings "Microsoft reserved partition" or "Basic data partition" (each character being separated by 0x00, or ".") followed by nulls (0x0). Most recently, I CRC checked a file after downloading and after transferring to the Seagate drive (both were correct).. and then had a CRC mismatch a few days later. There are no SMART errors on either of the drives, and I've owned them for some time (over a year, but under 2 years).

I'm using Auslogics Disk Defrag and my suspicion is that this might be the culprit. It was set to auto-defrag on idle, but I've since disabled it as a precaution. I've used it for many years and never had any problems, so it's a little surprising that issues are cropping up now. Is bad RAM a possibility? (I've used the same motherboard/CPU/memory sticks/etc for 2+ years)

Any thoughts or experiences?

@xviruz, I have seen silent data corruption in cases where a 32-bit LBA limitation arises after a configuration change, but this only happens on 2TB+ drives.

One way to determine whether your system is affected by such a limitation would be to examine sector 0 of your drive with DMDE (freeware disc editor) and then take note of the sector range at the top left of the screen.

For example, the following screenshot shows a range of 0 - 5860533167 (3TB):

http://s5.postimg.org/bwvxdq22d/DMDE_3t ... itions.jpg

_________________
A backup a day keeps DR away.

@Spildit Good question... The problem drives are internal but were always connected via a hotswap bay (CM HAF X), which is an internal board with direct SATA connections to the motherboard.


@fzabkar They are GPT volumes so I don't think that's the issue. Here's the DMDE for my backup 4TB, which has a corrupt file: http://s10.postimg.org/l2hgisl3t/4tb.jpg

The file was also corrupt on my "main" 4TB, but in a different way (i.e., the two corrupt files differ). I've since replaced it and the DMDE is pretty much identical to the above: http://s9.postimg.org/mbyqq3vnj/good_4tb.jpg (this drive has fewer files than the backup)

My PSU's a Corsair TX750, ample power for my setup. It's probably not defective, as I've got several other internal drives (directly via SATA) that have no issues.

The hot swap bay is a simple pass-through PCB: 4-pin molex to power two drives and two SATA ports (pic). I took it out and connected the drives directly. The corruption happens very rarely so I'll have to wait and see.

Also, any idea why the corruption shows up as (I believe) partition table metadata thrown into the file? The 2KB or so block goes pretty much as "Microsoft reserved partition [some hex] Basic data partition ... 0x0s ... EFI PART [more hex]". All the corrupt files I've encountered have had that.

I noticed the exact same problem on my drives.
The corrupt files always differ from the original ones only by a block containing "Microsoft reserved partition" (in the same format the OP described).
I use Seagate NAS and Desktop HDDs, all are 4 TB.
I'm quite sure the files were correct when I copied them on the drives because I verified the copies and I did the backups from the now-corrupt files.
And I did not touch the files at all! No explicit defrag or whatever, just read only operations!
This happened on several files on different drives.
I did a lot of reorganisation on my PC recently, so I don't know when the corruptions happened exactly.

My config: Windows 8.1 x64 + Asus Z87 Expert motherboard + Adaptec 6808H SATA controler + Startech PEXSAT34 SATA controler + Icy Box IB-318StU3-B USB3 enclosure + Icy Box IB-116StU3-B USB3 docking station.
Anything common to your config?

Hmm.. almost nothing in common. I've got Win 7 x64, Asus P8Z68-V Pro motherboard, and my SATA controller is just the built-in Intel Z68 chipset. I'm using 2x4GB of Corsair CMZ4GX3M1A1600C9 RAM + Corsair TX750 for PSU.. on the off chance either are the same. The drives I had issues with were both Seagate (ST4000DM000) and WD (WD20EARS). I've since re-enabled defragging and have been verifying the checksums occasionally---no problems since taking the hot swap PCB out. However, it happens so rarely and I don't write to the drives often enough (maybe a 2 or 3GB a week), so I can't say it's gone for sure.

Maybe it's one of your controllers?

I've been copying my files around with TeraCopy (has checksumming option) and doing checksums of all my files with SFVNinja (+ backup of course). Though I plan to switch my data drives to ZFS/btrfs + RAID. That'll avoid the headache of having to manage these things manually...

I have a Seasonic power supply and a different Corsair memory reference.
I don't think memory or power can be the issue anyway, it would lead to random data and not partition information inserted.

So, I would say it's more a software or firmware problem and I suppose it happens when Windows defrags in the background (with low level instructions that simple copy operations do not use).

My drives are only Seagate ST4000DM00-1F2168 and ST4000VN00-1H4168 (several of both, with different firmware versions).
Are you sure the problem happened on your Western Digital?
A bug in Seagate firmwares would be a good explanation.

A firmware issue in a controler is possible also, but we don't have the same controlers...

I will now restrict my use of controlers to the motherboard (Intel Z87) and my Startech card (which is just a Marvell 88SE9230 controler).
I will do more tests and report the results if I find anything interesting.

See here also:
http://forum.flexraid.com/index.php?topic=3405.0

Well, I just did a check of my WD drives (2TB and 3TB Caviar Greens) and the 3TB had one ~8GB file with that partition junk (exactly 16476 bytes of it). I had the drive sitting on my desk unconnected for the past month or so and connected it just last night to update it (shutdown the computer, plug it in, boot up again). Copied 30GB or so of new data over and then ran a full check. The corrupt file was an old one that was neither defragged nor modified nor read (until the checksum program). There's also no changes in SMART for that drive (no reallocated sectors or read/write errors, etc.). So either it got borked during boot up or, more likely, when writing unrelated data to the drive.

I'll do a check of my Seagates today and see if anything's happened on them.

Interesting, thanks for the link. Bad SATA controller does make the most sense for external cards. Not sure how robust controllers on motherboards are and if problems can appear after years of use (I've not upgraded mobo bios, etc. for years). If the corruption is very rare, maybe it's just a flaky SATA cable/connection?

I do not trust external controllers either. And sometimes even the secondary controller on the motherboard is problematic.

I just changed and attached all my SATA cables to be sure, but I don't think this can be the issue.
Why would we always have the same segment inserted, and why only on untouched files?
This really looks like a software symptom to me. Maybe 4K sectors related?

Your last test with your WD is scary!
I will also do tests according to that (copy other files on a drive and recheck everything).

What else could we have in common? Most of my files are also mkv but it also occured on iso files.
Do you have many drives running at the same time? I have 2 SSD + 6-8 HDD in the PC case.
Did you check for viruses? (I didn't)

I'm not at all well-versed with NTFS details or controller level stuff, so take what I say with a grain of salt. The small size of the corruption does suggest to me that it's a low-level hardware/controller issue. I'm assuming Windows/NTFS tries to create some filesystem metadata, which gets fired off to the SATA controller. The SATA controller then issues a series of low-level commands to the hard drive (the drive also has its own disk controller, which is another level of complexity, but let's ignore that). Then either (1) the SATA controller is the problem and it issues commands with incorrect/malformed addresses; or (2) the controller is okay but the addresses suffer some bit flips on their way to the hard drive, due to some faulty connection, and ECC fails to detect and correct these errors. In either case, because of the incorrect addresses, the metadata gets written to somewhere where it should not go. If it ends up in unused disk space, it will not affect your files and you won't see it as an error. If it ends up in occupied disk space, it will show up inside an existing file (old or new) and you'll see it as a checksum error. Since old files usually take up much more disk space than new files, there's a greater probability of it affecting an old file.

Again, that's all speculation on my part. If the above were true, it would be dumb luck (or good engineering) that there hasn't been an accidental and unrecoverable overwrite of filesystem metadata/journals. That would be disastrous. Also, the reason why I don't suspect it's a software issue (in Windows or in NTFS), is because it's so darn rare---surely other people would've encountered this problem en masse if this were a software bug.

Anyway, to answer your questions... So far I've only seen it happen on mkv files. However, this might be because a good 80% of my data is mkv.

I have all my SATA and e-SATA motherboard ports occupied: 1 SSD, 4 HDDs (2x 1TB, 2TB, 3TB), and a DVD drive. The two remaining SATA ports are usually occupied by 2x 4TBs but I occasionally swap them out for my 2TB and 3TB backups. I don't leave my system turned on 24/7, but when it is on all the connected devices are on as well.

I have anti-malware, AV, and firewall installed. None of them report anything wrong and I'm fairly careful about what I visit/download. Still, it would be pretty silly to write a virus that randomly corrupts files at a rate of 16KB per several TB... and not ask for a ransom .

Well, apparently my post disappeared into the void, so here it is again...

I don't know much about NTFS details or low level controllers, so take what I say with a grain of salt. I'm assuming Windows/NTFS decides at some point to create some filesystem metadata, so it fires off some commands to the SATA controller. The SATA controller then sends low-level commands to the hard drive (which also has its own drive controller that adds more complexity, but let's ignore that). My assumption is that either (1) the SATA controller is bad and sends malformed/incorrect addresses to the drive; or (2) the SATA controller is okay but the addresses have their bits flipped on the way to the hard drive, due to a bad connection, and ECC fails to detect and correct those errors. Either way, due to the incorrect addresses, that filesystem metadata ends up somewhere it shouldn't be. If this happens to be unused disk space, it will not affect any files and so you won't see it as an error. If it ends up in occupied disk space, it will show up inside an existing file (could be old or new) and you'd spot it with a checksum error. Since old files usually take up much more disk space than new ones, there's a greater probability of it showing up in an old file.

Again, that's just my speculation. If it is true, then it's dumb luck (or good engineering) that it hasn't accidentally overwritten enough filesystem metadata/journals to become unrecoverable. I guess occurring rarely helps. Also, I don't suspect it's a high-level software issue (in Windows or NTFS) because of how rare it is---if it were a software bug, I'd expect more people to have encountered this issue.

Anyway, to answer your questions... All of my errors have been in mkv files so far. However, that's probably because 80% of my data is mkv.

I have all my SATA and e-SATA motherboard ports occupied: 1 SSD, 4 HDDs (2x 1TB, 2TB, 3TB), and a DVD drive. The remaining two SATA ports are usually occupied by 2x 4TB but are occasionally swapped out for the backup 2TB and 3TB. I don't leave my computer on 24/7, but when it's on all the drives are on too.

I have anti-malware, AV, and firewall. No problems reported and I'm fairly careful about what I visit/download. Still, it'd be pretty silly to write a virus that randomly corrupts data at the rate of 16KB per few TBs... and not ask for ransom .

If your hardware/low level theory is right, we should have random data as well (bit flipping can also accur on data fields, not only on address fields). And if metadata for the new files are not written at the correct location, we should also have structural problems with the new files (chkdsk seems always ok on my side). And why bit flipping would only occur on metadata packets?

Software glitches CAN occur rarely, like a large counter overflow for example. Or a virus bug. Everything is possible...
Our problem is rare but quite deterministic. That's why I think about a code issue.
The symptoms can also be result of a failed correction of a random hardware error. But this seems less probable to me.

So, we both use many drives at the same time, and I also have two hot-swap bays.
My hot swap bays have an activity LED, so they are not 100% passive.
I don't know if corruptions only occur with them.
I continue my tests...

An update: I did a scan through both my 4TBs, and each had one corrupt file with the exact same symptom. It's always 16400 bytes of this followed by 76 bytes that are structurally similar but have, I'm assuming, different "IDs". An example here and here. Are yours like this? (I used HxD to do a hex compare).

I also realized that the problems I've had so far have been on drives connected to SATA ports driven by the Marvell PCIe SATA controller, rather than the Intel Z68 chipset. So something to watch out for since you've got an Asus as well. I've switched my drives to other ports.

Yes, I notice exactly the same pattern: 16400 bytes + 76 bytes.
But I also have other differences (second line in the examples below).

Corrupt file #1:


Corrupt file #2:


Also, I remember I had "Microsoft reserved partition" instead of "Basic data partition" on some other corrupt files.

What you say about your controller is interesting.
Is it a PCIe card or a controller embedded in the motherboard?
Do you know which Marvell chip is it exactly?
Did you install a driver for it or do you use the default Windows one?

Yeah, by low-level I mean driver or firmware code (which is still "software") rather than higher level OS or virtual file system code. As you say, the corrupt data is deterministic, so I am also doubtful it's a hardware error (i.e., flipped bits not detected by ECC). There are no SMART errors either.

The "Microsoft reserved partition" bit should come right before what you pasted. The hex I linked is the following string ("[...]" truncates some 0x00s):

All my SATA ports and controllers are built-in to the motherboard (I'm not using any external controller cards). Google tells me it's a Marvell 88SE9172 SATA controller. Driver version is 1.0.0.1045. I believe it was obtained from Asus's website but I have not updated for several years. Since we're getting nearly identical errors, you may want to focus on the Marvell controllers in your equipment (e.g., the StarTech PEXSAT34 perhaps?).

I now have my two 4TBs connected to ports driven by the Intel Z68 controller and JMicron JMB362 controller (driver ver 1.17.58.2) respectively. Copied ~30GB over to each, deleted it, and did a full checksum verify. No problems. Time will tell if the problem is actually solved, but hopefully the root cause was in fact the Marvell controller.

One thing I just discovered was a *ton* of Marvell-related errors in the Windows Event Log. For example,

You may want to check your logs for similar errors.

The "EFI PART" and "Microsoft reserved partition" data constitute backup copies of the GPT partitioning information in sectors 1 - 33. The backups are located within the 33 sectors at the end of the drive.

http://en.wikipedia.org/wiki/GUID_Partition_Table

The "EFI PART" GPT header sector defines the partition structure:

http://thestarman.pcministry.com/asm/mbr/GPT.htm

The reasons for the differences include checksums and physical drive capacities, among other things.

0x15d50a3b0 x 512 bytes = 3.00 TB
0x1d1c0beb0 x 512 bytes = 4.00 TB

The following (or preceding) 32 sectors define up to 128 partitions, each entry having a size of 128 bytes.

In short, ISTM that the GPT partitioning data at the end of the drive are corrupting various files on a random (?) basis. Very strange ...

_________________
A backup a day keeps DR away.

Perhaps the corruption isn't really random.

Instead of looking for corrupt files, perhaps we should be looking for corrupt sectors. In other words, maybe the GPT metadata are consistently corrupting the same sectors of the HDD. That would explain why only one file per HDD is corrupt. I would use a disc editor to locate these sectors. I suspect that the corruption might occur around the 2TiB point.

_________________
A backup a day keeps DR away.

I have a theory that might explain the corruption, but it's very speculative.

I'm thinking that the OS may test the consistency of the GPT partition structure at bootup or whenever a drive is attached to a USB port. Specifically, the OS would compare the GPT information at the beginning of the drive (sectors 1 - 33) with the backup copy at the end of the drive. If the backup GPT doesn't match the primary GPT, then the backup would be reconstructed and rewritten to the drive. This would explain why the corrupted data are always the same.

However, it's clear that the original backup GPT data are not corrupt, so ISTM that the OS may be looking for it in the wrong place. Not only that, but the OS also appears to be restoring the corrected GPT data to the wrong place. Sometimes this wrong place falls within an existing file, resulting in insidious corruption.

I'm wondering whether the OS is being affected by a 32-bit LBA limitation during this time. This would mean that only 3TB or 4TB drives would be affected whereas 2TB drives would be immune. One place to look for the bogus GPT copies would be at the 4TB - 2TiB point on the 4TB drive and the 3TB - 2TiB point on the 3TB drive.

For example, the last sector of the 4TB drive is 7814037167 (= 0x1d1c0beaf).

If we ignore the 33rd bit, this becomes 3519069871 (= 0xd1c0beaf).

Therefore I would use a disc editor (eg DMDE) to examine sector 3519069871 of the 4TB physical drive, plus the previous 32 sectors.

Similarly, a 3TB drive would have a sector range of 0 - 5860533167 (= 0x15D50A3AF).

In this case one would examine sector 1565565871 (= 0x5D50A3AF).

_________________
A backup a day keeps DR away.