WD5000AVVS shorting out computer

[MrSpiffdifilous]

Hey guys,

I work for a company that tests and redistributes hard drives. We've been having some issues with WD5000AVVS drives shorting out our equipment. Some so badly that they flame up and destroy the TVS. Some only short the power supply off, but then work fine once the power supply is reset. Does anyone have any idea whats going on here? The most common offending PN's are 63ZWBO, 63M8BO and 63HOB1.

[Vulcan]

Unless there's a faulty batch of TVS on those drives, the behaviour you describe would make me investigate the PSUs & test fixtures/cabling that are being used. Insufficient crowbar sensitivity, and spikes which trigger the TVS in the first place, are specific concerns. Even though only some drives seem to be "affected", that does not mean that the power to the drives is OK (even though not all drives are "affected").

[Amarbir[CDR-Labs]]

Hey guys,

I work for a company that tests and redistributes hard drives. We've been having some issues with WD5000AVVS drives shorting out our equipment. Some so badly that they flame up and destroy the TVS. Some only short the power supply off, but then work fine once the power supply is reset. Does anyone have any idea whats going on here? The most common offending PN's are 63ZWBO, 63M8BO and 63HOB1.

Well,
Best Way is To Test These TVS Diodes with a 10K Range On a analogue Multimeter Like This -> http://www.lynx-india.com/index.php?productID=16969 .

[Vulcan]

Best Way is To Test These TVS Diodes with a 10K Range On a analogue Multimeter

A multimeter can indicate when a TVS is now shorted, or at least low resistance - typically the result of EOS, as we know. However it cannot test other potential failure modes of TVS.

When I mentioned the possibility of "faulty" TVS, I did not mean "shorted". I would have said "shorted" if I meant that. Other TVS-related faults, though IMHO unlikely in the OP's case, are possible and that is what I was referring to.

My main concern is still the PSUs & test fixtures/cabling being used. But without more info from the OP, we're all guessing since we can't see or measure anything about the OP's hardware.

[Amarbir[CDR-Labs]]

Best Way is To Test These TVS Diodes with a 10K Range On a analogue Multimeter

A multimeter can indicate when a TVS is now shorted, or at least low resistance - typically the result of EOS, as we know. However it cannot test other potential failure modes of TVS.

When I mentioned the possibility of "faulty" TVS, I did not mean "shorted". I would have said "shorted" if I meant that. Other TVS-related faults, though IMHO unlikely in the OP's case, are possible and that is what I was referring to.

My main concern is still the PSUs & test fixtures/cabling being used. But without more info from the OP, we're all guessing since we can't see or measure anything about the OP's hardware.

Well Buddy ,
A analogue Meter Can Detect Potentially Failing TVs Also .If you Keep it At 1x Range Its even Better ,I Just Imported a Couple Of VTVM From USA To India

[Vulcan]

A analogue Meter Can Detect Potentially Failing TVs Also .

Really? So how do you detect a 5V or 12V TVS which has a slightly reduced breakdown voltage Vbr than it should have (due to manufacturing error - i.e. a batch problem) using an analogue multimeter? I don't believe it can be done using a multimeter alone, but I look forward to learning, since you claim differently. Thank you.

[Amarbir[CDR-Labs]]

A analogue Meter Can Detect Potentially Failing TVs Also .

Really? So how do you detect a 5V or 12V TVS which has a slightly reduced breakdown voltage Vbr than it should have (due to manufacturing error - i.e. a batch problem) using an analogue multimeter? I don't believe it can be done using a multimeter alone, but I look forward to learning, since you claim differently. Thank you.

Well,
Get One And Try This You Would Be Surprised

[Vulcan]

@MrSpiffdifilous:

So are you going to supply more details of your setup, any results of tests you've already done on the quality of the DC supplies to the drives etc.? Short duration spikes (potentially too short to show on any multimeter, analogue or digital ) would fit with some of your described symptoms, and could be induced from other sources. There's more analysis that could be done, but only if you participate and provide details...

[MrSpiffdifilous]

Thanks for all the responses guys. Sorry for taking so long to get back on here. Work's been crazy. I have testers frying HDD's and PSU's. It's gotten to the point that they burn so often I'm making molex to SATA power adapters in batches to avoid melting all the power connectors off the PSU.

We're wondering if it might be an issue of going from no power to full power too quickly. At the moment our power testing stations use a regular desktop computer 550W PSU("Ultra" brand). The techs just plug power and data cables and run the test, then unplug and repeat. I'm sure I have a multimeter around here somewhere. I'll look into the responses regarding that later today hopefully.

[Vulcan]

Thanks for coming back to us. If you're just going to use a multimeter, then I'll leave you with Amarbir, as I disagree with the usefulness of that approach.

IMHO more history is needed, as well as more config detail - e.g. for how long has this physical setup (with those PSUs) been used? When were problems first seen? Overall, have problems been seen on many / most / all test stations? If problems wasn't seen near the beginning of their use, then what changed between when things were first "OK" and when your reported problems first started? What else is plugged into each PSU apart from a drive? What length are the power cables from PSU to drive - have they been extended?

You mentioned frying HDDs and PSUs - that's new info. What do you mean by "frying" PSUs? Do you mean causing actual internal PSU damage, requiring repair/replacement?

I can guess what may be happening, but I'd be using a DSO (digital storage oscilloscope) to try to confirm or disprove that hypothesis, not a multimeter. Without you having a good DSO, and someone who knows how to "drive" it, then IMHO the accuracy / speed / success of any diagnosis will probably be affected, as more guessing will be involved.

[SAjunky]

I agree completely with Vulcan. Power supply problem, combined perhaps with cabling, type of load, etc.

[MrSpiffdifilous]

What I meant by "frying PSU's" is that they are damaged internally somehow. They have a strong odor of burnt electronics. I haven't really looked too much into what's damaged inside yet. There's a distinct possibility someone just screwed up switching it on with a paperclip, since it was being used as a secondary PSU, and had no switch. I know one person has already made the mistake of connecting Red to Black on the 24pin connector instead of Green to Black.

As for testing setups, not everyones station is the same. Most of them are using the Ultra 550w PSU I mentioned before, though some are using Dell 250W PSU's. Some people have asecondary PSU like I mentioned above, for the purpose of just powering the hard drives and preventing the computer from being shutdown upon a short. Other systems are testing directly off the PSU in the computer. So those are powering motherboard/CPU, CD-ROM, and case fans. The computers do not have hard drives. I have an order in for more PSU's to get every system running a secondary PSU so they stop wasting so much time waiting for the system to reboot.

There doesn't seem to be any sort of pattern to the drives burning up aside from the fact that every one of them has been a Western Digital Green Power 500 GB drive.

Oh and the cables are standard length for any PSU I've ever seen. Some people have started using short adapter cables I made to avoid melting all of the SATA power connectors off the PSU.

[Vulcan]

Thanks for the info. That confirms that, as a minimum, you have PSU-related problems - there is absolutely no fault in a disk drive, which would cause a well-designed PSU, being used correctly, to fail in the way you've described. Perhaps you have some genuinely faulty PSUs?

Based on what you've said, I don't think you have the exact details needed of historical events to properly analyse them e.g. which test stations, with which PSU type(s), had which type of problem(s). If you can go back and gather that info, you may find patterns in test station configurations that were involved in each case, which will help you to better understand the issue(s) (e.g. is it limited to specific test stations, or whether the drive was attached to a "secondary" PSU, or specific PSU models, or specific PSU serial number ranges, etc. etc.).

Unfortunately, as you explained, some human error is also involved. That will make any analysis more difficult, as you probably have at least 2 different problems (human error, PSU problems [see below], perhaps something else too...). The challenge is to understand what happened for each problem, and build-up a picture of events.

However I was concerned that you might be using PC PSUs in exactly that way you've now explaineed - as "secondary" PSUs. Many (most?) PC PSUs aren't designed for that type of usage. Some models (not necessarily the higher wattage ones) work better than others like that, but I've seen some PSUs which would cause exactly the disk drive damage which you are seeing, if used in that way. This is due to them needing a certain minimum load on one or more voltage rails (which is greater than the load provided by a disk drive) in order to properly regulate. A PC motherboard provides the necessary minimum load in a normal PC, but as you said, you're not attaching a motherboard and hence not providing that minimum load that some PSUs need to properly regulate their output voltages, when used as "secondary" PSUs.

Therefore, amongst other possible issues (some of which are more of a concern with IDE than with SATA), results of using PC PSUs in that way and hence with poor output regulation, can include overvoltage on their DC outputs which, if high enough, would then trigger disk drive TVSs (or worse damage), exactly as you described!

If you are seeing disk drive TVSs being triggered on test stations where the PSU is also powering a motherboard, then that is more of a concern, as that is a "normal" configuration. As I said before, you'd need to use a DSO or similar, to see precisely what is happening.

I strongly suspect that those WD drives are not the real problem - just that this model is less able than some others, to cope with an out-of-spec overvoltage on their power input rails, which is where the real problem is.

That's my hypothesis, based on my experiences as an electronics engineer and on the info provided so far - but since I can't do the testing of your equipment to prove or disprove it, the rest is up to you. Hope that helps

[MrSpiffdifilous]

Thank you very much Vulcan. That's very helpful to know. It looks like I'm going to have to start recording every single event going forward to try to get a better idea of what's going on here. I've had my suspicions that the drives weren't neccesarily to blame for the problems. Now it seems that was correct, atleast mostly correct. I have found several drives where the TVS has actually broken one of its solder joints on the board, but that seems to be more the exception than the rule in the overall issue.

[SAjunky]

Just to add to Vulcan post, most of PC power supplies have 5V power line regulation, 12V and 3.3V line is delivered by compensation choke and voltage on these lines depends on the ballance of the load. In your case 3.3V line is not loaded at all. It affects static voltage level, but also can affect dynamic load changes, by example when HDD spindle powers up.
+1 for creating optimum load. Old PC-AT power supply is better suitable, as it doesn't have 3.3V line.
Now the idea. You can protect hard drives from burning up TVS's, if you put resetable switch called PolySwitch in series on 5V and 12V supply lines. In parallel to PolySwitch you can put small bulb or LED indicator, it will light up when PolySwitch breaks circuit. Problem with such indicator is that some PCB's have build-in fuses and if it triggers earlier, you get no indication at all.

[Vulcan]

@MrSpiffdifilous,

Thank you very much Vulcan.

You're very welcome

It looks like I'm going to have to start recording every single event going forward to try to get a better idea of what's going on here.

If the historical data isn't available, or you can't be confident in its accuracy, then good recording of the data from now would certainly be part of my plan too. The pattern of which systems are affected, compared to which are not affected (assuming that those 2 groups have similar numbers of drives being tested, but note my comment below about the "not affected" group) should help to produce a list of suspects for further investigation. If you appear to find that every test station is affected, then you're either seeing multiple causes or one common cause affecting them all...

FYI, as with any intermittent problem, just be careful of believing that a test station is not affected, especially with a small sample size of test results. It's impossible to say with 100% confidence that any non-trivial system is "not affected" with a problem. For example (using some made-up numbers), all we can say is that test station X is probably not affected since after 100 tests, there had been no TVS events, whereas on test station Y there had been 10 TVS events during 100 tests. My point is that we can never know whether (in this example) the 101st test on station X would have caused the first TVS event.

That new testing & recording process will take time and cost money, and perhaps lead to more damaged drives which is a worry, so obviously I was hoping to use data from previous events, since that testing (and damage) has already happened. Therefore to reduce further testing time (and damage) if you can find any test station where you're sure an attached drive's TVS has previously conducted without human error being involved (whether the PSU shutdown or kept powered-on causing the TVS to overheat), I would start investigating that test station with a DSO as a priority, right now. My guess is that you'll (perhaps intermittently? perhaps depending on the length of time the PSU was left powered off?) see an overshoot of one or both power rails, when the PSU powering the drive is switched-on.

One piece of historical data which you might have, and which may help you to narrow down your investigation, is which TVS diode was damaged on previous drives. Was it always the 12V TVS? Always the 5V TVS? A mixture of both? For example, if it was always the 5V TVS, then that would be the DC output rail to start looking for evidence with the DSO, IMHO.

I have found several drives where the TVS has actually broken one of its solder joints on the board, but that seems to be more the exception than the rule in the overall issue.

I'd be interested to see a close-up photo, if you have time to take one, but in general, TVS (and surrounding area) damage depends on the length of time that the TVS diode is conducting. If a drive's PCB doesn't include some kind of fusible link in the supply rail with a TVS (whether it's a fuse, a polyswitch, a zero Ohm resistor or similar) then when the TVS diode conducts, if the PC PSU doesn't shutdown quickly, overheating of the TVS diode & surrounding drive PCB is very likely. This is more likely to happen with higher wattage PSUs. I don't want to prejudge whether those 550W PSUs are causing your original problem or not - DSO investigation is needed to check - but such high wattage PSUs do increase the risk that they won't shutdown when a TVS conducts, depending on exactly how the TVS behaves when abused in that way. Personally I wouldn't be using that type of PSU, but other options is a whole new conversation...

I see I've overlapped with some similar comments from SAjunky while I've been typing this I've got to get back to my day job now - I've got some code to debug... Hope the above comments help, or at least give you some ideas for investigating the problem. Good luck!

[SAjunky]

Forgotten to add that PolySwitches do not eleiminate need for ballanced load.
My second thought is to use for testing station power supply from the typical external 3.5' HDD enclosure. It matches power requirement of HDD and in this way eventual overshots have much lower enegy (ten times - at least).

[MrSpiffdifilous]

Thanks again for all the responses. You guys are great. At the moment we have an order of 250W PSU's being shipped to us for the power testing stations. I'm hoping that we fare a little better with these. I've placed a halt on testing 63ZWBO as they seem to be the main offender, until we can figure out a way to stop destroying so many of them. They're failing(one way or another) at about a 10-1 ratio right now. I've had several techs get second degree burns from the resulting flames, so placing a stop on those drives seems like the safest thing to do all around.

If I come across another TVS that's become seperated from the PCB I'll be sure to snap a picture for you. In most cases I've seen where the TVS failed its been just the 12v, but there have been several where the 5v went as well. Normally if that's the case about half of the board ends up burning up. It may be worth mentioning that on most of the drives I've inspected post-meltdown(and where the TVS wasn't completely destroyed) have the same TVS; LG89A. I just thought that was interesting.

I've talked with a collegue of mine about adding breakers to the 5v and 12v lines before, we just weren't sure it'd make a difference. I'll look more into now.

Getting to SAjunky's comment about 3.5" external enclosures; I actually went out and bought one of these( http://www.newegg.com/Product/Product.a ... 6817153071 ), in the hopes that it might solve our problems. Unfortunately I haven't been able to test it yet since it only comes with a eSATA cable and none of our computers have an eSATA connection at the moment. I have eSATA-SATA adapter cables on order right now to fix this issue. I'll post my findings whenever I get those cables.

[MrSpiffdifilous]

Hey guys, I'm back with an update.

I recieved my shipment of new parts finally. I tried my luck with the Thermaltake BlacX hotswap bay. It seemed to be going pretty well at first, until I hit a drive that (presumably) would have smoked up/caught fire and the unit seems to just not connect anymore. It's still getting power(LED power indicator is still on and drive spins up) but it's no longer communicating with the computer. I'm going to move it to another station to see if it's the computer or the unit before deeming it dead.

My next change was to swap out the 550W PSU's for 250W PSU's. So far my techs are saying they are noticing less drives smoking/catching fire, and in the instances where they do its much less intense than before. So it would seem that the drop in power is helping.

Lastly, one of my techs brought me a 63ZWBO drive that smoked earlier today. i took the board off to inspect the damage. I was surprised to find that the 12v TVS was fine. This time around he fried the 5v TVS at D5. I'm not quite sure why that is. Ideas?

[MrSpiffdifilous]

Just a quick update on the power supplies being used. The 550W PSU's had a max output of 120W on the 3.3v and 5V lines and a 384W max output on the 12v line. The new PSU's have a general max output of 250W regardles of voltage, though the amperage is lower.