Post by thread:[EE]: Damaged ICs

Folks: At the company I work for we primarily make PCI-bus PC cards and software to control them (most are designed for Videoconferencing). We have recently received a significant number of field failures with a strange (to us anyway) failure mode. Analyzing the failures, we found that on each board there was one or more IC that had a significantly lower DC resistance at the power pins than is normal. Some boards only have one such device while some have four or five parts doing this. Note that frequently the damaged parts still work if the PCs power supply can source enough current! Most of these are VLSI parts, but occasionally thay are simple 8-bit bus buffers. Most of the parts run from the same 3.3 volt regulator on our boards (which is fed from the PCI bus +5 volt supply), BUT NOT ALL! Some damaged parts are powered by +5 volts from the PCI bus and some are powered from an on-board +5 volt regulator that is powered by the PCI bus +12 volt supply. The only obvious similarity is that all of these failures come from the same customer and is used in a custom PC (which uses an off the shelf switching power supply to generate the power for the PC/PCI-bus & our board). This customer is very important to us, but they are based in a foreign country across the Atlantic. We do not yet have access to their system that our board fails in. Even if we did, it appears that most of the time everything works fine, but every so often they have a system that fails when first powered up at their factory and becomes functional when they replace our board with another. I have been told that they are not seeing any such failures from the field. So it either goes immediately in their system or it stays "good." And we've NEVER seen any failures like this from any other customer; this customer receives less than 25% of all the boards of that type shipped, which makes a design flaw of our board seem less likely. Worse, we are running at about 15% strength right now (over 80% of the people that were with us at this time last year have been laid off). Those of us left are wearing so many hats that we have very little time to spare - and no budget for "unnecessary travel." From the bench, we might see this; A field return is tested on one PC and appears to be operating normally. When placed into another PC (which has a PCI-bus extender card with over current protection & indication) it immediately fails and one of the over current LEDs activate. a good board measures several K-Ohms on the 3.3 volt bus, but one of the failing units measure only a few Ohms to the digital meter. Lifting IC legs one by one eventually identifies the faulty chip(s), and when replaced everything returns to normal. In another case it's the +12 that is over current and it is traced through a +5 volt regulator to one ICs' +5 supply pins. The majority of failures are 3.3 volt parts, then 5 volt, and finally those whose power is derived from +12 volts. The onboard 3.3 volt unit is a Linear Technology LT1506CR-3.3 switching regulator (the dash 3.3 is fixed at 3.3 volts), being employed as per their own specs/schematic. And NONE of these has failed hard; while it is possible that this circuit is failing, it always appears to be functioning properly when we analyze the faulty boards. Additionally, if this was the cause, how would that explain the failures of the parts powered by the PCI bus +5 and +12 volt supplies? And why only this customer? It seems reasonable that the power supply that this customer is using is occasionally doing something that our board finds disagreeable, but what? And why only when first assembled and never in the field? It would be extremely helpful if we had some idea what to ask this customer to investigate... I'm hoping some of you PIC-listers might present some opinions that help solve the problem - or at least help us start asking questions that lead to a solution. Any thoughts? Thanks in advance. Bruce. -- http://www.piclist.com hint: The list server can filter out subtopics (like ads or off topics) for you. See http://www.piclist.com/#topics

> It seems reasonable that the power supply that this customer is using is > occasionally doing something that our board finds disagreeable, but what? > And why only when first assembled and never in the field? It would be > extremely helpful if we had some idea what to ask this customer to > investigate... > > I'm hoping some of you PIC-listers might present some opinions that help > solve the problem - or at least help us start asking questions that lead to > a solution. This is just a wild guess, but maybe their power supply brings the various voltages up in an unusual order, maybe caused by other equipment also installed in the PC. This could cause I/O pins to be driven before the chip is powered up, which could cause latchup and damage to those parts. ***************************************************************** Embed Inc, embedded system specialists in Littleton Massachusetts (978) 742-9014, http://www.embedinc.com -- http://www.piclist.com hint: The list server can filter out subtopics (like ads or off topics) for you. See http://www.piclist.com/#topics

Wow, I have seen a similar fault on PICs, I had a project that would run fine on my 5V power supply but fail in the unit with it's wimpier power supply. I believed at the time that I had some pins that were either zapped by static electricity, or had lost a Bus Fight*. How about overvoltage protection on your board? Is there a chance spiky juice is coming in through the power supply or through an external pin? -- Lawrence Lile *Bus Fight: Usually started when somebody throws an apple core at the bus driver but misses. Olin Lathrop <spam_OUTolin_piclistTakeThisOuTEMBEDINC.COM> Sent by: pic microcontroller discussion list <.....PICLISTKILLspam@spam@MITVMA.MIT.EDU> 01/08/2003 02:06 PM Please respond to pic microcontroller discussion list To: PICLISTKILLspamMITVMA.MIT.EDU cc: Subject: Re: [EE]: Damaged ICs > It seems reasonable that the power supply that this customer is using is > occasionally doing something that our board finds disagreeable, but what? > And why only when first assembled and never in the field? It would be > extremely helpful if we had some idea what to ask this customer to > investigate... > > I'm hoping some of you PIC-listers might present some opinions that help > solve the problem - or at least help us start asking questions that lead to > a solution. This is just a wild guess, but maybe their power supply brings the various voltages up in an unusual order, maybe caused by other equipment also installed in the PC. This could cause I/O pins to be driven before the chip is powered up, which could cause latchup and damage to those parts. ***************************************************************** Embed Inc, embedded system specialists in Littleton Massachusetts (978) 742-9014, http://www.embedinc.com -- http://www.piclist.com hint: The list server can filter out subtopics (like ads or off topics) for you. See http://www.piclist.com/#topics -- http://www.piclist.com hint: The list server can filter out subtopics (like ads or off topics) for you. See http://www.piclist.com/#topics

Olin Lathrop wrote: > This is just a wild guess, but maybe their power supply > brings the various voltages up in an unusual order, > maybe caused by other equipment also installed in the > PC. This could cause I/O pins to be driven before the > chip is powered up, which could cause latchup and > damage to those parts. I think any order of bringing the various voltages up should be considered as usual. Faulty design may depend on this order. But good border design must not depend on this order just using "Power Good" PSU line to properly start-up logic. I'd recommend (also just a wild guess) to investigate initial low level time of the "Power Good" PSU line. Too short time could cause too early starting (voltages aren't within specs) PC and the board logic with all these "two outputs on one line fighting" problems. Also I'd recommend to check if board internal "reset" logic uses "Power Good" PSU line at all. If no then, may be, increasing board internal "resets" could help. Mike. -- http://www.piclist.com hint: The list server can filter out subtopics (like ads or off topics) for you. See http://www.piclist.com/#topics

On Wed, 8 Jan 2003, Bourdon, Bruce wrote: > Folks: > > > The only obvious similarity is that all of these failures come from the same > customer and is used in a custom PC (which uses an off the shelf switching > power supply to generate the power for the PC/PCI-bus & our board). > > This customer is very important to us, but they are based in a foreign > country across the Atlantic. I have ( unfortunately ) some experience in PC switching supply failures, including for those assembled in USA ( Compaq, Dell ) England or China. Depends where is located exactly your customer it could have some computers built with nasty power supplies. I have many doubts about the ESD cause as many piclisters have mentioned. The output impedances for all PS outputs are small enough to eat any moderate ESD stress. Maybe you'll be surprised to hear, but there are *a lot* of PSU which have 4 or 4.4V instead of 3.3V with nominal load. ( Compaq is the leader here ! ) Also there are cheap and ugly PSU designs, where the load hanged on 5V is determining the value of +3.3V which may vary a lot. Don't say about the +/-12V which are flying between +/-9 and +/-13V. You haven't enough information to solve the problem. I suggest to send there a specialist which must check exactly the environement. ( computer's type, mains variations, various tools interfaced to those computers, networks etc. ) best regards, Vasile -- http://www.piclist.com hint: The PICList is archived three different ways. See http://www.piclist.com/#archives for details.

Thanks to Bob, Olin, Lawrence, Mike and Vasile for the replies & thoughts. Several mentioned latchup as the likely cause of the device failures/damage. I tend to agree, in fact this was the opinion of this customers' lead engineer. As to the possibility of using the Power-OK signal from the power supply, unfortunately this will not help if the problem is latchup (unless all power and I/O signals were switched on when the OK signal was active - requiring a magor redesign)... it would help if the cause was bus contention, but that would require a significant design change to a board that is past it's prime and only has this issue with one customer. With our current budget and manpower issues, a redisign is very unlikely. A summarry of the more common triggers for latchup follow, feel free to add to this if you are aware of others. 1.) Supply voltages exceeding thier absolute maximum ratings. 2.) Input or output pin voltage exceeding either supply by more than one diode voltage drop. 3.) Improper power supply sequencing. 4.) Transient irradiation; gamma rays, x- rays and other ionizing radiation (such radiation can induce photocurrents which can active parasitic SCRs within the device). 5.) Electro Static Discharge to an I/O pin or a trace/device connected to an I/O pin (see numbers 2, 1). 6.) Inadequate power supply filtering or transient protection (see number 1). 7.) Coincedences of PCB track layout coupling voltages or inducing currents that result in overvoltage to an I/O pin or supply pin (see numbers 2, 1). It may be possible to add overvoltage protection and power filtering to the system as some suggest, I'll look into this. (I was surprised to read the degree of off voltage that Vasile mentioned!). Thanks again. Bruce. -- http://www.piclist.com hint: The PICList is archived three different ways. See http://www.piclist.com/#archives for details.

Thanks to Alexandre, Mike and Russell - sorry I missed you guys the first time; the "EE" portion of the subject had dropped. I too suspect ESD; I have images of their bench tech tossing these things together without wearing ground straps or taking any precautions for ESD. If zapped while the system is up but still open (perhaps they don't put the covers on until it's ready to ship) it could cause the affected I/Os to SCR... Second, I suspect the power supplies; overvoltage, noise/overshoot, turn-on times, and possibly even significant delays between each voltage coming up (sequencing). It appears that the customer may even be adding wire themselves between the supply and the various loads, if these lengths are excessive or gauges are not sufficient then this could cause problems... One other possibility that is unlikely, but shouldn't be dismissed without at least a cursory look is ionizing radiation. Though I'd expect other faults to occur if this was really to blame, not to mention health issues! Thanks guys, I'll review your comments again and then discuss it further with the customers technical people. Bruce. -- http://www.piclist.com hint: The PICList is archived three different ways. See http://www.piclist.com/#archives for details.

Bourdon, Bruce wrote: > > Thanks to Alexandre, Mike and Russell - sorry I missed you guys the first > time; the "EE" portion of the subject had dropped. > > I too suspect ESD; I have images of their bench tech tossing these things > together without wearing ground straps or taking any precautions for ESD. Hi Bruce, when a silicon junction is damaged it generally goes low ohms, due to fusing of the differing doped layers into a homogenous mass. In severe cases the entire die will burn through and the thing will only have a handful of ohms from + to gnd, like a dead FET. With a complex die (like a micro) you may get one or more silicon junctions fail, more like the reduced ohms from Vdd to Vss that you mention. Having seen ESD in the arcade machines I used to repair it always attacks the pins affected, so it was common to see a 7401 with one nand gate shorted, where that one gate went to the joystick control. I would bet that since your faults seem all related to the Vdd pins that you can rule out ESD and look toward overcurrent through the Vdd path, ie excess current from the psu... Note ESD on the Vdd pin is the LEAST likely situation as the power pins have large capacitors on board and ground planes etc. I really doubt you have to worry about board handling! Try replacing the psu in the unit giving problems, or add a 3.7v 1W zener and prove to the customer that the psu is going overvoltage. :o) -Roman -- http://www.piclist.com hint: The PICList is archived three different ways. See http://www.piclist.com/#archives for details.

Hi, > Having seen ESD in the arcade machines I used to > repair it always attacks the pins affected, so it > was common to see a 7401 with one nand gate shorted, > where that one gate went to the joystick control. > > I would bet that since your faults seem all related > to the Vdd pins that you can rule out ESD and look > toward overcurrent through the Vdd path, ie excess > current from the psu... > > Note ESD on the Vdd pin is the LEAST likely situation > as the power pins have large capacitors on board > and ground planes etc. I really doubt you have to > worry about board handling! In most microcontroller boards where I used the old "piezo lighter" trick to test weak points for ESD induced failures I mostly see the CPU failing in destructive ways when a latchup occurs. PIC's are nice and usually come back working once you take out power supply and repower the board but many others just go as if you have a short between VCC and GND. I think that smaller geometries would have a bigger tendency to more dramatic failures. What I imagine that is happening is that he is having ESD induce failure coming into the chip trough one of the outside connected pins or the operators hands and it is burning part of the chip and the latchup condition does the rest and burns the chip by overtemperature and just than you have the small ohms reading on the meter. If the power supply is as bad as it would need to be to create the failure it would also blow the motherboard chipset ! If the client is not lying about the problem being only on his boards and nowhere else then ESD makes much more sense than power supply problems. Best regards, Alexandre Guimaraes -- http://www.piclist.com hint: The PICList is archived three different ways. See http://www.piclist.com/#archives for details.

----- Original Message ----- From: "Bourdon, Bruce" <.....bbourdonKILLspam.....ZYDACRON.COM> To: <EraseMEPICLISTspam_OUTTakeThisOuTMITVMA.MIT.EDU> Sent: Wednesday, January 08, 2003 2:48 PM Subject: [EE]: Damaged ICs > Folks: > > At the company I work for we primarily make PCI-bus PC cards and software to > control them (most are designed for Videoconferencing). > > We have recently received a significant number of field failures with a > strange (to us anyway) failure mode. > > Analyzing the failures, we found that on each board there was one or more IC {Quote hidden}> that had a significantly lower DC resistance at the power pins than is > normal. Some boards only have one such device while some have four or five > parts doing this. > > Note that frequently the damaged parts still work if the PCs power supply > can source enough current! > > Most of these are VLSI parts, but occasionally thay are simple 8-bit bus > buffers. > > Most of the parts run from the same 3.3 volt regulator on our boards (which > is fed from the PCI bus +5 volt supply), BUT NOT ALL! Some damaged parts are > powered by +5 volts from the PCI bus and some are powered from an on-board > +5 volt regulator that is powered by the PCI bus +12 volt supply. > > The only obvious similarity is that all of these failures come from the same > customer and is used in a custom PC (which uses an off the shelf switching > power supply to generate the power for the PC/PCI-bus & our board). > > This customer is very important to us, but they are based in a foreign > country across the Atlantic. We do not yet have access to their system that > our board fails in. Even if we did, it appears that most of the time > everything works fine, but every so often they have a system that fails when > first powered up at their factory and becomes functional when they replace > our board with another. I have been told that they are not seeing any such > failures from the field. So it either goes immediately in their system or it > stays "good." And we've NEVER seen any failures like this from any other > customer; this customer receives less than 25% of all the boards of that > type shipped, which makes a design flaw of our board seem less likely. > > Worse, we are running at about 15% strength right now (over 80% of the > people that were with us at this time last year have been laid off). Those > of us left are wearing so many hats that we have very little time to spare - > and no budget for "unnecessary travel." > > From the bench, we might see this; A field return is tested on one PC and > appears to be operating normally. When placed into another PC (which has a > PCI-bus extender card with over current protection & indication) it > immediately fails and one of the over current LEDs activate. a good board > measures several K-Ohms on the 3.3 volt bus, but one of the failing units > measure only a few Ohms to the digital meter. Lifting IC legs one by one > eventually identifies the faulty chip(s), and when replaced everything > returns to normal. In another case it's the +12 that is over current and it > is traced through a +5 volt regulator to one ICs' +5 supply pins. > > The majority of failures are 3.3 volt parts, then 5 volt, and finally those > whose power is derived from +12 volts. > > The onboard 3.3 volt unit is a Linear Technology LT1506CR-3.3 switching > regulator (the dash 3.3 is fixed at 3.3 volts), being employed as per their > own specs/schematic. And NONE of these has failed hard; while it is possible > that this circuit is failing, it always appears to be functioning properly > when we analyze the faulty boards. Additionally, if this was the cause, how {Quote hidden}> would that explain the failures of the parts powered by the PCI bus +5 and > +12 volt supplies? And why only this customer? > > It seems reasonable that the power supply that this customer is using is > occasionally doing something that our board finds disagreeable, but what? > And why only when first assembled and never in the field? It would be > extremely helpful if we had some idea what to ask this customer to > investigate... > > I'm hoping some of you PIC-listers might present some opinions that help > solve the problem - or at least help us start asking questions that lead to > a solution. > > Any thoughts? > Thanks in advance. > Bruce. While troubleshooting, you could consider looking at the ground & returns of your board. As you have done for DC areas, consider 4 lead meter resistance verification on the ground connection and board in the affected areas and compare among boards. Hair line short from trace to trace for a batch could be a suspect. Are there any other PCI pcb's in the unit? Best of luck, {slewrate} -- http://www.piclist.com hint: The PICList is archived three different ways. See http://www.piclist.com/#archives for details.