Toshiba & WD NAND Production Hit By Power Outage: 6 Exabytes Lost

148 Comments

View All Comments

• azazel1024 - Tuesday, July 2, 2019 - link

  Of course there is. You aren't talking gigawatts of power. You are talking at most several dozen megawatts. I've seen the backup power supplies for many industrial operations that draw far more power than a fab. You are talking a backup power solution that likely at most runs in to the low hundred million range for what is several billion dollars of equipment. Even ignoring the cost of the equipment that might be damaged, 6 exabytes of NAND production is in the ballpark of $150-200 million dollars worth of NAND down the drain (plus Toshiba's losses, which if the article is correct in speculation, is likely another $200-400 million in losses).
  
  Example of backup "UPS" on a grand scale.
  
  https://electrek.co/2018/09/24/tesla-powerpack-bat...
  
  $66 million installation cost for 100mw of battery power, 300mwh storage (so 3 HOURS of backup power). Yes, the Tesla project was done well after whatever backup power system the fab complex has would have been installed. The point is, for what is clearly higher output than what the fab would need (again, the power requirements for the entire fab complex are likely to be in the 10-40mw range), it is drop in the bucket compared to the losses they suffered.
  
  I am sure that the fab complex DOES have backup power sufficient to the task. What I am also pretty sure of is it isn't redundant (which it should be) and also very likely wasn't tested sufficiently to catch problems (as that would likely require shutting down the fab temporarily, maybe once a year, to test the system live). Most industries that have/rely on backup power do an annual plugs out test.
  
  Looks like maybe WD/Toshiba didn't adhere to industry standards there.

• whyaname - Friday, July 5, 2019 - link

  Actually there is, you only need a few seconds of UPS service till the back up generators of such factories kick in.
  You usually have several protection levels for this stuff.
  For example it is not uncommon for such large facillities to have a complete discrete power plant or to be connected to more than one power plant/grid at a time in addition to UPS and generators.
  
  Did you never wonder why industrial hardware has like 2-4 power connections, that is not just a redundant PSU, it enables you to have the hardware on just as many discrete UPS and powergrids to mitigate the likelyhood of failure.
  
  I remember seeing the new some time ago that some flash manufacturers want to reduce their production because or prior over production and lots of flash still in stock at warehouses.
  This is 13min failure that could have been prevented, but instead somehow causes 3months of issues.
  This sounds like a way to avoid issues with markt price manipulation, but artificialy reducing the available amount of flash.
  
  This should under no scenario have caused an issue similar to the flood back then in south east asia.
  
  If the power goes out you do not suddenly start pumping polluted air into the clean rooms and making stuff unusuable for months.

• Dug - Monday, July 1, 2019 - link

  Your use of the word dude is a clear indication about your knowledge of power. Aircraft carriers use direct nuclear power for one, so I'm not sure why you included that? And thinking they didn't have a power backup is another indication you have no idea how things work in a fab. Just the fact you have no clue how much power is used or how it's produced is another indication. Or the fact a backup can't produce enough power if you are right in the middle of production.
  
  I'm sure they'll be contacting you soon to revamp power distribution sense you know so much about it.

• Gunbuster - Monday, July 1, 2019 - link

  A high end data-center can run 30kW racks. Why can a data center plan and economically execute full time backup power with that energy density but a fab can not?

• Samus - Tuesday, July 2, 2019 - link

  Gunbuster is correct with his assumption. I worked at Argonne National Laboratory through college in the APS (Advanced Photon Source) building - which is a large synchrotron particle accelerator. It had 32, and recently expanded to 36 sectors. It's actually larger than Fermi lab. But nowhere near as large as the large haldron collider.
  
  But collisions it does. The Argonne campus has its own power plant. So do fabs. As anybody who knows anything about power generations knows, power plants produce at capacity in real time, and this capacity is always beyond grid demand to account for spikes or sudden demand surges where power cannot be diverted elsewhere from the grid. This makes them inherently inefficient because the excess power can't be stored, so it is wasted in the form of simply not generating power that could be generated using the supplied pressure\heat\whatever type of fuel is being used. This is especially true with nuclear power production (although no sites like these use nuclear power production on-site, that is overkill) as it takes weeks, and even months with old designs, to roll back fuel processing and scale down power production. During the cooldown, they are super inefficient in the sense they COULD be making a lot more power for the same amount of fuel spent.
  
  ANYWAY. This site clearly has a oil fuel power plant on site. I can spot it on the low-res campus photo in the top right. This plant was built to supply more than enough power to the site EVEN IF grid power seized. But built for, and producing, are two different things.
  
  Something happened at the power processing facility, or something malfunctioned causing a blackout (which seems to be the case considering the length of time there was a power outage...they could divert grid power almost immediately like Samsung did during their recent production woes that lasted in the milliseconds.)
  
  That said, this is hard to wrap my head around. These mistakes just don't happen. But mistakes can be manufactured. And mistakes that take down an entire campus of electricity for minutes almost has to be manufactured...even natural gas\diesel generators warm up and produce power in 20 seconds...

• close - Tuesday, July 2, 2019 - link

  @Gunbuster - A high end data-center can run 30kW racks
  
  Problem is in this case we're talking 2 orders of magnitude higher. In the tens of MW range, maybe even close to 100MW (GloFo Fab 8 needs 80MW *on average* as a reference). And supplying that from backup constantly until the power is restored in not something most industries ever have to deal with so people should stop using their UPS as a reference for anything. There's no real way to save any power during such incidents, there's no "power saving mode", no shutting down part of the line. The backup power has to provide exactly what the main did. Usually you manage to supply enough power to shut down the lines with no damage to the equipment but all the silicon in the pipeline just became trash anyway.
  
  This happened to Samsung, it happened to Toshiba/WD. It's just AT commenters that seem to have the wrong career. AT became home to all of the "I'm an expert dude" type people.

• Gunbuster - Tuesday, July 2, 2019 - link

  And most data centers don't cost a billion to stand up. You scale up to provide services. Plenty of 10MW gensets to choose from. Sure they are the size of a semi trailer but that's a huge building. Line up 15 of them.

• whyaname - Friday, July 5, 2019 - link

  @close
  You realize that is supposed to mean 30kW per rack?
  
  scale that
  
  Aside from that somebody very well posted already that backup power solutions for this scale exist.

• Round - Tuesday, July 9, 2019 - link

  Not related to the article, but regarding " how can a whole country be stupid enough to fall for something like a bolshevik revolution".
  
  We have almost half the US population teetering on the edge of a similar mental catastrophe right now. Intelligence is not distributed equally...

• lazarpandar - Friday, June 28, 2019 - link

  ..What? Darwin awards are awarded for deaths.