[AnandTech] Toshiba & WD NAND Production Hit By Power Outage: 6 Exabytes Lost - Page 3 - Overclock.net - An Overclocking Community

Forum Jump: 

[AnandTech] Toshiba & WD NAND Production Hit By Power Outage: 6 Exabytes Lost

Reply
 
Thread Tools
post #21 of 70 (permalink) Old 07-03-2019, 05:59 PM
New to Overclock.net
 
battlenut's Avatar
 
Join Date: Feb 2007
Location: Yokosuka, Japan
Posts: 1,523
Rep: 58 (Unique: 52)
Power outages from my experience do not happen very often here in japan. I have lived here for 26 years and I do not remember the Japanese loosing power. The US base looses power on a quarterly basis. Not saying that outages do not happen in japan, just saying I personally never seen one.

CPU: AMD Ryzen Threadripper 1950X Processor
Mobo: Asrock X399 Taichi
VGA: Asrock Phantom Gaming Radeon VII
RAM: G-Skill 3200 RipJaws F4-3200C16-16GVK
Storage: Samsung M.2 850 EVO 1TB
PSU: Seasonic Prime Ultra 1000 Platinum
Cooling: HeatKiller For CPU & Bykski for GPU


battlenut is offline  
Sponsored Links
Advertisement
 
post #22 of 70 (permalink) Old 07-03-2019, 06:33 PM
New to Overclock.net
 
Imouto's Avatar
 
Join Date: Mar 2012
Posts: 1,872
Rep: 213 (Unique: 98)
Quote: Originally Posted by Asmodian View Post
Have you worked anywhere that pulled 60MW? That is pretty high for any generator system. That is not a generator, that is a power plant.
In my experience, most the places with over 5 MW have been transitioning from only grid to grid + backup to self generating (and selling the excess production in some cases) + critical grid backup.

That said, the cost of electricity in my country is bonkers, so no wonder most of them went that route. In case they need an insane amount of power they usually discard backup power except for critical tasks.

Going back to the news. Here utilities are enforced to have some sort of compensation for outages at factories and such so taking in mind the MTBF and the compensation along with the expected price hikes I'm quite sure this wasn't such an enormous loss. So yeah, maybe a backup system wasn't the smartest option in this case.

#EnthusiastLivesMatter
Imouto is offline  
post #23 of 70 (permalink) Old 07-03-2019, 08:50 PM
Super Lurker
 
voidwarranty's Avatar
 
Join Date: Mar 2011
Location: Vancouver, WA
Posts: 112
Rep: 12 (Unique: 12)
Quote: Originally Posted by SystemTech View Post
Ok im sorry for the ignorance, but :

1. how can a 13 minute outage effect a month and a half's supply? Why is production partially halted? Sure they would lose the wafers in production at that point in time, but would you not just reset everything, start a new wafer and all is back to normal?

2. If the fab's are that sensitive to a power outage, would you not have redundant power supplies. Generators/Batteries etc?
Wow some of these comments in this thread are interesting. Let me answer this as someone who has worked in a fab, and has experienced a power blip. Quoted, because legitimately good questions.

1. How can a 13 minute outage effect a month and a half's supply? Why is production partially halted? Sure they would lose the wafers in production at that point in time, but would you not just reset everything, start a new wafer and all is back to normal?

Answer: With any large fab there are several areas of production that are running at any time. When I worked in a fab I worked in the etching area. Pretty easy to break down. We would take wafers after they came from photo. Wafers would run through several tools, etching through areas of the wafer that were to be etched through, while preserving the covered/protected areas that photo had put down. These wafers would spend anywhere from 5-15 minutes going through different acid/rinse baths until they were complete, and would then come out, and be ready to move to the next area.

When we had a power blip, and I mean blip, it was literally a quick lights off, and right back on, we lost more than 70% of our tools to restarts. That was only in our department, I'm not even sure how many other tools went down that day across the fab. That triggered the "Oh ****" protocol, which was basically, everything needs to be manually extracted asap.

Once things were manually extracted all of our tools had to be re-calibrated (Our maintenance guys were not happy that day) , and all wafers had to go through our process engineers to see if they were salvageable. All said and done we had to scrap 12-15 percent of our production that night, which were various lots, at various stages of production. Massive impact to our customers, but hell we had overtime for the next 2 months.

2. If the fab's are that sensitive to a power outage, would you not have redundant power supplies. Generators/Batteries etc?

Answer: Yes redundancies are put in place, but several things could have occurred. They hadn't properly run routine testing of their backup power systems, which jumps out to me as a root cause. Power outage somehow overloaded redundancies, doubtful but who knows.

Either way, a power outage that lasted that long would definitely impact a massive chunk of production. The restart and re-calibration time alone would really kill them, since they'll have to run test batches through every tool in that fab before even clearing for production lots.

voidwarranty is offline  
Sponsored Links
Advertisement
 
post #24 of 70 (permalink) Old 07-04-2019, 12:05 PM
⎬⎛⎝⎠⎞⎨
 
TK421's Avatar
 
Join Date: May 2011
Posts: 5,228
Rep: 157 (Unique: 122)
Quote: Originally Posted by Nick the Slick View Post
Very true. Anyone eyeing one of those $100 1TB Phison + Toshiba NVMe drives might want to pull the trigger though. Not sure how long that will last.
which drive is this?

nͫٴiͤٴcͫٴeͤ੮Һ૯ ცɿ૭ ૭คעٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴ ٴٴٴ
Desktop for gaming
(17 items)
ThinkPad T480
(9 items)
ThinkPad T430 XM
(9 items)
CPU
i7 5820K
Motherboard
X99 Deluxe
GPU
RTX 2080 Ti XC Ultra
RAM
Crucial 4x4GB
Hard Drive
Samsung 950 Pro 512GB
Hard Drive
Samsung 850 EVO 1TB
Hard Drive
WD Black 4TB
Hard Drive
WD Red 10TB
Optical Drive
Dell CH03N BD
Power Supply
EVGA 750 G2
Cooling
Swiftech Drive H360X3
Case
Phanteks Enthoo Luxe
Operating System
Windows 10 Enterprise LTSC 1809
Monitor
BenQ XL2430T
Monitor
ASUS PG279Q
Keyboard
Bloody B850
Mouse
Logitech G102 Prodigy (Mercury)
CPU
i7 8550U
Motherboard
Lenovo Windu-2 20L5
GPU
Intel UHD 620
RAM
F4-3000C16D-16GRS | 2x8GB 2400MHz
Hard Drive
Intel NVME SSD 256GB
Power Supply
Lenovo 65w USB-C
Cooling
Delta 01YR200
Operating System
Windows 10 Enterprise LTSC 1809
Monitor
LP140WF6-SPB7 FHD IPS
CPU
i7 3920XM
Motherboard
Nozomi 4 Type 2344 BZU
GPU
Intel HD4000
RAM
F3-2133C11-8GSRL | 2x8GB 2133MHz
Hard Drive
Samsung 850 Pro 256GB
Power Supply
Lenovo 90W
Operating System
Windows 10 Enterprise LTSC 1809
Monitor
AUO B140HAN01.3
Keyboard
Classic 7-row NMB 45N2211
▲ hide details ▲


TK421 is offline  
post #25 of 70 (permalink) Old 07-04-2019, 12:23 PM - Thread Starter
Jedi Knight
 
Nick the Slick's Avatar
 
Join Date: Mar 2015
Location: Kentucky, USA
Posts: 598
Rep: 41 (Unique: 41)
Quote: Originally Posted by TK421 View Post
which drive is this?

Inland Premium
Sabrent Rocket
Silicon Power


I think there's a few others but can't find them, and there's a couple others that use this combo but are $125+. I picked up the Inland Premium myself and am very happy with it.

Vibranium
(16 items)
CPU
Intel Core i7 7700k
Motherboard
ASUS Z170-PRO
GPU
MSI GeForce GTX 1070 Sea Hawk EK X
RAM
G. Skill Trident Z
Hard Drive
MyDigitalSSD BPX
Hard Drive
Crucial MX300
Hard Drive
ADATA SU800
Power Supply
Seasonic SS-750KM3
Cooling
Custom Liquid Cooling
Case
Phanteks “Enthoo Pro Series"
Operating System
Windows 10
Monitor
VIOTEK GN32DB
Keyboard
Logitech K800
Mouse
Logitech G603
Audio
JBL LIVE 650BTNC
Audio
Fiio E10K DAC/AMP
▲ hide details ▲


Nick the Slick is online now  
post #26 of 70 (permalink) Old 07-04-2019, 12:30 PM
⎬⎛⎝⎠⎞⎨
 
TK421's Avatar
 
Join Date: May 2011
Posts: 5,228
Rep: 157 (Unique: 122)
Quote: Originally Posted by Nick the Slick View Post
Inland Premium
Sabrent Rocket
Silicon Power


I think there's a few others but can't find them, and there's a couple others that use this combo but are $125+. I picked up the Inland Premium myself and am very happy with it.

Are there any drives made/branded by major brands?

Not too keen on the warranty on those drives.

nͫٴiͤٴcͫٴeͤ੮Һ૯ ცɿ૭ ૭คעٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴ ٴٴٴ
Desktop for gaming
(17 items)
ThinkPad T480
(9 items)
ThinkPad T430 XM
(9 items)
CPU
i7 5820K
Motherboard
X99 Deluxe
GPU
RTX 2080 Ti XC Ultra
RAM
Crucial 4x4GB
Hard Drive
Samsung 950 Pro 512GB
Hard Drive
Samsung 850 EVO 1TB
Hard Drive
WD Black 4TB
Hard Drive
WD Red 10TB
Optical Drive
Dell CH03N BD
Power Supply
EVGA 750 G2
Cooling
Swiftech Drive H360X3
Case
Phanteks Enthoo Luxe
Operating System
Windows 10 Enterprise LTSC 1809
Monitor
BenQ XL2430T
Monitor
ASUS PG279Q
Keyboard
Bloody B850
Mouse
Logitech G102 Prodigy (Mercury)
CPU
i7 8550U
Motherboard
Lenovo Windu-2 20L5
GPU
Intel UHD 620
RAM
F4-3000C16D-16GRS | 2x8GB 2400MHz
Hard Drive
Intel NVME SSD 256GB
Power Supply
Lenovo 65w USB-C
Cooling
Delta 01YR200
Operating System
Windows 10 Enterprise LTSC 1809
Monitor
LP140WF6-SPB7 FHD IPS
CPU
i7 3920XM
Motherboard
Nozomi 4 Type 2344 BZU
GPU
Intel HD4000
RAM
F3-2133C11-8GSRL | 2x8GB 2133MHz
Hard Drive
Samsung 850 Pro 256GB
Power Supply
Lenovo 90W
Operating System
Windows 10 Enterprise LTSC 1809
Monitor
AUO B140HAN01.3
Keyboard
Classic 7-row NMB 45N2211
▲ hide details ▲


TK421 is offline  
post #27 of 70 (permalink) Old 07-04-2019, 12:47 PM - Thread Starter
Jedi Knight
 
Nick the Slick's Avatar
 
Join Date: Mar 2015
Location: Kentucky, USA
Posts: 598
Rep: 41 (Unique: 41)
Quote: Originally Posted by TK421 View Post
Are there any drives made/branded by major brands?

Not too keen on the warranty on those drives.

Not sure what you're looking for but the inland premium has a three year warranty according to micro center, I would assume it's the same when bought through Amazon. The MyDigitalSSD one has a 5 year warranty but it's the most expensive of the ones I know of and out of stock. As far as major brands go there's the Corsair one, but it's also more expensive and currently out of stock.

Vibranium
(16 items)
CPU
Intel Core i7 7700k
Motherboard
ASUS Z170-PRO
GPU
MSI GeForce GTX 1070 Sea Hawk EK X
RAM
G. Skill Trident Z
Hard Drive
MyDigitalSSD BPX
Hard Drive
Crucial MX300
Hard Drive
ADATA SU800
Power Supply
Seasonic SS-750KM3
Cooling
Custom Liquid Cooling
Case
Phanteks “Enthoo Pro Series"
Operating System
Windows 10
Monitor
VIOTEK GN32DB
Keyboard
Logitech K800
Mouse
Logitech G603
Audio
JBL LIVE 650BTNC
Audio
Fiio E10K DAC/AMP
▲ hide details ▲


Nick the Slick is online now  
post #28 of 70 (permalink) Old 07-04-2019, 07:40 PM
⎬⎛⎝⎠⎞⎨
 
TK421's Avatar
 
Join Date: May 2011
Posts: 5,228
Rep: 157 (Unique: 122)
Quote: Originally Posted by Nick the Slick View Post
Not sure what you're looking for but the inland premium has a three year warranty according to micro center, I would assume it's the same when bought through Amazon. The MyDigitalSSD one has a 5 year warranty but it's the most expensive of the ones I know of and out of stock. As far as major brands go there's the Corsair one, but it's also more expensive and currently out of stock.
They're all essentially the same drive right?

nͫٴiͤٴcͫٴeͤ੮Һ૯ ცɿ૭ ૭คעٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴٴ ٴٴٴ
Desktop for gaming
(17 items)
ThinkPad T480
(9 items)
ThinkPad T430 XM
(9 items)
CPU
i7 5820K
Motherboard
X99 Deluxe
GPU
RTX 2080 Ti XC Ultra
RAM
Crucial 4x4GB
Hard Drive
Samsung 950 Pro 512GB
Hard Drive
Samsung 850 EVO 1TB
Hard Drive
WD Black 4TB
Hard Drive
WD Red 10TB
Optical Drive
Dell CH03N BD
Power Supply
EVGA 750 G2
Cooling
Swiftech Drive H360X3
Case
Phanteks Enthoo Luxe
Operating System
Windows 10 Enterprise LTSC 1809
Monitor
BenQ XL2430T
Monitor
ASUS PG279Q
Keyboard
Bloody B850
Mouse
Logitech G102 Prodigy (Mercury)
CPU
i7 8550U
Motherboard
Lenovo Windu-2 20L5
GPU
Intel UHD 620
RAM
F4-3000C16D-16GRS | 2x8GB 2400MHz
Hard Drive
Intel NVME SSD 256GB
Power Supply
Lenovo 65w USB-C
Cooling
Delta 01YR200
Operating System
Windows 10 Enterprise LTSC 1809
Monitor
LP140WF6-SPB7 FHD IPS
CPU
i7 3920XM
Motherboard
Nozomi 4 Type 2344 BZU
GPU
Intel HD4000
RAM
F3-2133C11-8GSRL | 2x8GB 2133MHz
Hard Drive
Samsung 850 Pro 256GB
Power Supply
Lenovo 90W
Operating System
Windows 10 Enterprise LTSC 1809
Monitor
AUO B140HAN01.3
Keyboard
Classic 7-row NMB 45N2211
▲ hide details ▲


TK421 is offline  
post #29 of 70 (permalink) Old 07-04-2019, 09:07 PM
Avid Something-or-Other
 
shellashock's Avatar
 
Join Date: Jan 2017
Posts: 58
Rep: 2 (Unique: 2)
Quote: Originally Posted by voidwarranty View Post
Wow some of these comments in this thread are interesting. Let me answer this as someone who has worked in a fab, and has experienced a power blip. Quoted, because legitimately good questions.

1. How can a 13 minute outage effect a month and a half's supply? Why is production partially halted? Sure they would lose the wafers in production at that point in time, but would you not just reset everything, start a new wafer and all is back to normal?

Answer: With any large fab there are several areas of production that are running at any time. When I worked in a fab I worked in the etching area. Pretty easy to break down. We would take wafers after they came from photo. Wafers would run through several tools, etching through areas of the wafer that were to be etched through, while preserving the covered/protected areas that photo had put down. These wafers would spend anywhere from 5-15 minutes going through different acid/rinse baths until they were complete, and would then come out, and be ready to move to the next area.

When we had a power blip, and I mean blip, it was literally a quick lights off, and right back on, we lost more than 70% of our tools to restarts. That was only in our department, I'm not even sure how many other tools went down that day across the fab. That triggered the "Oh ****" protocol, which was basically, everything needs to be manually extracted asap.

Once things were manually extracted all of our tools had to be re-calibrated (Our maintenance guys were not happy that day) , and all wafers had to go through our process engineers to see if they were salvageable. All said and done we had to scrap 12-15 percent of our production that night, which were various lots, at various stages of production. Massive impact to our customers, but hell we had overtime for the next 2 months.

2. If the fab's are that sensitive to a power outage, would you not have redundant power supplies. Generators/Batteries etc?

Answer: Yes redundancies are put in place, but several things could have occurred. They hadn't properly run routine testing of their backup power systems, which jumps out to me as a root cause. Power outage somehow overloaded redundancies, doubtful but who knows.

Either way, a power outage that lasted that long would definitely impact a massive chunk of production. The restart and re-calibration time alone would really kill them, since they'll have to run test batches through every tool in that fab before even clearing for production lots.
Thank you for sharing! Out of curiosity, do factory machines lose their calibrated settings during the power interruption/outage or is this more of a "we can't trust that the machines are working properly after the incident" and manual calibration is required for QA? I would have thought that machines would have internal batteries to remember their calibrated settings so they could more or less pick up where they stopped; although I guess manually moving parts around would be more of a hardware calibration issue than software.
shellashock is offline  
post #30 of 70 (permalink) Old 07-04-2019, 09:08 PM - Thread Starter
Jedi Knight
 
Nick the Slick's Avatar
 
Join Date: Mar 2015
Location: Kentucky, USA
Posts: 598
Rep: 41 (Unique: 41)
Quote: Originally Posted by TK421 View Post
They're all essentially the same drive right?

Aside from firmware version, warranty, and branding, yep. They all use the Phison E12 controller + Toshiba 64 layer 3D TLC NAND. Now one thing I haven't looked close enough to find out is if they all use slc + dram cache. I know the Inland uses slc + has 2 dram chips (one on either side of the disk).



And just to be clear I'm not trying to convince you or anyone else to buy these drives. I just personally found them to be really good value and if the power outage affected Toshiba's supply that badly, the prices on these is most likely going to go up a fair bit.

Vibranium
(16 items)
CPU
Intel Core i7 7700k
Motherboard
ASUS Z170-PRO
GPU
MSI GeForce GTX 1070 Sea Hawk EK X
RAM
G. Skill Trident Z
Hard Drive
MyDigitalSSD BPX
Hard Drive
Crucial MX300
Hard Drive
ADATA SU800
Power Supply
Seasonic SS-750KM3
Cooling
Custom Liquid Cooling
Case
Phanteks “Enthoo Pro Series"
Operating System
Windows 10
Monitor
VIOTEK GN32DB
Keyboard
Logitech K800
Mouse
Logitech G603
Audio
JBL LIVE 650BTNC
Audio
Fiio E10K DAC/AMP
▲ hide details ▲


Nick the Slick is online now  
Reply

Quick Reply
Message:
Options

Register Now

In order to be able to post messages on the Overclock.net - An Overclocking Community forums, you must first register.
Please enter your desired user name, your email address and other required details in the form below.
User Name:
If you do not want to register, fill this field only and the name will be used as user name for your post.
Password
Please enter a password for your user account. Note that passwords are case-sensitive.
Password:
Confirm Password:
Email Address
Please enter a valid email address for yourself.
Email Address:

Log-in



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may post new threads
You may post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off