Overclock.net › Forums › Overclock.net Forum › Overclock.net Related News and Information › Downtime Followup - Jan 24th
New Posts  All Forums:Forum Nav:

Downtime Followup - Jan 24th

post #1 of 13
Thread Starter 
Hi everyone,

As you may have noticed, the site was down for roughly 3.5 hours this afternoon. This post is to explain why that was, what the results were, and what we're doing in the future to prevent the situation from happening again.

I'll start with the summary of news:
  • A database table that suffered a failure stored private messages.
  • 252 private messages were irreversibly lost.
  • These 252 messages were sent between the hours of 7:40am - 2:16pm EST today, January 24th. Any message sent during this time is not in the database.

There were several causes for the above failure, most notably a two-tier failure in our backup system. In the name of full disclosure, here is what transpired:

We make backups transactionally - that is, every time an action is taken on the database the results of that action are immediately pushed to local backups. Two programs are continuously monitoring these on-site backups. The programs are designed to run synchronously with one another, one shuffling data to off-site storage in Amazon's network and the other moving it to dedicated backup storage within the same datacenter as the production servers. These processes eventually worked their way out of sync, and as a result, a copy of the local transaction data was being deleted from the production system before it was fully moved onto the on-site and off-site backups. Integrity of backups is routinely checked, but this problem developed between the most recent integrity check and today.

The problem described above was discovered as a result of a failure during a session of maintenance this morning. Typically such a failure would be easy to recover from with the transactional backups, but when it was determined that they were not available the site was immediately taken offline to further diagnose and minimize the amount of new data introduced to the system while we could not say for sure it was being backed up. Restoration work began with the most recent copy of data we had, which was last night's complete dump. Incremental backups are made on a a constant basis, while complete database dumps are taken nightly.

The only effected database table was the one that stores private messages - only private messages suffered loss, all other data has been verified as intact. The only messages effected were sent in the time between last night's complete dump and the discovery of the sync problem this morning - in total, this was 252 private messages.

Next steps from here include more frequent verification of incremental backups, and redevelopment of the programs monitoring the internal data shuffle of transactional backups to prevent sync related issues from causing data loss. This event was a multi-stage failure, any single part of which would have been recoverable, but the combined total of these failures leaves us short data, as mentioned above. We're terribly sorry for that - its our first data loss in over 7 years, and we're laying out plans to make it our last.

If we can answer any specific questions, please do let us know, and again - our most sincere apologies for this. It could have been much worse; but that does not take away from the fact it shouldn't have happened at all.

Thanks,
Chipp
Morpheus Mini
(7 items)
 
  
CPUMotherboardGraphicsRAM
i7-6700T SN970 GTX 960 Crucial CT102464BF160B 
Hard DriveOSAudio
Samsung 850 Evo M.2 Windows 10 Pro MOTU Audio Express 
  hide details  
Reply
Morpheus Mini
(7 items)
 
  
CPUMotherboardGraphicsRAM
i7-6700T SN970 GTX 960 Crucial CT102464BF160B 
Hard DriveOSAudio
Samsung 850 Evo M.2 Windows 10 Pro MOTU Audio Express 
  hide details  
Reply
post #2 of 13
Ouch, that's no good, but it could have been much worse, and I have seen worse, even on relatively large forums. Thanks for the clarification, Chipp thumb.gif
post #3 of 13
So that's why two of my PM's were gone when I got home from school.. I thought to myself "I swear I sent and received a PM this morning".. lol. Anyhow, I am very glad it wasn't worse! Thanks for the update Chipp!
Talon
(14 items)
 
  
CPUMotherboardGraphicsRAM
1 Intel Core I5 2500k 3.3G 6M R BX80623I52500K Asus P8Z68-V LE Z68 LGA1155 R  1 Zotac Geforce GTX 465 ZT-40301-10P 4Gx2 G-Skill F3-12800CL9D-8GBRL  
Optical DriveCoolingOSMonitor
1 LG Dual Layer Dvd / Memorex 52x cdrw 5 System Fans Windows 7 Ultimate x64 / Mint Linux acer G235HAbd 23'' LCD monitor 
KeyboardPowerCaseMouse
Logitech Ultra X XFX 850 Watt Modded Rosewill R223-P-BK Microsoft Laser Mouse 6000-v1.0 
  hide details  
Reply
Talon
(14 items)
 
  
CPUMotherboardGraphicsRAM
1 Intel Core I5 2500k 3.3G 6M R BX80623I52500K Asus P8Z68-V LE Z68 LGA1155 R  1 Zotac Geforce GTX 465 ZT-40301-10P 4Gx2 G-Skill F3-12800CL9D-8GBRL  
Optical DriveCoolingOSMonitor
1 LG Dual Layer Dvd / Memorex 52x cdrw 5 System Fans Windows 7 Ultimate x64 / Mint Linux acer G235HAbd 23'' LCD monitor 
KeyboardPowerCaseMouse
Logitech Ultra X XFX 850 Watt Modded Rosewill R223-P-BK Microsoft Laser Mouse 6000-v1.0 
  hide details  
Reply
post #4 of 13
Bump for visibility.
Gunmetal Tower
(23 items)
 
  
CPUMotherboardGraphicsRAM
Intel 2600K ASUS P8Z68-V GEN3 Asus GTX 580 CORSAIR Vengeance 16GB 1600 
Hard DriveHard DriveOptical DriveCooling
Samsung 830 256GB Western Digital Caviar Black 640GB LG DVD Corsair H80 
OSMonitorMonitorKeyboard
Microsoft Windows 8.1 Professional 64 Bit Asus PB278Q Dell 1907FPc  Ducky 9008-G2 Browns and Reds 
PowerCaseMouseMouse Pad
ABS SL1050 1050W Antec P280 Logitech G700 Mionix Propus 380 
AudioAudioAudioAudio
HT | OMEGA Claro Halo Audio Technica ATH-A900X Audio Technica ATH-AD900 M-Audio AV 40 
OtherOtherOther
Logitech USB Desktop Microphone APC Back-UPS XS 1500 Wacom Intuos4 
  hide details  
Reply
Gunmetal Tower
(23 items)
 
  
CPUMotherboardGraphicsRAM
Intel 2600K ASUS P8Z68-V GEN3 Asus GTX 580 CORSAIR Vengeance 16GB 1600 
Hard DriveHard DriveOptical DriveCooling
Samsung 830 256GB Western Digital Caviar Black 640GB LG DVD Corsair H80 
OSMonitorMonitorKeyboard
Microsoft Windows 8.1 Professional 64 Bit Asus PB278Q Dell 1907FPc  Ducky 9008-G2 Browns and Reds 
PowerCaseMouseMouse Pad
ABS SL1050 1050W Antec P280 Logitech G700 Mionix Propus 380 
AudioAudioAudioAudio
HT | OMEGA Claro Halo Audio Technica ATH-A900X Audio Technica ATH-AD900 M-Audio AV 40 
OtherOtherOther
Logitech USB Desktop Microphone APC Back-UPS XS 1500 Wacom Intuos4 
  hide details  
Reply
post #5 of 13
LOL I totally thought I was going crazy or had dreams about sending PMs. That explains my missing 3 frown.gif
Verktøy Cu
(20 items)
 
XForma MBX MKII
(18 items)
 
 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 V4 Intel Xeon E5-2670 V4 Asus Z10PE-D8 WS Nvidia Titan X 
GraphicsGraphicsGraphicsRAM
Nvidia Titan X Nvidia Titan X Nvidia Titan X 128GB of Kingston DDR4 ECC 2133Mhz 
Hard DriveHard DriveHard DriveHard Drive
Intel 750 NVME 1.2TB U.2 Intel 750 NVME 1.2TB AIC Intel 750 NVME 1.2TB AIC Intel 750 NVME 1.2TB AIC 
CoolingOSMonitorKeyboard
EK Water Blocks Windows 10 Pro Dell UP2715K Cherry MX 6.0 
PowerCaseMouseAudio
Corsair AX1500i Case Labs S8 Logitech MX Master Pioneer SE-L40 
CPUMotherboardGraphicsGraphics
Intel 5960X Asus Rampage V Extreme Nvidia Titan X Nvidia Titan X 
GraphicsRAMHard DriveHard Drive
Nvidia Titan X Corsair Dominator Platinum 4x8GB 2400Mhz (2) Intel 530 180GB (2) HGST 4TB 
Optical DriveCoolingOSMonitor
Blu-Ray Slot Load EK Water Blocks Liquid Cooling Windows 8.1 Dell UP2715K 
KeyboardPowerCaseMouse
Ducky OCN Blues Seasonic 1200W Platinum XForma MBX MKII TBD 
AudioAudio
Asus Xonar Essence One (2) Mackie MR5MK2 
  hide details  
Reply
Verktøy Cu
(20 items)
 
XForma MBX MKII
(18 items)
 
 
CPUCPUMotherboardGraphics
Intel Xeon E5-2670 V4 Intel Xeon E5-2670 V4 Asus Z10PE-D8 WS Nvidia Titan X 
GraphicsGraphicsGraphicsRAM
Nvidia Titan X Nvidia Titan X Nvidia Titan X 128GB of Kingston DDR4 ECC 2133Mhz 
Hard DriveHard DriveHard DriveHard Drive
Intel 750 NVME 1.2TB U.2 Intel 750 NVME 1.2TB AIC Intel 750 NVME 1.2TB AIC Intel 750 NVME 1.2TB AIC 
CoolingOSMonitorKeyboard
EK Water Blocks Windows 10 Pro Dell UP2715K Cherry MX 6.0 
PowerCaseMouseAudio
Corsair AX1500i Case Labs S8 Logitech MX Master Pioneer SE-L40 
CPUMotherboardGraphicsGraphics
Intel 5960X Asus Rampage V Extreme Nvidia Titan X Nvidia Titan X 
GraphicsRAMHard DriveHard Drive
Nvidia Titan X Corsair Dominator Platinum 4x8GB 2400Mhz (2) Intel 530 180GB (2) HGST 4TB 
Optical DriveCoolingOSMonitor
Blu-Ray Slot Load EK Water Blocks Liquid Cooling Windows 8.1 Dell UP2715K 
KeyboardPowerCaseMouse
Ducky OCN Blues Seasonic 1200W Platinum XForma MBX MKII TBD 
AudioAudio
Asus Xonar Essence One (2) Mackie MR5MK2 
  hide details  
Reply
post #6 of 13
Glad it's fixed, we need more random facts though biggrin.gif
CHILZ - Lan Rig
(17 items)
 
CANARY - Main Rig
(16 items)
 
CADILLAC - HTPC
(14 items)
 
CPUMotherboardGraphicsRAM
Intel i5-4570s Asus H97M-Plus AMD R9 280 G.Skill RipjawsX 16 GB (2x8) 
Hard DriveHard DriveHard DriveCooling
250 GB Samsung 840 240 GB Kingston 3 TB USB 3.0 Drive Custom Loop 
OSMonitorKeyboardPower
Windows 10 Preview BenQ GL2450 Filco MajesTouch2 Ninja PC P&C Silencer Mk III 600 W 
CaseMouseMouse PadAudio
Fractal Design Core 1000 Mionix Castor Monoprice XXL JL Amps + Custom Morel Bookshelf speakers 
Other
Scythe Kama-Panel 3 
CPUMotherboardGraphicsRAM
FX-8150 @ 4.6 GHz Fatal1ty 990FX Pro 9800 GTX+ 512 MB G.Skill Ripjaws X 1866 CL9 
Hard DriveHard DriveCoolingOS
120 GB OCZ Vertex 3 1 TB WD Black 5x 120mm + MCP350 + EK Supreme HF + MicroRes Windows 8 Consumer Preview 
MonitorKeyboardPowerCase
2x Dell U2212HM Logitech G110 Cooler Master 850W Silent Pro Cooler Master 690 II Adv. 
MouseMouse PadAudio
Razer Death Adder 3.5G Staples Gel Cushion Asus Xonar DG + Senn. PC333D 
CPUMotherboardGraphicsRAM
Q6600 Acer X1800 ATI 5670 2 GB Kingston 
Hard DriveOptical DriveOSMonitor
2 TB WD Green Asus BD-R Windows 7 Home Premium Sony 50" LCD 
KeyboardPowerCaseMouse
Acer Media 220 W SFF Acer X1800 Acer Optical 
Mouse PadAudio
The TV cabinet Denon 2808 7.1 AVR + Dahlquist 350W 8" Sub + Kl... 
  hide details  
Reply
CHILZ - Lan Rig
(17 items)
 
CANARY - Main Rig
(16 items)
 
CADILLAC - HTPC
(14 items)
 
CPUMotherboardGraphicsRAM
Intel i5-4570s Asus H97M-Plus AMD R9 280 G.Skill RipjawsX 16 GB (2x8) 
Hard DriveHard DriveHard DriveCooling
250 GB Samsung 840 240 GB Kingston 3 TB USB 3.0 Drive Custom Loop 
OSMonitorKeyboardPower
Windows 10 Preview BenQ GL2450 Filco MajesTouch2 Ninja PC P&C Silencer Mk III 600 W 
CaseMouseMouse PadAudio
Fractal Design Core 1000 Mionix Castor Monoprice XXL JL Amps + Custom Morel Bookshelf speakers 
Other
Scythe Kama-Panel 3 
CPUMotherboardGraphicsRAM
FX-8150 @ 4.6 GHz Fatal1ty 990FX Pro 9800 GTX+ 512 MB G.Skill Ripjaws X 1866 CL9 
Hard DriveHard DriveCoolingOS
120 GB OCZ Vertex 3 1 TB WD Black 5x 120mm + MCP350 + EK Supreme HF + MicroRes Windows 8 Consumer Preview 
MonitorKeyboardPowerCase
2x Dell U2212HM Logitech G110 Cooler Master 850W Silent Pro Cooler Master 690 II Adv. 
MouseMouse PadAudio
Razer Death Adder 3.5G Staples Gel Cushion Asus Xonar DG + Senn. PC333D 
CPUMotherboardGraphicsRAM
Q6600 Acer X1800 ATI 5670 2 GB Kingston 
Hard DriveOptical DriveOSMonitor
2 TB WD Green Asus BD-R Windows 7 Home Premium Sony 50" LCD 
KeyboardPowerCaseMouse
Acer Media 220 W SFF Acer X1800 Acer Optical 
Mouse PadAudio
The TV cabinet Denon 2808 7.1 AVR + Dahlquist 350W 8" Sub + Kl... 
  hide details  
Reply
post #7 of 13

I was wondering where the hell all my PM's went from a couple days ago! I know a couple of those PMs were mine.

 
Media Server
(15 items)
 
Various snapshots
(3 photos)
CPUMotherboardGraphicsRAM
Intel Core i7 920 @ 4.2Ghz DFI Lanparty UT X58 T3eh8 XFX Radeon HD 6990 4GB G.Skill Pi Series 
RAMHard DriveHard DriveOptical Drive
Mushkin Enhanced Intel 510 Series 250GB SSD Seagate 1.5TB 7200RPM  Sony External USB DVD Combo Drive 
OSMonitorMonitorMonitor
Windows 7 Enterprise 64-bit ASUS VH238H Dell E207WFPc Samsung PN58B650 
KeyboardPowerCaseMouse
Razer Blackwidow Ultimate Corsair AX 1200 Custom MountainMods Monticle 24 w/Acoustics & LCS Logitech G500 
Mouse PadAudioAudio
Razer Destructor Ultrasone H-Fi 780 Custom built up 5.1 surround sound  
CPUMotherboardGraphicsGraphics
AMD Phenom 9950 Black Edition Gigabyte mATX Galaxy Ge-Force GTX460 1GB Radeon HD3200 
RAMHard DriveHard DriveCooling
6GB DDR2 (2x2GB Patriot+2x1GB G.Skill) Western Digital Caviar Black Western Digital Caviar Green Cooler Master GeminII 
OSMonitorKeyboardPower
Windows 7 Home Server Samsung PN58B650  Logitech Dinovo Mini Silverstone 350W 
CaseMouse
Antec Veralis Logitech Dinovo Mini 
  hide details  
Reply
 
Media Server
(15 items)
 
Various snapshots
(3 photos)
CPUMotherboardGraphicsRAM
Intel Core i7 920 @ 4.2Ghz DFI Lanparty UT X58 T3eh8 XFX Radeon HD 6990 4GB G.Skill Pi Series 
RAMHard DriveHard DriveOptical Drive
Mushkin Enhanced Intel 510 Series 250GB SSD Seagate 1.5TB 7200RPM  Sony External USB DVD Combo Drive 
OSMonitorMonitorMonitor
Windows 7 Enterprise 64-bit ASUS VH238H Dell E207WFPc Samsung PN58B650 
KeyboardPowerCaseMouse
Razer Blackwidow Ultimate Corsair AX 1200 Custom MountainMods Monticle 24 w/Acoustics & LCS Logitech G500 
Mouse PadAudioAudio
Razer Destructor Ultrasone H-Fi 780 Custom built up 5.1 surround sound  
CPUMotherboardGraphicsGraphics
AMD Phenom 9950 Black Edition Gigabyte mATX Galaxy Ge-Force GTX460 1GB Radeon HD3200 
RAMHard DriveHard DriveCooling
6GB DDR2 (2x2GB Patriot+2x1GB G.Skill) Western Digital Caviar Black Western Digital Caviar Green Cooler Master GeminII 
OSMonitorKeyboardPower
Windows 7 Home Server Samsung PN58B650  Logitech Dinovo Mini Silverstone 350W 
CaseMouse
Antec Veralis Logitech Dinovo Mini 
  hide details  
Reply
post #8 of 13
You guys archive in addition to backups, right?
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
post #9 of 13
Thanks for the heads up smile.gif
post #10 of 13

Happens to the best of us. Thankfully I didn't lose any AFAIK. wheee.gif

New Posts  All Forums:Forum Nav:
  Return Home
Overclock.net › Forums › Overclock.net Forum › Overclock.net Related News and Information › Downtime Followup - Jan 24th