Overclock.net › Forums › Industry News › Technology and Science News › [The Guardian] BlackBerry outage for three days caused by faulty router
New Posts  All Forums:Forum Nav:

[The Guardian] BlackBerry outage for three days caused by faulty router

post #1 of 3
Thread Starter 
Quote:
Something broke down in the internal systems. It is still unclear quite what, though it has been blamed on a router (a piece of equipment which directs internet data in or across a network). RIM later called it a "core switch failure".

Even a momentary lapse in the flow of RIM's data could be calamitous: the Guardian understands the Slough NOC servers normally deal with 8 gigabytes of data every second – equivalent to 16 hours of BBC iPlayer TV viewing. In a typical month, the unassuming two-storey building in Slough sees up to 20 petabytes, or 20m gigabytes, pass through it.
Quote:
"RIM grew in popularity too quickly, and got complacent over the iPhone," one former staffer with close knowledge of its network setup told the Guardian.

He said that rather than rewrite its core networking software so that it could deal with the rapid boom in the smartphone market after 2005, RIM tried simply to increase the number of servers running the software, and spread demand among them
Quote:
The reason why the amount of data passing through Slough is so colossal is that it is one of just four NOCs maintained by RIM to route data between all its global users – and it serves almost everyone in Europe, the Middle East, Africa and India. It also handles all of the GPRS "packets" – data sent worldwide via low-speed phone networks – to BlackBerry devices.
Quote:
The real point of failure appears to be a fundamental piece of RIM's own software called the "Relay" which directs traffic within each of the four NOCs.

"I actually think the Relay has reached melting point and, err, melted," the former RIM staffer said.

Source

Long quotes, I know. The article gives fairly in-depth information on what happened; definitely an interesting read. I suspect this issue will be used as a case study in several networking courses. biggrin.gif
Ares v2.0
(8 items)
 
Chaos v2.2
(12 items)
 
Eos v2.0
(8 items)
 
CPUGraphicsRAMHard Drive
Intel Core i7 6700HQ 2.60GHz NVIDIA GTX 960M 16GB DDR4 2133MHz Western Digital 1TB 
OSMonitorKeyboardCase
Windows 10 x64 15.6" FHD IPS matte-black display Built-in backlit ASUS Republic of Gamers 
CPUMotherboardRAMHard Drive
Core i3-3240 ASRock B75 Pro3-M Kingston DDR3 16GB 1333MHz 1.5v 4x WD Red NASware 3TB 
Hard DriveHard DriveCoolingOS
4x Samsung F4 2TB HD204UI WD Caviar Blue 500GB Arctic Freezer 13 Ubuntu 14.04 LTS x64 
PowerCaseOtherOther
Corsair CX430 v2 X-Case RM424 - 24 bay mini-SAS LSI SAS-9207-4i4e 6Gbps PCI-E 3.0 HBA Chenbro CK22804 28 port SAS Expander 
CPUGraphicsRAMHard Drive
Intel Core i3 6100U 2.3 GHz Intel HD Graphics 520 8 GB Kingston DDR3 1600 MHz Crucial MX200 250GB M2.SSD 
Hard DriveOSMonitorKeyboard
Kingston SSDNow V300 120GB SSD Windows 10 Pro x64 13.3" Full HD IPS 1920 x 1080 Backlit 
  hide details  
Reply
Ares v2.0
(8 items)
 
Chaos v2.2
(12 items)
 
Eos v2.0
(8 items)
 
CPUGraphicsRAMHard Drive
Intel Core i7 6700HQ 2.60GHz NVIDIA GTX 960M 16GB DDR4 2133MHz Western Digital 1TB 
OSMonitorKeyboardCase
Windows 10 x64 15.6" FHD IPS matte-black display Built-in backlit ASUS Republic of Gamers 
CPUMotherboardRAMHard Drive
Core i3-3240 ASRock B75 Pro3-M Kingston DDR3 16GB 1333MHz 1.5v 4x WD Red NASware 3TB 
Hard DriveHard DriveCoolingOS
4x Samsung F4 2TB HD204UI WD Caviar Blue 500GB Arctic Freezer 13 Ubuntu 14.04 LTS x64 
PowerCaseOtherOther
Corsair CX430 v2 X-Case RM424 - 24 bay mini-SAS LSI SAS-9207-4i4e 6Gbps PCI-E 3.0 HBA Chenbro CK22804 28 port SAS Expander 
CPUGraphicsRAMHard Drive
Intel Core i3 6100U 2.3 GHz Intel HD Graphics 520 8 GB Kingston DDR3 1600 MHz Crucial MX200 250GB M2.SSD 
Hard DriveOSMonitorKeyboard
Kingston SSDNow V300 120GB SSD Windows 10 Pro x64 13.3" Full HD IPS 1920 x 1080 Backlit 
  hide details  
Reply
post #2 of 3
I fail to see how an EU RIM Router problem caused the exact same outage type in the US unless everything traverses some singular infrastructure..../tinfoil N.S.A.
post #3 of 3
Quote:
Originally Posted by Ghoxt View Post
I fail to see how an EU RIM Router problem caused the exact same outage type in the US unless everything traverses some singular infrastructure..../tinfoil N.S.A.
Everything has a central office which directs traffic. If that central office stops directing traffic, it can wreak havoc across multiple networks, even intercontinental....

I have ties with AT&T and when there was a fiber (line) cut in the Atlantic Ocean, somehow Georgia lost AT&T IP Services. You'd be surprised how far fiber networks can span.
 
Media Server
(15 items)
 
Various snapshots
(3 photos)
CPUMotherboardGraphicsRAM
Intel Core i7 920 @ 4.2Ghz DFI Lanparty UT X58 T3eh8 XFX Radeon HD 6990 4GB G.Skill Pi Series 
RAMHard DriveHard DriveOptical Drive
Mushkin Enhanced Intel 510 Series 250GB SSD Seagate 1.5TB 7200RPM  Sony External USB DVD Combo Drive 
OSMonitorMonitorMonitor
Windows 7 Enterprise 64-bit ASUS VH238H Dell E207WFPc Samsung PN58B650 
KeyboardPowerCaseMouse
Razer Blackwidow Ultimate Corsair AX 1200 Custom MountainMods Monticle 24 w/Acoustics & LCS Logitech G500 
Mouse PadAudioAudio
Razer Destructor Ultrasone H-Fi 780 Custom built up 5.1 surround sound  
CPUMotherboardGraphicsGraphics
AMD Phenom 9950 Black Edition Gigabyte mATX Galaxy Ge-Force GTX460 1GB Radeon HD3200 
RAMHard DriveHard DriveCooling
6GB DDR2 (2x2GB Patriot+2x1GB G.Skill) Western Digital Caviar Black Western Digital Caviar Green Cooler Master GeminII 
OSMonitorKeyboardPower
Windows 7 Home Server Samsung PN58B650  Logitech Dinovo Mini Silverstone 350W 
CaseMouse
Antec Veralis Logitech Dinovo Mini 
  hide details  
Reply
 
Media Server
(15 items)
 
Various snapshots
(3 photos)
CPUMotherboardGraphicsRAM
Intel Core i7 920 @ 4.2Ghz DFI Lanparty UT X58 T3eh8 XFX Radeon HD 6990 4GB G.Skill Pi Series 
RAMHard DriveHard DriveOptical Drive
Mushkin Enhanced Intel 510 Series 250GB SSD Seagate 1.5TB 7200RPM  Sony External USB DVD Combo Drive 
OSMonitorMonitorMonitor
Windows 7 Enterprise 64-bit ASUS VH238H Dell E207WFPc Samsung PN58B650 
KeyboardPowerCaseMouse
Razer Blackwidow Ultimate Corsair AX 1200 Custom MountainMods Monticle 24 w/Acoustics & LCS Logitech G500 
Mouse PadAudioAudio
Razer Destructor Ultrasone H-Fi 780 Custom built up 5.1 surround sound  
CPUMotherboardGraphicsGraphics
AMD Phenom 9950 Black Edition Gigabyte mATX Galaxy Ge-Force GTX460 1GB Radeon HD3200 
RAMHard DriveHard DriveCooling
6GB DDR2 (2x2GB Patriot+2x1GB G.Skill) Western Digital Caviar Black Western Digital Caviar Green Cooler Master GeminII 
OSMonitorKeyboardPower
Windows 7 Home Server Samsung PN58B650  Logitech Dinovo Mini Silverstone 350W 
CaseMouse
Antec Veralis Logitech Dinovo Mini 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Technology and Science News
Overclock.net › Forums › Industry News › Technology and Science News › [The Guardian] BlackBerry outage for three days caused by faulty router