Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Linux, Unix › Need some assistance debugging these kernel panics
New Posts  All Forums:Forum Nav:

Need some assistance debugging these kernel panics

post #1 of 5
Thread Starter 
http://svn.dd-wrt.com/attachment/ticket/2580/dd_wrt-kernel-fault-opt.txt
http://svn.dd-wrt.com/attachment/ticket/2580/dd-wrt-kernel-fault-writing-small-file.txt

If anyone could assist me in debugging these two files for the kernel panics (located at the very bottom), I would greatly appreciate it.

Just let me know how I could either find where the stack traces are pointing to, etc.
Gaming
(23 items)
 
  
CPUMotherboardGraphicsGraphics
AMD Phenom II X6 1090T Crosshair IV Formula GTX 560 GTX 580 
RAMRAMRAMRAM
G. Skill F3-12800CL6D-4GBPI G. Skill F3-12800CL6D-4GBPI G. Skill F3-12800CL6D-4GBPI  G. Skill F3-12800CL6D-4GBPI 
Hard DriveHard DriveHard DriveOptical Drive
Seagate Hard Drive Seagate Hard Drive Crucial M4 SSD Sony Optiarc 
CoolingOSMonitorMonitor
Corsair H70 Windows 7 Professional x64 ASUS VH242H 23" Monitor ASUS VH242H 23" Monitor 
MonitorKeyboardPowerCase
Samsung SyncMaster 906BW 19" Monitor Logitech G15 Corsair 1K PSU Lian-Li 70A 
MouseMouse PadAudio
Logitech Performance MX Razer Vespula HT Omega Pro+ 
  hide details  
Reply
Gaming
(23 items)
 
  
CPUMotherboardGraphicsGraphics
AMD Phenom II X6 1090T Crosshair IV Formula GTX 560 GTX 580 
RAMRAMRAMRAM
G. Skill F3-12800CL6D-4GBPI G. Skill F3-12800CL6D-4GBPI G. Skill F3-12800CL6D-4GBPI  G. Skill F3-12800CL6D-4GBPI 
Hard DriveHard DriveHard DriveOptical Drive
Seagate Hard Drive Seagate Hard Drive Crucial M4 SSD Sony Optiarc 
CoolingOSMonitorMonitor
Corsair H70 Windows 7 Professional x64 ASUS VH242H 23" Monitor ASUS VH242H 23" Monitor 
MonitorKeyboardPowerCase
Samsung SyncMaster 906BW 19" Monitor Logitech G15 Corsair 1K PSU Lian-Li 70A 
MouseMouse PadAudio
Logitech Performance MX Razer Vespula HT Omega Pro+ 
  hide details  
Reply
post #2 of 5
DD-wrt is a little exotic of a Linux distro, andI assume, given the log output you aren't running the x86 build, and it looks like the kernel driver responsible for interfacing with your /opt/ partition for optware is probably throwing the exception (since the first one was caused by installing and then attempting to use something from /opt/ and the second was caused by explicitly creating and then reading a file from /opt/. The two types of errors reported are different, but may share the same root cause. The first one is a kernel paging error, and you can read up more on that (including some useful troubleshooting information) here:
http://www.dedoimedo.com/computers/crash-analyze.html#mozTocId782257

I would do some light reading on the OpenWRT forums and CentOS forums on similar issues from the first stacktrace (as it is more likely to lead you to the root cause) There are a load of issues involving the EHCI/OHCI device drivers for USB devices on ARM based chips (mostly broadcom) You may try switching between the EHCI and OHCI kernel modules to see which one is more stable, but that module is the likely culprit from just a little bit of light reading. How up to date is your DD-WRT build? Are you using the one from the wiki for your router or the latest one from the router database? Also, given how stale development on DD-WRT has been and how out dated its device and network modules as well as kernel has become I have a hard time recommending its use - I would reccomend switching to Open-WRT at this point in time.
    
CPUMotherboardGraphicsRAM
Core i7 920 D0 4.2ghz HT (1.3625v) Asus R3E 2xGTX 460 (non SLi, no overclock) 6x2gb G.skill @ 6-8-6-24-1T 
Hard DriveOptical DriveOSMonitor
WD-VR 300GBx1, 2xWD 1tb,2x60gb Agility Some crappy combo burner... Arch x64 3xDell U2410f rev A02 
KeyboardPowerCaseMouse
X-Armor U9BL TT Toughpower 1200w (NTB more efficient) Mountain Mods Pinnacle 24 CYO Roccat Kone (R.I.P. A4Tech x7) 
Mouse Pad
Steelpad Experience I-1 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
Core i7 920 D0 4.2ghz HT (1.3625v) Asus R3E 2xGTX 460 (non SLi, no overclock) 6x2gb G.skill @ 6-8-6-24-1T 
Hard DriveOptical DriveOSMonitor
WD-VR 300GBx1, 2xWD 1tb,2x60gb Agility Some crappy combo burner... Arch x64 3xDell U2410f rev A02 
KeyboardPowerCaseMouse
X-Armor U9BL TT Toughpower 1200w (NTB more efficient) Mountain Mods Pinnacle 24 CYO Roccat Kone (R.I.P. A4Tech x7) 
Mouse Pad
Steelpad Experience I-1 
  hide details  
Reply
post #3 of 5
well what you posted is a kernel OOPS which is a bit different than a panic. A panic forces the machine to halt, while as an oops is just logged and the process down the stack is just killed off. Oops!
Code:
770  Stack : 80074b0c 805db620 878da308 00000000 87d71170 fe14a000 00000001 80014a18
771             878da284 00000000 80356e64 87d070f8 87d71160 0000c000 878da2e4 87d8be54
772             0000c000 87d0937c 80025924 00100100 00200200 00000000 878da284 0000c000
773             c0013100 87d8be54 0000c000 0000001f 878da284 00000000 00000000 87d917fc
774             87aa3e94 8059b108 c0013100 00000000 87d71160 00000009 0000c000 00000010
775             ...
776     Call Trace:[<80074b0c>][<80014a18>][<80356e64>][<87d070f8>][<87d0937c>][<80025924>][<87d917fc>][<800546b4>][<87d91bb0>][<80024408>][<87d92248>][<8035602c>][<800258d0>][<87d64628>][<87d93064>][<8035622c>][<800270f4>][<87d92e58>][<87d92e58>][<80043708>][<800436e8>][<8000bf38>][<8000bf28>]
777     
778     Code: 00041823  00432824  02231024 <bc550000> 1445fffe  00441021  8fbf001c  8fb20018  8fb10014

that is going to drive you nuts. You need to rebuild your kernel with debugging symbols compiled in, so you'll see the object and function names right in the trace. Much easier than looking each one up.

i'm about to get my hands dirty debugging a panic for myself .. luckily mine is happening in a VM so it should be much less of a hassle. I will probably have some more info and tips for you in a few days.

But to start with, find out exactly what you don't have enabled with regards to debugging symbols, rebuild with that, and post the trace with the names in it. You're lucky it's just an oops as the trace is right there on tty1 ... with a panic most of the time you have to use a serial console (again I am lucky mine was in a VM).

Also before you waste a ton of time falling down a rabbit hole that you may not even have to, try it with the latest version.. it may be a known bug and well fixed by now. If there's no suitable version in your distros (un)stable repositories, just try the vanilla 3.4 one from kernel.org with the same config.
stable again
(25 items)
 
  
CPUCPUMotherboardGraphics
E5-2687W E5-2687W ASUS Z9PED8-WS EVGA GTX 570 (Linux host) 
GraphicsRAMHard DriveHard Drive
EVGA GTX 970 FTW (win7 guest) 64GB G.SKILL 2133 2x Crucial M4 256GB raid1 4x 3TB raid 10 
CoolingCoolingCoolingCooling
2x Apogee HD  2x RX 480 2x MCP 655 RP-452x2 rev2 (new) 
CoolingCoolingOSOS
16x Cougar Turbine CFT12SB4 (new) EK FC 580 Gentoo (host) Gentoo (x23 guests) 
OSMonitorMonitorPower
windows 7 (guest w/ vfio-pci) Viewsonic 23" 1080P Viewsonic 19" Antec HCP Platinum 1000 (new) 
CaseOtherOther
Case Labs TH10 (still the best ever) 2x Lamptron FC-5 IOGEAR 2 way DVI KVM Switch 
  hide details  
Reply
stable again
(25 items)
 
  
CPUCPUMotherboardGraphics
E5-2687W E5-2687W ASUS Z9PED8-WS EVGA GTX 570 (Linux host) 
GraphicsRAMHard DriveHard Drive
EVGA GTX 970 FTW (win7 guest) 64GB G.SKILL 2133 2x Crucial M4 256GB raid1 4x 3TB raid 10 
CoolingCoolingCoolingCooling
2x Apogee HD  2x RX 480 2x MCP 655 RP-452x2 rev2 (new) 
CoolingCoolingOSOS
16x Cougar Turbine CFT12SB4 (new) EK FC 580 Gentoo (host) Gentoo (x23 guests) 
OSMonitorMonitorPower
windows 7 (guest w/ vfio-pci) Viewsonic 23" 1080P Viewsonic 19" Antec HCP Platinum 1000 (new) 
CaseOtherOther
Case Labs TH10 (still the best ever) 2x Lamptron FC-5 IOGEAR 2 way DVI KVM Switch 
  hide details  
Reply
post #4 of 5
Thread Starter 
Both half the router and cause the router to restart.

Unfortunately, I can't rebuild the kernel with debugging systems, due to this being DD-WRT (attempted to build it from source, wasn't able to), same goes for me upgrading to a newer kernel. I was also informed that it is impossible to upgrade the kernel because broadcom drivers will fail to work on 3.x (which doesn't seem to be possible).
Gaming
(23 items)
 
  
CPUMotherboardGraphicsGraphics
AMD Phenom II X6 1090T Crosshair IV Formula GTX 560 GTX 580 
RAMRAMRAMRAM
G. Skill F3-12800CL6D-4GBPI G. Skill F3-12800CL6D-4GBPI G. Skill F3-12800CL6D-4GBPI  G. Skill F3-12800CL6D-4GBPI 
Hard DriveHard DriveHard DriveOptical Drive
Seagate Hard Drive Seagate Hard Drive Crucial M4 SSD Sony Optiarc 
CoolingOSMonitorMonitor
Corsair H70 Windows 7 Professional x64 ASUS VH242H 23" Monitor ASUS VH242H 23" Monitor 
MonitorKeyboardPowerCase
Samsung SyncMaster 906BW 19" Monitor Logitech G15 Corsair 1K PSU Lian-Li 70A 
MouseMouse PadAudio
Logitech Performance MX Razer Vespula HT Omega Pro+ 
  hide details  
Reply
Gaming
(23 items)
 
  
CPUMotherboardGraphicsGraphics
AMD Phenom II X6 1090T Crosshair IV Formula GTX 560 GTX 580 
RAMRAMRAMRAM
G. Skill F3-12800CL6D-4GBPI G. Skill F3-12800CL6D-4GBPI G. Skill F3-12800CL6D-4GBPI  G. Skill F3-12800CL6D-4GBPI 
Hard DriveHard DriveHard DriveOptical Drive
Seagate Hard Drive Seagate Hard Drive Crucial M4 SSD Sony Optiarc 
CoolingOSMonitorMonitor
Corsair H70 Windows 7 Professional x64 ASUS VH242H 23" Monitor ASUS VH242H 23" Monitor 
MonitorKeyboardPowerCase
Samsung SyncMaster 906BW 19" Monitor Logitech G15 Corsair 1K PSU Lian-Li 70A 
MouseMouse PadAudio
Logitech Performance MX Razer Vespula HT Omega Pro+ 
  hide details  
Reply
post #5 of 5
Ah .. so you purchased a black box from someone and software wise it is a defective product. I guess you have pretty much no other recourse but to take it up with them. smile.gif Although it sounds like they can't correct the problem either due to broadcom being stuck on 2.6. rolleyes.gif

Debugging symbols or not, if the kernel is oopsing it's not going to stop unless A) you find what the process is, what it is actually doing before the oops, and somehow don't do that or workaround it to prevent it from happening. Or, B) patch the kernel with the required fix and rebuild it.

For option A, you see
Code:
Process usb-storage (pid: 1639, threadinfo=87aa2000, task=805db620)

And then ... if you could see the stack we'd have something to go on. Very strange they excluded debugging symbols .. redface.gif If you search lkml.org you may see an existing bug report that has probably been merged in.

There may be a way to look up the symbol names, I am not sure. If so, have fun.

Can you rip the data off of this thing and spin it up in a semi-functional VM to poke at it?
stable again
(25 items)
 
  
CPUCPUMotherboardGraphics
E5-2687W E5-2687W ASUS Z9PED8-WS EVGA GTX 570 (Linux host) 
GraphicsRAMHard DriveHard Drive
EVGA GTX 970 FTW (win7 guest) 64GB G.SKILL 2133 2x Crucial M4 256GB raid1 4x 3TB raid 10 
CoolingCoolingCoolingCooling
2x Apogee HD  2x RX 480 2x MCP 655 RP-452x2 rev2 (new) 
CoolingCoolingOSOS
16x Cougar Turbine CFT12SB4 (new) EK FC 580 Gentoo (host) Gentoo (x23 guests) 
OSMonitorMonitorPower
windows 7 (guest w/ vfio-pci) Viewsonic 23" 1080P Viewsonic 19" Antec HCP Platinum 1000 (new) 
CaseOtherOther
Case Labs TH10 (still the best ever) 2x Lamptron FC-5 IOGEAR 2 way DVI KVM Switch 
  hide details  
Reply
stable again
(25 items)
 
  
CPUCPUMotherboardGraphics
E5-2687W E5-2687W ASUS Z9PED8-WS EVGA GTX 570 (Linux host) 
GraphicsRAMHard DriveHard Drive
EVGA GTX 970 FTW (win7 guest) 64GB G.SKILL 2133 2x Crucial M4 256GB raid1 4x 3TB raid 10 
CoolingCoolingCoolingCooling
2x Apogee HD  2x RX 480 2x MCP 655 RP-452x2 rev2 (new) 
CoolingCoolingOSOS
16x Cougar Turbine CFT12SB4 (new) EK FC 580 Gentoo (host) Gentoo (x23 guests) 
OSMonitorMonitorPower
windows 7 (guest w/ vfio-pci) Viewsonic 23" 1080P Viewsonic 19" Antec HCP Platinum 1000 (new) 
CaseOtherOther
Case Labs TH10 (still the best ever) 2x Lamptron FC-5 IOGEAR 2 way DVI KVM Switch 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Linux, Unix
Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Linux, Unix › Need some assistance debugging these kernel panics