Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Linux, Unix › dazed and confused: server configuration problem
New Posts  All Forums:Forum Nav:

dazed and confused: server configuration problem

post #1 of 15
Thread Starter 

bear with me on this, as this is a sorta hard to explain and follow problem. it is something i quite haven't seen before, and i've already hit up the usual suspects, but none of them are installed.

 

its on my personal server, it is a vps server ran by a friend of mines company, it has centos and cpanel installed. 

 

we noticed this problem a few weeks back, while editing a css template for Invision Power Board, since we've already eliminated it being a IPB bug, their tech support will not help, and exhaustive searches of google has not yielded a satisfactory answer, it has me stumped, another developer and the sysadmin just looked at me dumbfouned when i skyped him earlier.

 

so here is the problem, you can see it first hand here at this link:

 

http://demo.codingjungle.com/d.php

 

just click on "dont 500 ne" to see it, leave the text that is there intact.

 

basically on $_REQUEST/$_POST/$_GET, if it contains a combination of at least these 3 words:

 

update settings =  

 

^ yes, they have to be in that order (which often times in IPB templates, they can appear in this order). when posted thru a post form or ajax, it will throw a 500 server configuration page.

 

the link above, will post to itself, so i know the page exist, just not sure what the problem is, as at first i thought it was a database charset encoding problem, but the above link doesn't post to a database, then i thought maybe it was suhosin not being configured correctly, but suhosin is not installed. then i checked the apache configuration, thinking it was mod_fcgid, but that is not enabled (as FcgidMaxRequestLen if it is too small, will produce a similar problem). 

 

the standard set of php modules are enabled, and from what i can tell from my look at the http.conf, only fastinclude and bw_limit are enabled for apache.

 

so i'm stumped have spent the entire day looking into this problem, as it is now effecting myself and my clients ability to run their websites...any help will be appreciated.

Bazinga Punk
(12 items)
 
ooh shiny!
(6 items)
 
 
CPUMotherboardGraphicsRAM
Intel Xeon 3440 AsRock P55 extreme Evga 8800 GT 512 MB Gskill Ripjaws 
Hard DriveCoolingOSMonitor
Western Digital Blue Antec Khuler 620 Ubuntu 11.10 Asus vw264H 
KeyboardPowerCaseMouse
GIGABYTE KM7600 CORSAIR TX 650 Cooler Master 590 GIGABYTE GM-M6800 
CPUMotherboardGraphicsRAM
Intel Core I5 6500 Gigabyte z170xp-SLI Nvidia 970gtx Corsair 16gb ddr4 2666mhz  
Hard DriveOS
250gb Samsung Evo 850 Windows 10 & Ubuntu 15.10 
  hide details  
Reply
Bazinga Punk
(12 items)
 
ooh shiny!
(6 items)
 
 
CPUMotherboardGraphicsRAM
Intel Xeon 3440 AsRock P55 extreme Evga 8800 GT 512 MB Gskill Ripjaws 
Hard DriveCoolingOSMonitor
Western Digital Blue Antec Khuler 620 Ubuntu 11.10 Asus vw264H 
KeyboardPowerCaseMouse
GIGABYTE KM7600 CORSAIR TX 650 Cooler Master 590 GIGABYTE GM-M6800 
CPUMotherboardGraphicsRAM
Intel Core I5 6500 Gigabyte z170xp-SLI Nvidia 970gtx Corsair 16gb ddr4 2666mhz  
Hard DriveOS
250gb Samsung Evo 850 Windows 10 & Ubuntu 15.10 
  hide details  
Reply
post #2 of 15
Have you tried restoring from a backup, or re-installing the PHP modules?
The Claw
(23 items)
 
Mates
(20 items)
 
Dream PC Comp
(7 items)
 
CPUMotherboardGraphicsGraphics
I7-950 @ 3.8GHz ASRock X58 Extreme3 Gigabyte GTX 760 @ 1350 EVGA GTX 560 ( PhysX ) 
RAMHard DriveHard DriveHard Drive
12GB Corsair 1600Mhz Corsair Force 80GB SSD Samsung Spinpoint 1TB Samsung Spinpoint 1TB 
Hard DriveHard DriveOptical DriveCooling
Seagate 2TB Corsair Force GS 128GB Pioneer 12X Blu-Ray Corsair H100i 
OSMonitorMonitorKeyboard
Windows 8.1 Pro 64 Bit Samsung 22 Inch LCD Asus 27 Inch LED Logitech G110 
PowerCaseMouseMouse Pad
Corsair AX 750 Coolermaster CM 690 II Advanced Nvidia Edition. Logitech G5 Razor Goliath 
AudioAudioOther
Logitech 5.1 Corsair Vengeance 1500 7.1  IKEA DESK!!!!!! 
CPUMotherboardGraphicsGraphics
I5-2500K ASrock Z68 Extreme3 Gen3 Sapphire OC 7970 Sapphire OC 7970 
RAMHard DriveHard DriveHard Drive
Crosair Vengance 8GB 1600 Mhz OCZ Agility 3 120GB WD 500GB 7200 RPM WD 2TB 7200 RPM 
Hard DriveOptical DriveCoolingOS
Corsair Forice GS 128GB LG Optical CD/DVD Coolermaster V6 GT Windows 7 Pro 
MonitorMonitorMonitorKeyboard
Asus VE248Q Asus VE248Q Asus VE248Q Microsoft Essentials 
PowerCaseMouseMouse Pad
Corsair AX 760 CoolerMaster nvidia Edition CM 690 II Advanced  Logitech G500 Razor Golithias 
CPUMotherboardGraphicsRAM
Intel I7 4770K ASUS Maximus VI Extreme Motherboard MSI GTX 780 Corsair Vengeance 32GB 
CoolingPowerCase
Corsair H100 Corsair AX 860 corsair 600T White 
  hide details  
Reply
The Claw
(23 items)
 
Mates
(20 items)
 
Dream PC Comp
(7 items)
 
CPUMotherboardGraphicsGraphics
I7-950 @ 3.8GHz ASRock X58 Extreme3 Gigabyte GTX 760 @ 1350 EVGA GTX 560 ( PhysX ) 
RAMHard DriveHard DriveHard Drive
12GB Corsair 1600Mhz Corsair Force 80GB SSD Samsung Spinpoint 1TB Samsung Spinpoint 1TB 
Hard DriveHard DriveOptical DriveCooling
Seagate 2TB Corsair Force GS 128GB Pioneer 12X Blu-Ray Corsair H100i 
OSMonitorMonitorKeyboard
Windows 8.1 Pro 64 Bit Samsung 22 Inch LCD Asus 27 Inch LED Logitech G110 
PowerCaseMouseMouse Pad
Corsair AX 750 Coolermaster CM 690 II Advanced Nvidia Edition. Logitech G5 Razor Goliath 
AudioAudioOther
Logitech 5.1 Corsair Vengeance 1500 7.1  IKEA DESK!!!!!! 
CPUMotherboardGraphicsGraphics
I5-2500K ASrock Z68 Extreme3 Gen3 Sapphire OC 7970 Sapphire OC 7970 
RAMHard DriveHard DriveHard Drive
Crosair Vengance 8GB 1600 Mhz OCZ Agility 3 120GB WD 500GB 7200 RPM WD 2TB 7200 RPM 
Hard DriveOptical DriveCoolingOS
Corsair Forice GS 128GB LG Optical CD/DVD Coolermaster V6 GT Windows 7 Pro 
MonitorMonitorMonitorKeyboard
Asus VE248Q Asus VE248Q Asus VE248Q Microsoft Essentials 
PowerCaseMouseMouse Pad
Corsair AX 760 CoolerMaster nvidia Edition CM 690 II Advanced  Logitech G500 Razor Golithias 
CPUMotherboardGraphicsRAM
Intel I7 4770K ASUS Maximus VI Extreme Motherboard MSI GTX 780 Corsair Vengeance 32GB 
CoolingPowerCase
Corsair H100 Corsair AX 860 corsair 600T White 
  hide details  
Reply
post #3 of 15
Thread Starter 

yeah we've tried that, seems to be a configuration problem as far as i can tell, just not sure where to go. as the apache error logs are of no use, neither are any of the system logs. i've gone over everything and the dev server i use at home is configured almost identical with a few differences and it does not have this problem...and we have already made the changes to the production server that differ on the dev server with no effect.

 

our last thing we are going to try to do, is back up all the sites, bring down the server, and redo the vps image and start from scratch, but without knowing what caused it just asking for it to happen again...so we want to try to resolve it before taking a drastic action like that.

 

thanks for the help tho.

Bazinga Punk
(12 items)
 
ooh shiny!
(6 items)
 
 
CPUMotherboardGraphicsRAM
Intel Xeon 3440 AsRock P55 extreme Evga 8800 GT 512 MB Gskill Ripjaws 
Hard DriveCoolingOSMonitor
Western Digital Blue Antec Khuler 620 Ubuntu 11.10 Asus vw264H 
KeyboardPowerCaseMouse
GIGABYTE KM7600 CORSAIR TX 650 Cooler Master 590 GIGABYTE GM-M6800 
CPUMotherboardGraphicsRAM
Intel Core I5 6500 Gigabyte z170xp-SLI Nvidia 970gtx Corsair 16gb ddr4 2666mhz  
Hard DriveOS
250gb Samsung Evo 850 Windows 10 & Ubuntu 15.10 
  hide details  
Reply
Bazinga Punk
(12 items)
 
ooh shiny!
(6 items)
 
 
CPUMotherboardGraphicsRAM
Intel Xeon 3440 AsRock P55 extreme Evga 8800 GT 512 MB Gskill Ripjaws 
Hard DriveCoolingOSMonitor
Western Digital Blue Antec Khuler 620 Ubuntu 11.10 Asus vw264H 
KeyboardPowerCaseMouse
GIGABYTE KM7600 CORSAIR TX 650 Cooler Master 590 GIGABYTE GM-M6800 
CPUMotherboardGraphicsRAM
Intel Core I5 6500 Gigabyte z170xp-SLI Nvidia 970gtx Corsair 16gb ddr4 2666mhz  
Hard DriveOS
250gb Samsung Evo 850 Windows 10 & Ubuntu 15.10 
  hide details  
Reply
post #4 of 15
Best thing I can say is image that problem box to another server, and start looking through the configs from the ground up...

I reckon a PHP module has been corrupted... from the looks of a 500 internal error.

I know what its like working with dev and production servers and all the shenanigans between.
The Claw
(23 items)
 
Mates
(20 items)
 
Dream PC Comp
(7 items)
 
CPUMotherboardGraphicsGraphics
I7-950 @ 3.8GHz ASRock X58 Extreme3 Gigabyte GTX 760 @ 1350 EVGA GTX 560 ( PhysX ) 
RAMHard DriveHard DriveHard Drive
12GB Corsair 1600Mhz Corsair Force 80GB SSD Samsung Spinpoint 1TB Samsung Spinpoint 1TB 
Hard DriveHard DriveOptical DriveCooling
Seagate 2TB Corsair Force GS 128GB Pioneer 12X Blu-Ray Corsair H100i 
OSMonitorMonitorKeyboard
Windows 8.1 Pro 64 Bit Samsung 22 Inch LCD Asus 27 Inch LED Logitech G110 
PowerCaseMouseMouse Pad
Corsair AX 750 Coolermaster CM 690 II Advanced Nvidia Edition. Logitech G5 Razor Goliath 
AudioAudioOther
Logitech 5.1 Corsair Vengeance 1500 7.1  IKEA DESK!!!!!! 
CPUMotherboardGraphicsGraphics
I5-2500K ASrock Z68 Extreme3 Gen3 Sapphire OC 7970 Sapphire OC 7970 
RAMHard DriveHard DriveHard Drive
Crosair Vengance 8GB 1600 Mhz OCZ Agility 3 120GB WD 500GB 7200 RPM WD 2TB 7200 RPM 
Hard DriveOptical DriveCoolingOS
Corsair Forice GS 128GB LG Optical CD/DVD Coolermaster V6 GT Windows 7 Pro 
MonitorMonitorMonitorKeyboard
Asus VE248Q Asus VE248Q Asus VE248Q Microsoft Essentials 
PowerCaseMouseMouse Pad
Corsair AX 760 CoolerMaster nvidia Edition CM 690 II Advanced  Logitech G500 Razor Golithias 
CPUMotherboardGraphicsRAM
Intel I7 4770K ASUS Maximus VI Extreme Motherboard MSI GTX 780 Corsair Vengeance 32GB 
CoolingPowerCase
Corsair H100 Corsair AX 860 corsair 600T White 
  hide details  
Reply
The Claw
(23 items)
 
Mates
(20 items)
 
Dream PC Comp
(7 items)
 
CPUMotherboardGraphicsGraphics
I7-950 @ 3.8GHz ASRock X58 Extreme3 Gigabyte GTX 760 @ 1350 EVGA GTX 560 ( PhysX ) 
RAMHard DriveHard DriveHard Drive
12GB Corsair 1600Mhz Corsair Force 80GB SSD Samsung Spinpoint 1TB Samsung Spinpoint 1TB 
Hard DriveHard DriveOptical DriveCooling
Seagate 2TB Corsair Force GS 128GB Pioneer 12X Blu-Ray Corsair H100i 
OSMonitorMonitorKeyboard
Windows 8.1 Pro 64 Bit Samsung 22 Inch LCD Asus 27 Inch LED Logitech G110 
PowerCaseMouseMouse Pad
Corsair AX 750 Coolermaster CM 690 II Advanced Nvidia Edition. Logitech G5 Razor Goliath 
AudioAudioOther
Logitech 5.1 Corsair Vengeance 1500 7.1  IKEA DESK!!!!!! 
CPUMotherboardGraphicsGraphics
I5-2500K ASrock Z68 Extreme3 Gen3 Sapphire OC 7970 Sapphire OC 7970 
RAMHard DriveHard DriveHard Drive
Crosair Vengance 8GB 1600 Mhz OCZ Agility 3 120GB WD 500GB 7200 RPM WD 2TB 7200 RPM 
Hard DriveOptical DriveCoolingOS
Corsair Forice GS 128GB LG Optical CD/DVD Coolermaster V6 GT Windows 7 Pro 
MonitorMonitorMonitorKeyboard
Asus VE248Q Asus VE248Q Asus VE248Q Microsoft Essentials 
PowerCaseMouseMouse Pad
Corsair AX 760 CoolerMaster nvidia Edition CM 690 II Advanced  Logitech G500 Razor Golithias 
CPUMotherboardGraphicsRAM
Intel I7 4770K ASUS Maximus VI Extreme Motherboard MSI GTX 780 Corsair Vengeance 32GB 
CoolingPowerCase
Corsair H100 Corsair AX 860 corsair 600T White 
  hide details  
Reply
post #5 of 15
Thread Starter 

I really do not believe it is related to php. I did an update this morning, built apache, php, mysql from source and installed them, problem remains.

Bazinga Punk
(12 items)
 
ooh shiny!
(6 items)
 
 
CPUMotherboardGraphicsRAM
Intel Xeon 3440 AsRock P55 extreme Evga 8800 GT 512 MB Gskill Ripjaws 
Hard DriveCoolingOSMonitor
Western Digital Blue Antec Khuler 620 Ubuntu 11.10 Asus vw264H 
KeyboardPowerCaseMouse
GIGABYTE KM7600 CORSAIR TX 650 Cooler Master 590 GIGABYTE GM-M6800 
CPUMotherboardGraphicsRAM
Intel Core I5 6500 Gigabyte z170xp-SLI Nvidia 970gtx Corsair 16gb ddr4 2666mhz  
Hard DriveOS
250gb Samsung Evo 850 Windows 10 & Ubuntu 15.10 
  hide details  
Reply
Bazinga Punk
(12 items)
 
ooh shiny!
(6 items)
 
 
CPUMotherboardGraphicsRAM
Intel Xeon 3440 AsRock P55 extreme Evga 8800 GT 512 MB Gskill Ripjaws 
Hard DriveCoolingOSMonitor
Western Digital Blue Antec Khuler 620 Ubuntu 11.10 Asus vw264H 
KeyboardPowerCaseMouse
GIGABYTE KM7600 CORSAIR TX 650 Cooler Master 590 GIGABYTE GM-M6800 
CPUMotherboardGraphicsRAM
Intel Core I5 6500 Gigabyte z170xp-SLI Nvidia 970gtx Corsair 16gb ddr4 2666mhz  
Hard DriveOS
250gb Samsung Evo 850 Windows 10 & Ubuntu 15.10 
  hide details  
Reply
post #6 of 15
  • Have you checked Apache's error log (every HTTP Status 500 will write a message in the log)?
  • Have you gone through php.ini?
  • Have you checked your .htaccess file to see if there's any mod_rewrite rules acting up?


If you want, drop me a PM and I can hope on the server and take a look. However I can understand if you didn't want to give me that kind of access smile.gif
post #7 of 15
Thread Starter 

[Thu Nov 15 11:27:46 2012] [error] [client <blah don't want to expose an ip>] File does not exist: /home/codingju/demo/500.shtml, referer: http://demo.codingjungle.com/d.php

 

^this all the error log contains about the problem.

 

php.ini has been reverted to a stock one with the basics enabled.

 

 

 

Quote:
<IfModule mod_rewrite.c>
Options -MultiViews
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !.*\.(jpeg|jpg|gif|png|ico)$
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . / [L]
</IfModule>

 

 

and this is the contents of my main sites htaccess, the demo board doesn't have one.

 

and i do wish i could give you access to the server in such a fashion, as i don't even have access in that manner without being given it when i ask. but i can get you any information you require about it.

Bazinga Punk
(12 items)
 
ooh shiny!
(6 items)
 
 
CPUMotherboardGraphicsRAM
Intel Xeon 3440 AsRock P55 extreme Evga 8800 GT 512 MB Gskill Ripjaws 
Hard DriveCoolingOSMonitor
Western Digital Blue Antec Khuler 620 Ubuntu 11.10 Asus vw264H 
KeyboardPowerCaseMouse
GIGABYTE KM7600 CORSAIR TX 650 Cooler Master 590 GIGABYTE GM-M6800 
CPUMotherboardGraphicsRAM
Intel Core I5 6500 Gigabyte z170xp-SLI Nvidia 970gtx Corsair 16gb ddr4 2666mhz  
Hard DriveOS
250gb Samsung Evo 850 Windows 10 & Ubuntu 15.10 
  hide details  
Reply
Bazinga Punk
(12 items)
 
ooh shiny!
(6 items)
 
 
CPUMotherboardGraphicsRAM
Intel Xeon 3440 AsRock P55 extreme Evga 8800 GT 512 MB Gskill Ripjaws 
Hard DriveCoolingOSMonitor
Western Digital Blue Antec Khuler 620 Ubuntu 11.10 Asus vw264H 
KeyboardPowerCaseMouse
GIGABYTE KM7600 CORSAIR TX 650 Cooler Master 590 GIGABYTE GM-M6800 
CPUMotherboardGraphicsRAM
Intel Core I5 6500 Gigabyte z170xp-SLI Nvidia 970gtx Corsair 16gb ddr4 2666mhz  
Hard DriveOS
250gb Samsung Evo 850 Windows 10 & Ubuntu 15.10 
  hide details  
Reply
post #8 of 15
Did you definitely check both the apache logs and your vhost logs? (sorry to ask this again, and I'm sure you did, but I've never known Apache not to log the fault before so just want to check).

Bit of a long shot, but you cuold try turning off mod_rewrite for now and see if that still happens. (I know you'll need it going forward).

How are your vhosts configured? How is PHP configured?

Not sure if this would highlight anything useful, but you could also try dropping apache down to one thread then running an strace against it to record the processes leading up to the 500.

Also, is this running on FreeBSD?

My gut instinct says it's some security policy in php.ini or an Apache module, given the text that generates the fail. Possibly something designed to prevent SQL injection attacks.
Edited by Plan9 - 11/15/12 at 10:23am
post #9 of 15
Thread Starter 
server is running centos 6, latest available kernel
apache version : 2.2.23
loaded modules: mod_bwlimited, mod_fastinclude (cpanel thing).
 
php: 5.3.17 link to phpinfo() dump
 
 
that is what i'm thinking, there is something hardened, and i just can't figure out what it is, i'm waiting on my guy to send me the apache.conf file, and to get me access to the complete system logs, but when i checked them last night, they were riddled with similar as what i posted before (the apache error and the one for my vhost).
 
part of my frustration right now, is this having to go thru him to get the information, as i'm sure i would be a lot closer or already solved it if i had complete access to said server.
Bazinga Punk
(12 items)
 
ooh shiny!
(6 items)
 
 
CPUMotherboardGraphicsRAM
Intel Xeon 3440 AsRock P55 extreme Evga 8800 GT 512 MB Gskill Ripjaws 
Hard DriveCoolingOSMonitor
Western Digital Blue Antec Khuler 620 Ubuntu 11.10 Asus vw264H 
KeyboardPowerCaseMouse
GIGABYTE KM7600 CORSAIR TX 650 Cooler Master 590 GIGABYTE GM-M6800 
CPUMotherboardGraphicsRAM
Intel Core I5 6500 Gigabyte z170xp-SLI Nvidia 970gtx Corsair 16gb ddr4 2666mhz  
Hard DriveOS
250gb Samsung Evo 850 Windows 10 & Ubuntu 15.10 
  hide details  
Reply
Bazinga Punk
(12 items)
 
ooh shiny!
(6 items)
 
 
CPUMotherboardGraphicsRAM
Intel Xeon 3440 AsRock P55 extreme Evga 8800 GT 512 MB Gskill Ripjaws 
Hard DriveCoolingOSMonitor
Western Digital Blue Antec Khuler 620 Ubuntu 11.10 Asus vw264H 
KeyboardPowerCaseMouse
GIGABYTE KM7600 CORSAIR TX 650 Cooler Master 590 GIGABYTE GM-M6800 
CPUMotherboardGraphicsRAM
Intel Core I5 6500 Gigabyte z170xp-SLI Nvidia 970gtx Corsair 16gb ddr4 2666mhz  
Hard DriveOS
250gb Samsung Evo 850 Windows 10 & Ubuntu 15.10 
  hide details  
Reply
post #10 of 15
Quote:
Originally Posted by Transhour View Post

server is running centos 6, latest available kernel
apache version : 2.2.23
loaded modules: mod_bwlimited, mod_fastinclude (cpanel thing).
 
You'll have more modules loaded than that. mod_rewrite being the obvious one (because of your .htaccess file). Though they might be compiled into to httpd.
Can you run
Code:
httpd -l
(lowcase L) to view the compiled in modules?

Also, if you don't mind me asking, why did you compile your own build from source rather than pulling a copy from CentOS' repositories? (we do a similar thing at my work so I'm not making a criticism here smile.gif )
Quote:
Originally Posted by Transhour View Post

php: 5.3.17 link to phpinfo() dump 
Cheers mate. I'll have a look through that and see how it compares to my PHP boxes.

 
Quote:
Originally Posted by Transhour View Post

that is what i'm thinking, there is something hardened, and i just can't figure out what it is, i'm waiting on my guy to send me the apache.conf file, and to get me access to the complete system logs, but when i checked them last night, they were riddled with similar as what i posted before (the apache error and the one for my vhost).
 
part of my frustration right now, is this having to go thru him to get the information, as i'm sure i would be a lot closer or already solved it if i had complete access to said server.
You could write a PHP routine to read the logs for you - side stepping him completely.

Also, it might be worth disabling the bespoke error pages in your apache config so you don't get 404's on 500's (etc), which might make the logs more manageable.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Linux, Unix
Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Linux, Unix › dazed and confused: server configuration problem