Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Linux, Unix › CentOS instance problems
New Posts  All Forums:Forum Nav:

CentOS instance problems

post #1 of 11
Thread Starter 
Hey guys, this has been driving me crazy all day.

One of my cloud instances crashed in the morning and resulted in Apache and a few other things getting completely messed up. I finally got httpd to start after recreating some directories but now I can't seem to access the files that I can clearly see via ssh.

If i log onto the server via ssh I can provoke the "locate sitename" command and all of my website files will come up. However, when I attempt to access them (ie. vi *file/path/webfile.txt) vi opens up a new file instead like it would if it didn't exist. SSH is also the only way I can get the files to show up in any manner, if I ftp into the server, all I see are empty directories.

The whole thing is really strange. Any ideas?
T3k
(13 items)
 
The Tibaldi
(9 items)
 
WooJoo
(13 items)
 
CPUMotherboardGraphicsGraphics
Intel i5 3210 h77 HD4000 GeForce 640 LE 
RAMHard DriveOSOS
4gb of something 5400rpm slowness 13.04 Ubuntu x64  Windows 8 Pro 
Monitor
1920x1080 
CPUMotherboardGraphicsRAM
965 BE 4.0GHZ (250 x 15.5) @1.5V m2n32 sli deluxe - 2000mhz @ 1.3V Asus TOP 5850 1GB 950/1200 Kingston HyperX 5-5-5-15 
Hard DriveOptical DriveOSMonitor
WD Velociraptor 150GB, WD raptor 74gb TDK 880N DVDRW Windows 7 x64 Ultimate Westinghouse 22" lcd 
KeyboardPowerCaseMouse
saitek eclipse II Corsair 620W modular Cooler Master ATCS 840 black emprex 
  hide details  
Reply
T3k
(13 items)
 
The Tibaldi
(9 items)
 
WooJoo
(13 items)
 
CPUMotherboardGraphicsGraphics
Intel i5 3210 h77 HD4000 GeForce 640 LE 
RAMHard DriveOSOS
4gb of something 5400rpm slowness 13.04 Ubuntu x64  Windows 8 Pro 
Monitor
1920x1080 
CPUMotherboardGraphicsRAM
965 BE 4.0GHZ (250 x 15.5) @1.5V m2n32 sli deluxe - 2000mhz @ 1.3V Asus TOP 5850 1GB 950/1200 Kingston HyperX 5-5-5-15 
Hard DriveOptical DriveOSMonitor
WD Velociraptor 150GB, WD raptor 74gb TDK 880N DVDRW Windows 7 x64 Ultimate Westinghouse 22" lcd 
KeyboardPowerCaseMouse
saitek eclipse II Corsair 620W modular Cooler Master ATCS 840 black emprex 
  hide details  
Reply
post #2 of 11
Thread Starter 
Well after many hours of checking and double checking it looks like someone brute forced their way into the server through ssh and deleted everything. This should be fun to fix...
T3k
(13 items)
 
The Tibaldi
(9 items)
 
WooJoo
(13 items)
 
CPUMotherboardGraphicsGraphics
Intel i5 3210 h77 HD4000 GeForce 640 LE 
RAMHard DriveOSOS
4gb of something 5400rpm slowness 13.04 Ubuntu x64  Windows 8 Pro 
Monitor
1920x1080 
CPUMotherboardGraphicsRAM
965 BE 4.0GHZ (250 x 15.5) @1.5V m2n32 sli deluxe - 2000mhz @ 1.3V Asus TOP 5850 1GB 950/1200 Kingston HyperX 5-5-5-15 
Hard DriveOptical DriveOSMonitor
WD Velociraptor 150GB, WD raptor 74gb TDK 880N DVDRW Windows 7 x64 Ultimate Westinghouse 22" lcd 
KeyboardPowerCaseMouse
saitek eclipse II Corsair 620W modular Cooler Master ATCS 840 black emprex 
  hide details  
Reply
T3k
(13 items)
 
The Tibaldi
(9 items)
 
WooJoo
(13 items)
 
CPUMotherboardGraphicsGraphics
Intel i5 3210 h77 HD4000 GeForce 640 LE 
RAMHard DriveOSOS
4gb of something 5400rpm slowness 13.04 Ubuntu x64  Windows 8 Pro 
Monitor
1920x1080 
CPUMotherboardGraphicsRAM
965 BE 4.0GHZ (250 x 15.5) @1.5V m2n32 sli deluxe - 2000mhz @ 1.3V Asus TOP 5850 1GB 950/1200 Kingston HyperX 5-5-5-15 
Hard DriveOptical DriveOSMonitor
WD Velociraptor 150GB, WD raptor 74gb TDK 880N DVDRW Windows 7 x64 Ultimate Westinghouse 22" lcd 
KeyboardPowerCaseMouse
saitek eclipse II Corsair 620W modular Cooler Master ATCS 840 black emprex 
  hide details  
Reply
post #3 of 11
Quote:
Originally Posted by rusky1;13750708 
Well after many hours of checking and double checking it looks like someone brute forced their way into the server through ssh and deleted everything. This should be fun to fix...

no public/private key set up on your ssh access? :/ that shoulda been a first priority, it keeps brute forcing out pretty well honestly...

also, dont you have logs for failed ssh login attempts? thats a good thing to glance at for remote servers occasionally
Kinda meh now...
(13 items)
 
  
CPUMotherboardGraphicsRAM
PhII 940 BE asus M4N82 Deluxe 2x 8800gts(g92) 2x 1g ocz ddr2 1066 
Hard DriveOptical DriveOSMonitor
/dev/sd[abc] /dev/sr0 WinXP/Linux 22in acer 
PowerCaseMouseMouse Pad
950w rosewill timebomb Antec 900 (2small) Voodoo Deathadder (RED!) X-Trac Pads Ripper XL 
  hide details  
Reply
Kinda meh now...
(13 items)
 
  
CPUMotherboardGraphicsRAM
PhII 940 BE asus M4N82 Deluxe 2x 8800gts(g92) 2x 1g ocz ddr2 1066 
Hard DriveOptical DriveOSMonitor
/dev/sd[abc] /dev/sr0 WinXP/Linux 22in acer 
PowerCaseMouseMouse Pad
950w rosewill timebomb Antec 900 (2small) Voodoo Deathadder (RED!) X-Trac Pads Ripper XL 
  hide details  
Reply
post #4 of 11
Thread Starter 
I do have the keys set up. Very recently started managing this server after the previous person decided to run a bot net from it and got fired. He probably left some nice surprises behind when he left which most likely aided the break in. I'm going to rebuild from scratch to make sure nothing funny is going on this time around.
T3k
(13 items)
 
The Tibaldi
(9 items)
 
WooJoo
(13 items)
 
CPUMotherboardGraphicsGraphics
Intel i5 3210 h77 HD4000 GeForce 640 LE 
RAMHard DriveOSOS
4gb of something 5400rpm slowness 13.04 Ubuntu x64  Windows 8 Pro 
Monitor
1920x1080 
CPUMotherboardGraphicsRAM
965 BE 4.0GHZ (250 x 15.5) @1.5V m2n32 sli deluxe - 2000mhz @ 1.3V Asus TOP 5850 1GB 950/1200 Kingston HyperX 5-5-5-15 
Hard DriveOptical DriveOSMonitor
WD Velociraptor 150GB, WD raptor 74gb TDK 880N DVDRW Windows 7 x64 Ultimate Westinghouse 22" lcd 
KeyboardPowerCaseMouse
saitek eclipse II Corsair 620W modular Cooler Master ATCS 840 black emprex 
  hide details  
Reply
T3k
(13 items)
 
The Tibaldi
(9 items)
 
WooJoo
(13 items)
 
CPUMotherboardGraphicsGraphics
Intel i5 3210 h77 HD4000 GeForce 640 LE 
RAMHard DriveOSOS
4gb of something 5400rpm slowness 13.04 Ubuntu x64  Windows 8 Pro 
Monitor
1920x1080 
CPUMotherboardGraphicsRAM
965 BE 4.0GHZ (250 x 15.5) @1.5V m2n32 sli deluxe - 2000mhz @ 1.3V Asus TOP 5850 1GB 950/1200 Kingston HyperX 5-5-5-15 
Hard DriveOptical DriveOSMonitor
WD Velociraptor 150GB, WD raptor 74gb TDK 880N DVDRW Windows 7 x64 Ultimate Westinghouse 22" lcd 
KeyboardPowerCaseMouse
saitek eclipse II Corsair 620W modular Cooler Master ATCS 840 black emprex 
  hide details  
Reply
post #5 of 11
Quote:
Originally Posted by rusky1;13751498 
I do have the keys set up. Very recently started managing this server after the previous person decided to run a bot net from it and got fired. He probably left some nice surprises behind when he left which most likely aided the break in. I'm going to rebuild from scratch to make sure nothing funny is going on this time around.

server administrator job termination without server security overhaul? frown.gif that kinda makes me sad actually...
Kinda meh now...
(13 items)
 
  
CPUMotherboardGraphicsRAM
PhII 940 BE asus M4N82 Deluxe 2x 8800gts(g92) 2x 1g ocz ddr2 1066 
Hard DriveOptical DriveOSMonitor
/dev/sd[abc] /dev/sr0 WinXP/Linux 22in acer 
PowerCaseMouseMouse Pad
950w rosewill timebomb Antec 900 (2small) Voodoo Deathadder (RED!) X-Trac Pads Ripper XL 
  hide details  
Reply
Kinda meh now...
(13 items)
 
  
CPUMotherboardGraphicsRAM
PhII 940 BE asus M4N82 Deluxe 2x 8800gts(g92) 2x 1g ocz ddr2 1066 
Hard DriveOptical DriveOSMonitor
/dev/sd[abc] /dev/sr0 WinXP/Linux 22in acer 
PowerCaseMouseMouse Pad
950w rosewill timebomb Antec 900 (2small) Voodoo Deathadder (RED!) X-Trac Pads Ripper XL 
  hide details  
Reply
post #6 of 11
we had that happen... disgruntled employee got ****canned and deleted a bunch of stuff right before going out the door. i wanted to prosecute him since I was the one that had to deal with and resolve the issues. mgmt didn't care a day after since everything was fixed. rolleyes.gif

are you sure that's what happened? sounds to me like you just need to run fsck, but probably too late for that.

isn't your network firewalled? how the hell would he have gotten in if you terminated his access? i'd be interested to hear more about how you found out this is what happened, and what happened.
stable again
(25 items)
 
  
CPUCPUMotherboardGraphics
E5-2687W E5-2687W ASUS Z9PED8-WS EVGA GTX 570 (Linux host) 
GraphicsRAMHard DriveHard Drive
EVGA GTX 970 FTW (win7 guest) 64GB G.SKILL 2133 2x Crucial M4 256GB raid1 4x 3TB raid 10 
CoolingCoolingCoolingCooling
2x Apogee HD  2x RX 480 2x MCP 655 RP-452x2 rev2 (new) 
CoolingCoolingOSOS
16x Cougar Turbine CFT12SB4 (new) EK FC 580 Gentoo (host) Gentoo (x23 guests) 
OSMonitorMonitorPower
windows 7 (guest w/ vfio-pci) Viewsonic 23" 1080P Viewsonic 19" Antec HCP Platinum 1000 (new) 
CaseOtherOther
Case Labs TH10 (still the best ever) 2x Lamptron FC-5 IOGEAR 2 way DVI KVM Switch 
  hide details  
Reply
stable again
(25 items)
 
  
CPUCPUMotherboardGraphics
E5-2687W E5-2687W ASUS Z9PED8-WS EVGA GTX 570 (Linux host) 
GraphicsRAMHard DriveHard Drive
EVGA GTX 970 FTW (win7 guest) 64GB G.SKILL 2133 2x Crucial M4 256GB raid1 4x 3TB raid 10 
CoolingCoolingCoolingCooling
2x Apogee HD  2x RX 480 2x MCP 655 RP-452x2 rev2 (new) 
CoolingCoolingOSOS
16x Cougar Turbine CFT12SB4 (new) EK FC 580 Gentoo (host) Gentoo (x23 guests) 
OSMonitorMonitorPower
windows 7 (guest w/ vfio-pci) Viewsonic 23" 1080P Viewsonic 19" Antec HCP Platinum 1000 (new) 
CaseOtherOther
Case Labs TH10 (still the best ever) 2x Lamptron FC-5 IOGEAR 2 way DVI KVM Switch 
  hide details  
Reply
post #7 of 11
Thread Starter 
Quote:
Originally Posted by EntTheGod;13751855 
server administrator job termination without server security overhaul? frown.gif that kinda makes me sad actually...

I've been overhauling it ever since I took over. It's just been one issue after another. Guess I learned the hard way, if you want something done right then you have to do it yourself.
Quote:
Originally Posted by lloyd mcclendon;13751879 
we had that happen... disgruntled employee got ****canned and deleted a bunch of stuff right before going out the door. i wanted to prosecute him since I was the one that had to deal with and resolve the issues. mgmt didn't care a day after since everything was fixed. rolleyes.gif

are you sure that's what happened? sounds to me like you just need to run fsck, but probably too late for that.

isn't your network firewalled? how the hell would he have gotten in if you terminated his access? i'd be interested to hear more about how you found out this is what happened, and what happened.

I'm starting to think that this is retaliation for them prosecuting him after they fired him.

Am I sure that's what happened? Well I am fairly certain that the server was brought down by a syn flood attack since the logs show the kernel throwing a "possible syn flood detected" followed by a bunch of low resources errors and several loops of httpd attempting to kill processes. This is when the cpu usage spiked and made the server crash. The logs also show attempted brute force attacks (login attempts with words going from a to z) and lots of scripted/bot attacks (most checking for certain directories and files). Something must have broken through after the server crashed and was restarted. One day everything worked perfectly fine and the next httpd wouldn't even start up because directories were missing. I recreated the directories hoping that it was just a freak accident only to find out that all of the web pages are gone. Did a locate /ec2volume/*.html to see if they might have gotten moved or something but not even one file was found. So yeah, I think my hypothesis holds some ground.
T3k
(13 items)
 
The Tibaldi
(9 items)
 
WooJoo
(13 items)
 
CPUMotherboardGraphicsGraphics
Intel i5 3210 h77 HD4000 GeForce 640 LE 
RAMHard DriveOSOS
4gb of something 5400rpm slowness 13.04 Ubuntu x64  Windows 8 Pro 
Monitor
1920x1080 
CPUMotherboardGraphicsRAM
965 BE 4.0GHZ (250 x 15.5) @1.5V m2n32 sli deluxe - 2000mhz @ 1.3V Asus TOP 5850 1GB 950/1200 Kingston HyperX 5-5-5-15 
Hard DriveOptical DriveOSMonitor
WD Velociraptor 150GB, WD raptor 74gb TDK 880N DVDRW Windows 7 x64 Ultimate Westinghouse 22" lcd 
KeyboardPowerCaseMouse
saitek eclipse II Corsair 620W modular Cooler Master ATCS 840 black emprex 
  hide details  
Reply
T3k
(13 items)
 
The Tibaldi
(9 items)
 
WooJoo
(13 items)
 
CPUMotherboardGraphicsGraphics
Intel i5 3210 h77 HD4000 GeForce 640 LE 
RAMHard DriveOSOS
4gb of something 5400rpm slowness 13.04 Ubuntu x64  Windows 8 Pro 
Monitor
1920x1080 
CPUMotherboardGraphicsRAM
965 BE 4.0GHZ (250 x 15.5) @1.5V m2n32 sli deluxe - 2000mhz @ 1.3V Asus TOP 5850 1GB 950/1200 Kingston HyperX 5-5-5-15 
Hard DriveOptical DriveOSMonitor
WD Velociraptor 150GB, WD raptor 74gb TDK 880N DVDRW Windows 7 x64 Ultimate Westinghouse 22" lcd 
KeyboardPowerCaseMouse
saitek eclipse II Corsair 620W modular Cooler Master ATCS 840 black emprex 
  hide details  
Reply
post #8 of 11
ouch.... frown.gif

yeah sounds like either a pissed off former employee tongue.gif or just some kids that decided to hack your server... the latter does happen occasionally and if it does no hard feelings just learn from what broke so you can make an even more secure server smile.gif

although... you should get an IPS solution for the network the server is on between your firewall and your server perhaps, could have possibly stopped the brute force and the DOS
Kinda meh now...
(13 items)
 
  
CPUMotherboardGraphicsRAM
PhII 940 BE asus M4N82 Deluxe 2x 8800gts(g92) 2x 1g ocz ddr2 1066 
Hard DriveOptical DriveOSMonitor
/dev/sd[abc] /dev/sr0 WinXP/Linux 22in acer 
PowerCaseMouseMouse Pad
950w rosewill timebomb Antec 900 (2small) Voodoo Deathadder (RED!) X-Trac Pads Ripper XL 
  hide details  
Reply
Kinda meh now...
(13 items)
 
  
CPUMotherboardGraphicsRAM
PhII 940 BE asus M4N82 Deluxe 2x 8800gts(g92) 2x 1g ocz ddr2 1066 
Hard DriveOptical DriveOSMonitor
/dev/sd[abc] /dev/sr0 WinXP/Linux 22in acer 
PowerCaseMouseMouse Pad
950w rosewill timebomb Antec 900 (2small) Voodoo Deathadder (RED!) X-Trac Pads Ripper XL 
  hide details  
Reply
post #9 of 11
Quote:
logs also show attempted brute force attacks (login attempts with words going from a to z)

uh.. where is your firewall and access through a VPN?

whatever service he was logging into .. is exposed on the internet? unless this is some kind of http authentication within a site, i think you have bigger problems...
stable again
(25 items)
 
  
CPUCPUMotherboardGraphics
E5-2687W E5-2687W ASUS Z9PED8-WS EVGA GTX 570 (Linux host) 
GraphicsRAMHard DriveHard Drive
EVGA GTX 970 FTW (win7 guest) 64GB G.SKILL 2133 2x Crucial M4 256GB raid1 4x 3TB raid 10 
CoolingCoolingCoolingCooling
2x Apogee HD  2x RX 480 2x MCP 655 RP-452x2 rev2 (new) 
CoolingCoolingOSOS
16x Cougar Turbine CFT12SB4 (new) EK FC 580 Gentoo (host) Gentoo (x23 guests) 
OSMonitorMonitorPower
windows 7 (guest w/ vfio-pci) Viewsonic 23" 1080P Viewsonic 19" Antec HCP Platinum 1000 (new) 
CaseOtherOther
Case Labs TH10 (still the best ever) 2x Lamptron FC-5 IOGEAR 2 way DVI KVM Switch 
  hide details  
Reply
stable again
(25 items)
 
  
CPUCPUMotherboardGraphics
E5-2687W E5-2687W ASUS Z9PED8-WS EVGA GTX 570 (Linux host) 
GraphicsRAMHard DriveHard Drive
EVGA GTX 970 FTW (win7 guest) 64GB G.SKILL 2133 2x Crucial M4 256GB raid1 4x 3TB raid 10 
CoolingCoolingCoolingCooling
2x Apogee HD  2x RX 480 2x MCP 655 RP-452x2 rev2 (new) 
CoolingCoolingOSOS
16x Cougar Turbine CFT12SB4 (new) EK FC 580 Gentoo (host) Gentoo (x23 guests) 
OSMonitorMonitorPower
windows 7 (guest w/ vfio-pci) Viewsonic 23" 1080P Viewsonic 19" Antec HCP Platinum 1000 (new) 
CaseOtherOther
Case Labs TH10 (still the best ever) 2x Lamptron FC-5 IOGEAR 2 way DVI KVM Switch 
  hide details  
Reply
post #10 of 11
Thread Starter 
Quote:
Originally Posted by lloyd mcclendon;13756279 
uh.. where is your firewall and access through a VPN?

whatever service he was logging into .. is exposed on the internet? unless this is some kind of http authentication within a site, i think you have bigger problems...

He was trying to brute force the ssh connection even though it needed a key pair to access. It obviously didn't work but the attempts were there.
T3k
(13 items)
 
The Tibaldi
(9 items)
 
WooJoo
(13 items)
 
CPUMotherboardGraphicsGraphics
Intel i5 3210 h77 HD4000 GeForce 640 LE 
RAMHard DriveOSOS
4gb of something 5400rpm slowness 13.04 Ubuntu x64  Windows 8 Pro 
Monitor
1920x1080 
CPUMotherboardGraphicsRAM
965 BE 4.0GHZ (250 x 15.5) @1.5V m2n32 sli deluxe - 2000mhz @ 1.3V Asus TOP 5850 1GB 950/1200 Kingston HyperX 5-5-5-15 
Hard DriveOptical DriveOSMonitor
WD Velociraptor 150GB, WD raptor 74gb TDK 880N DVDRW Windows 7 x64 Ultimate Westinghouse 22" lcd 
KeyboardPowerCaseMouse
saitek eclipse II Corsair 620W modular Cooler Master ATCS 840 black emprex 
  hide details  
Reply
T3k
(13 items)
 
The Tibaldi
(9 items)
 
WooJoo
(13 items)
 
CPUMotherboardGraphicsGraphics
Intel i5 3210 h77 HD4000 GeForce 640 LE 
RAMHard DriveOSOS
4gb of something 5400rpm slowness 13.04 Ubuntu x64  Windows 8 Pro 
Monitor
1920x1080 
CPUMotherboardGraphicsRAM
965 BE 4.0GHZ (250 x 15.5) @1.5V m2n32 sli deluxe - 2000mhz @ 1.3V Asus TOP 5850 1GB 950/1200 Kingston HyperX 5-5-5-15 
Hard DriveOptical DriveOSMonitor
WD Velociraptor 150GB, WD raptor 74gb TDK 880N DVDRW Windows 7 x64 Ultimate Westinghouse 22" lcd 
KeyboardPowerCaseMouse
saitek eclipse II Corsair 620W modular Cooler Master ATCS 840 black emprex 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Linux, Unix
Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Linux, Unix › CentOS instance problems