Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Linux, Unix › Best Linux Distro For Cluster
New Posts  All Forums:Forum Nav:

Best Linux Distro For Cluster - Page 4

post #31 of 44
11/12/12 at 12:41pm
  • Shrak
  • (╯°□°）╯︵ ┻━┻
Quote:
Originally Posted by jrl1357 View Post

I may give it a shot. ATM, however, I am completely unconvinced nor expect to be based on what I've read.

http://www.youtube.com/watch?v=UX8Cz6xKYlU

Base install his virtual box on Ubuntu server runs 24mb.
Reply
Reply
post #32 of 44
11/12/12 at 12:43pm
@jrl1357

I still don't see how Debian is "better". I've just rebooted the server in 128MB of RAM and it's still running nicely. If memory usage is your argument, I'm not buying it. I'm not an Ubuntu fanboi or anything like that, I just don't see the basis of your argument. smile.gif
Mythica
(14 items)
  		  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Hitachi Deskstar 7K500 Samsung HD204UI Linux Mint 13 HP L1800 
KeyboardPowerCaseMouse
Trust EasyScroll Silverline Corsair HX520 Lian-Li PC-A04B Logitech Trackman Wheel 
View all
  hide details  
Reply
Mythica
(14 items)
  		  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Hitachi Deskstar 7K500 Samsung HD204UI Linux Mint 13 HP L1800 
KeyboardPowerCaseMouse
Trust EasyScroll Silverline Corsair HX520 Lian-Li PC-A04B Logitech Trackman Wheel 
View all
  hide details  
Reply
post #33 of 44
11/12/12 at 1:13pm
  • Joined: Feb 2012
  • Location: Halifax N.S. Canada
  • Posts: 3,383
  • Rep: 128 (Unique: 98)
  • Reviews: 3
  • Folding Team Rank: 3,079
  •  
    Name Team Rank Points / May Work Units / May
    JLloyd13 3,079 139,050 / 0 70 / 0
     
  • Select All Posts By This User
Quote:
Originally Posted by parityboy View Post

@jrl1357

I still don't see how Debian is "better". I've just rebooted the server in 128MB of RAM and it's still running nicely. If memory usage is your argument, I'm not buying it. I'm not an Ubuntu fanboi or anything like that, I just don't see the basis of your argument. smile.gif


its not much better, its just that ubuntu server seems pretty redundant and pointless. still, that 24 is very impressive, I may have been quick to pass judgment. I would still use debian, but that might work fine as well.
The Daily (K)Module
(14 items)
  		  
CPUMotherboardGraphicsRAM
AMD Phenom II X4 B55 Biostar A880GZ  AMD Radeon HD 4250 iGPU 8GB (2x4GB) Patriot Sector 5 DDR3 
Hard DriveHard DriveHard DriveOptical Drive
Seagate Barracuda 1TB (ST1000DM003) Western Digital Caviar SE 250GB Seagate External 1TB LG DVD Burner 
OSOSOSPower
Debian GNU/Linux 6.0.6 'Squeeze' 64-bit Arch Linux 64-bit FreeBSD 9.0 64-bit LPS Ultra 550 watt 
Case
Thermaltake V4  
View all
  hide details  
Reply
The Daily (K)Module
(14 items)
  		  
CPUMotherboardGraphicsRAM
AMD Phenom II X4 B55 Biostar A880GZ  AMD Radeon HD 4250 iGPU 8GB (2x4GB) Patriot Sector 5 DDR3 
Hard DriveHard DriveHard DriveOptical Drive
Seagate Barracuda 1TB (ST1000DM003) Western Digital Caviar SE 250GB Seagate External 1TB LG DVD Burner 
OSOSOSPower
Debian GNU/Linux 6.0.6 'Squeeze' 64-bit Arch Linux 64-bit FreeBSD 9.0 64-bit LPS Ultra 550 watt 
Case
Thermaltake V4  
View all
  hide details  
Reply
post #34 of 44
11/12/12 at 3:29pm
I think the main point behind Ubuntu server is a) the kernel is better tuned for server type loads than a desktop kernel and b) there's the option of commercial support.

I don't know how the Debian default kernel is tuned, but I'm willing to bet it's probably not optimised for either server nor desktop workloads, since the Debian guys probably don't draw that kind of distinction for the distro. smile.gif
Mythica
(14 items)
  		  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Hitachi Deskstar 7K500 Samsung HD204UI Linux Mint 13 HP L1800 
KeyboardPowerCaseMouse
Trust EasyScroll Silverline Corsair HX520 Lian-Li PC-A04B Logitech Trackman Wheel 
View all
  hide details  
Reply
Mythica
(14 items)
  		  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Hitachi Deskstar 7K500 Samsung HD204UI Linux Mint 13 HP L1800 
KeyboardPowerCaseMouse
Trust EasyScroll Silverline Corsair HX520 Lian-Li PC-A04B Logitech Trackman Wheel 
View all
  hide details  
Reply
post #35 of 44
11/12/12 at 5:36pm
  • Joined: Feb 2012
  • Location: Halifax N.S. Canada
  • Posts: 3,383
  • Rep: 128 (Unique: 98)
  • Reviews: 3
  • Folding Team Rank: 3,079
  •  
    Name Team Rank Points / May Work Units / May
    JLloyd13 3,079 139,050 / 0 70 / 0
     
  • Select All Posts By This User
Quote:
Originally Posted by parityboy View Post

I think the main point behind Ubuntu server is a) the kernel is better tuned for server type loads than a desktop kernel and b) there's the option of commercial support.

I don't know how the Debian default kernel is tuned, but I'm willing to bet it's probably not optimised for either server nor desktop workloads, since the Debian guys probably don't draw that kind of distinction for the distro. smile.gif

actully debian a heavy server orientated distro

EDIT--

for support I favor Red Hat
The Daily (K)Module
(14 items)
  		  
CPUMotherboardGraphicsRAM
AMD Phenom II X4 B55 Biostar A880GZ  AMD Radeon HD 4250 iGPU 8GB (2x4GB) Patriot Sector 5 DDR3 
Hard DriveHard DriveHard DriveOptical Drive
Seagate Barracuda 1TB (ST1000DM003) Western Digital Caviar SE 250GB Seagate External 1TB LG DVD Burner 
OSOSOSPower
Debian GNU/Linux 6.0.6 'Squeeze' 64-bit Arch Linux 64-bit FreeBSD 9.0 64-bit LPS Ultra 550 watt 
Case
Thermaltake V4  
View all
  hide details  
Reply
The Daily (K)Module
(14 items)
  		  
CPUMotherboardGraphicsRAM
AMD Phenom II X4 B55 Biostar A880GZ  AMD Radeon HD 4250 iGPU 8GB (2x4GB) Patriot Sector 5 DDR3 
Hard DriveHard DriveHard DriveOptical Drive
Seagate Barracuda 1TB (ST1000DM003) Western Digital Caviar SE 250GB Seagate External 1TB LG DVD Burner 
OSOSOSPower
Debian GNU/Linux 6.0.6 'Squeeze' 64-bit Arch Linux 64-bit FreeBSD 9.0 64-bit LPS Ultra 550 watt 
Case
Thermaltake V4  
View all
  hide details  
Reply
post #36 of 44
11/13/12 at 12:32am
Quote:
Originally Posted by parityboy View Post

I think the main point behind Ubuntu server is a) the kernel is better tuned for server type loads than a desktop kernel and b) there's the option of commercial support.
I don't know how the Debian default kernel is tuned, but I'm willing to bet it's probably not optimised for either server nor desktop workloads, since the Debian guys probably don't draw that kind of distinction for the distro. smile.gif

Kernel tuning isn't going to make a huge amount of difference even in most server cases and it certainly shouldn't be relied upon for the other few scenarios.
Reply
Reply
post #37 of 44
11/13/12 at 8:34am
@jrl1357

Thanks for that, I've learned something new. smile.gif Never played with raw Debian, only its variants. And yes when it comes to server distro support, RedHat have been in the game far longer.


@Plan9

So, in a scenario where a machine is under heavy load (web serving, databases etc) are you saying a vanilla kernel will perform out of the box nearly as well as a tuned/patched kernel? (love your avatar by the way biggrin.gif Is that your cat?)
Mythica
(14 items)
  		  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Hitachi Deskstar 7K500 Samsung HD204UI Linux Mint 13 HP L1800 
KeyboardPowerCaseMouse
Trust EasyScroll Silverline Corsair HX520 Lian-Li PC-A04B Logitech Trackman Wheel 
View all
  hide details  
Reply
Mythica
(14 items)
  		  
CPUMotherboardGraphicsRAM
Intel i3 530 Gigabyte GA-H55M-D2H Palit nVidia GT430 Corsair Dominator 4GB TW3X4G1333C9A 
Hard DriveHard DriveOSMonitor
Hitachi Deskstar 7K500 Samsung HD204UI Linux Mint 13 HP L1800 
KeyboardPowerCaseMouse
Trust EasyScroll Silverline Corsair HX520 Lian-Li PC-A04B Logitech Trackman Wheel 
View all
  hide details  
Reply
post #38 of 44
11/13/12 at 11:09am
Quote:
Originally Posted by parityboy View Post

So, in a scenario where a machine is under heavy load (web serving, databases etc) are you saying a vanilla kernel will perform out of the box nearly as well as a tuned/patched kernel?

It depends on the type of load really. What you've described could generate a whole plethora of different bottlenecks depending on the specific hardware set up, the daemons running / their config and the type of incoming requests.

For example, website serving static content would have the following potential bottlenecks:
  • bandwidth (obviously); what's you're physical bandwidth? are you sending compressed or uncompressed data?
  • storage access speed (depending on the size of the static content and the number of files - for smaller, fewer files, Linux will cache the data),
  • your kernel's TCP/IP configuration (time outs, stack size, etc)
  • your web daemon config (eg Apache). (Are you running keep-alive? what's your connection limit and thread limit? what's your threading model? (different threading models can affect the speed new TCP/IP sockets are opened)
  • and -obviously- the number of simultaneous requests
  • your physical infrastructure (switches, firewalls, number of hops within your datacenter, etc)
  • additional software running (are folding on your web server lol?, running XWindows on a headless box, etc)
However when you throw a database into the equation, you're then adding the following variables:
  • database throughput (which is dependent on the complexity of SQL queries (number of joins, record calculations, nested sub-queries, etc), table optimizations (indexing etc), table storage engines, size of the data to return, if a different box then the hardware the database is running on (eg RAID controllers, etc) and so on.
  • server side language and APIs used (even in the case of Perl, mod_perl massively out performs FastCGI as mod_perl uses Apache APIs directly instead of interfacing with a "go-between" interface (namely CGI).
  • optimizations in your code (bad code will naturally run slower)
  • web server config (again); as some languages can be further performance tuned for in the http daemon
  • is your db server and web server the same box? and if not, what's their link speed?
And on top of that there's additional tweaks you can perform:
  • reducing output file sizes by "minifying" source code (eg removing white space from CSS, JS and HTML) and using shorter variable names
  • reducing the number of HTTP requests per page (eg using rewrites instead of redirects, having image clips (i think they're called) which have multiple "slides" per image and thus you have fewer images to download but each image is then sliced by JS / CSS when rendered. I've explained that very badly though.
  • clever use of domain names to reduce the number of DNS look ups on the client side (this is purely tuning to make the clients page load faster, you wouldn't see any impact on the server load)
  • or going the other way and having more domain names two bypass web browsers parallel download per domain name limit - thus allowing a webpages resources to load quicker (as above, this wouldn't impact server load but it would make a difference on the client side).

...and out of all of them, only one of those items is relating to the kernel.

So while you could make some kernel tweaks (eg like the ones suggested here) to squeeze a little more performance out of your web servers, there are so many bigger bottlenecks to account for. And by the time you reach the point where the make-or-break of performance is down to kernel tweaks, then that's the time you really need to invest in a few more boxes in your web farm.

I think where kernel tweaks really come into their own is when building ultra-low latency real time systems. But in those cases a server OS may not be the best fit anyway; which again depends massively on the specific set up and it's requirements.
Quote:
Originally Posted by parityboy View Post

(love your avatar by the way biggrin.gif Is that your cat?)

Thanks mate biggrin.gif. No, it's not mine, but my two cats (particularly the youngest) are just as bonkers so it does remind me of them laugher.gif
Edited by Plan9 - 11/13/12 at 11:12am
Reply
Reply
post #39 of 44
11/13/12 at 4:34pm
Ubuntu server no longer has a separate kernel to desktop...
Subjugate the Weak
(19 items)
  		  
CPUMotherboardGraphicsGraphics
Intel Core i7 860 Asus P7P55D-E Pro MSI GTX560 Ti TwinFrozr II SLI MSI GTX560 Ti TwinFrozr II SLI 
RAMHard DriveHard DriveHard Drive
Corsair 8GB DDR3 1600MHz CL9 XMS3 (2 x 4GB) OCZ Vertex 3 SSD Western Digital Caviar Black 1TB 7200RPM 64MB C... Western Digital Caviar Green 1TB ~5900RPM 64MB ... 
Optical DriveCoolingOSMonitor
Lite-On 24x DVD-RW CoolerMaster V8 Windows 7 Professional SP1 3 x Samsung S22B350H 
KeyboardPowerCaseMouse
Ducky Shine II Corsair HX850 CoolerMaster Storm Enforcer Logitech M500 
Mouse PadAudio
Razer Goliathus Microsoft LifeChat LX 3000 
View all
  hide details  
Reply
Subjugate the Weak
(19 items)
  		  
CPUMotherboardGraphicsGraphics
Intel Core i7 860 Asus P7P55D-E Pro MSI GTX560 Ti TwinFrozr II SLI MSI GTX560 Ti TwinFrozr II SLI 
RAMHard DriveHard DriveHard Drive
Corsair 8GB DDR3 1600MHz CL9 XMS3 (2 x 4GB) OCZ Vertex 3 SSD Western Digital Caviar Black 1TB 7200RPM 64MB C... Western Digital Caviar Green 1TB ~5900RPM 64MB ... 
Optical DriveCoolingOSMonitor
Lite-On 24x DVD-RW CoolerMaster V8 Windows 7 Professional SP1 3 x Samsung S22B350H 
KeyboardPowerCaseMouse
Ducky Shine II Corsair HX850 CoolerMaster Storm Enforcer Logitech M500 
Mouse PadAudio
Razer Goliathus Microsoft LifeChat LX 3000 
View all
  hide details  
Reply
post #40 of 44
11/13/12 at 9:11pm
Quote:
Originally Posted by Plan9 View Post

It depends on the type of load really. What you've described could generate a whole plethora of different bottlenecks depending on the specific hardware set up, the daemons running / their config and the type of incoming requests.
For example, website serving static content would have the following potential bottlenecks:
  • bandwidth (obviously); what's you're physical bandwidth? are you sending compressed or uncompressed data?
  • storage access speed (depending on the size of the static content and the number of files - for smaller, fewer files, Linux will cache the data),
  • your kernel's TCP/IP configuration (time outs, stack size, etc)
  • your web daemon config (eg Apache). (Are you running keep-alive? what's your connection limit and thread limit? what's your threading model? (different threading models can affect the speed new TCP/IP sockets are opened)
  • and -obviously- the number of simultaneous requests
  • your physical infrastructure (switches, firewalls, number of hops within your datacenter, etc)
  • additional software running (are folding on your web server lol?, running XWindows on a headless box, etc)
However when you throw a database into the equation, you're then adding the following variables:
  • database throughput (which is dependent on the complexity of SQL queries (number of joins, record calculations, nested sub-queries, etc), table optimizations (indexing etc), table storage engines, size of the data to return, if a different box then the hardware the database is running on (eg RAID controllers, etc) and so on.
  • server side language and APIs used (even in the case of Perl, mod_perl massively out performs FastCGI as mod_perl uses Apache APIs directly instead of interfacing with a "go-between" interface (namely CGI).
  • optimizations in your code (bad code will naturally run slower)
  • web server config (again); as some languages can be further performance tuned for in the http daemon
  • is your db server and web server the same box? and if not, what's their link speed?
And on top of that there's additional tweaks you can perform:
  • reducing output file sizes by "minifying" source code (eg removing white space from CSS, JS and HTML) and using shorter variable names
  • reducing the number of HTTP requests per page (eg using rewrites instead of redirects, having image clips (i think they're called) which have multiple "slides" per image and thus you have fewer images to download but each image is then sliced by JS / CSS when rendered. I've explained that very badly though.
  • clever use of domain names to reduce the number of DNS look ups on the client side (this is purely tuning to make the clients page load faster, you wouldn't see any impact on the server load)
  • or going the other way and having more domain names two bypass web browsers parallel download per domain name limit - thus allowing a webpages resources to load quicker (as above, this wouldn't impact server load but it would make a difference on the client side).
...and out of all of them, only one of those items is relating to the kernel.
So while you could make some kernel tweaks (eg like the ones suggested here) to squeeze a little more performance out of your web servers, there are so many bigger bottlenecks to account for. And by the time you reach the point where the make-or-break of performance is down to kernel tweaks, then that's the time you really need to invest in a few more boxes in your web farm.
I think where kernel tweaks really come into their own is when building ultra-low latency real time systems. But in those cases a server OS may not be the best fit anyway; which again depends massively on the specific set up and it's requirements.
Thanks mate biggrin.gif. No, it's not mine, but my two cats (particularly the youngest) are just as bonkers so it does remind me of them laugher.gif

That's a really good breakdown of the inner workings of a web farm and a lot of optimization paths. I'd love to have a write up like that to pass along to my peers at work to help understand what we're supporting....I work in an operations center for a .com. would it be possible to use that as such?
Hecate
(10 items)
 
Frejya
(7 items)
  		 
CPUMotherboardGraphicsRAM
i5 3570k z77-v pro EVGA GeForce GTX670 FTW+ 4096MB Samsung MV-3V4G3D 
Hard DriveHard DriveKeyboardPower
Samsung 830 WD Blue 1 TB Ducky Shine II Pc Power and cooling 
CaseMouse
Switch 810 Gun-Metal Corsair M90 
View all
CPUMotherboardGraphicsRAM
AMD Phenom II X4 940 M4A79 Deluxe NVIDIA GeForce 9800 GTX/9800 GTX+ Corsair  
OSKeyboardCase
Windows XP64 MS Natural Ergonomic Cooler Master CM Stacker 830 
View all
  hide details  
Reply
Hecate
(10 items)
 
Frejya
(7 items)
  		 
CPUMotherboardGraphicsRAM
i5 3570k z77-v pro EVGA GeForce GTX670 FTW+ 4096MB Samsung MV-3V4G3D 
Hard DriveHard DriveKeyboardPower
Samsung 830 WD Blue 1 TB Ducky Shine II Pc Power and cooling 
CaseMouse
Switch 810 Gun-Metal Corsair M90 
View all
CPUMotherboardGraphicsRAM
AMD Phenom II X4 940 M4A79 Deluxe NVIDIA GeForce 9800 GTX/9800 GTX+ Corsair  
OSKeyboardCase
Windows XP64 MS Natural Ergonomic Cooler Master CM Stacker 830 
View all
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Linux, Unix
Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Linux, Unix › Best Linux Distro For Cluster