Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › Free Cluster Access for OCN!
New Posts  All Forums:Forum Nav:

Free Cluster Access for OCN!

post #1 of 37
Thread Starter 
Interested in parallel or distributed programming but don’t have a cluster?? No problem!

Please read all of the information below!

Some of you have probably already seen my thread about the creation of my cluster aptly titled The Tower of Power. If not, check it: Tower of Power!

This system usually sits idle, so I’m opening up the opportunity to use it to OCN. It’s already configured with a resource manager, scheduler, MPI, etc. So all you have to do is write, compile, and submit your programs to the scheduler.

So, are you interested??? If so, keep on reading!
To get access, first read the rules below. Then send me a PM requesting access, PLEASE DO NOT REQUEST ACCESS IF YOU ARE NOT DOING PARALLEL PROGRAMMING OR RELATED WORK. It takes time and resources to create and manage users. Not to mention there isn’t much else you can do other than programming over a secure shell. If in doubt, just ask me.


RULES: PLEASE READ BEFORE REQUESTING ACCESS!
1.Use of the cluster is only for parallel programming or related activities.
2.Your usage will be logged, do not assume privacy (this is on a university network after all)
3.Do not use the cluster for mission critical work, or any commercial work. This is for education!
4.OCN and myself are not responsible for any data you put on the cluster.
5.Keep files small, don’t create massive data files, disk space is limited.
6.Do not run anything outside of the job scheduler, you will mess up yours and other people’s work. (This only applies if you are running things on nodes, feel free to execute directly on the head node, the scheduler doesn’t use the head node.)
7.No malicious behavior, remember everything is logged. Any illegal activity will not only be reported to OCN, but also to local authorities. Again this is a University network, they don’t take kindly to that kind of stuff.
8. NO FOLDING! I love F@H, and I am working to get it up and running on the nodes when they are idle. But F@H is not a cluster application to the end user, so there's no reason to be using it on here. Oh and the nodes don't have access to the net

This stuff should be common sense, but it needs to be said. If you’re just looking to play around in Linux, build your own box.


INSTRUCTIONS:
1.When you get your login information, you can SSH into the server which is at: xravin.com
2.As soon as you login, a script will run to get your account setup. Just hit enter to any prompts, DO NOT ENTER ANYTHING! Otherwise you’ll find stuff won’t work.
3.When the script gets you setup, it will run an MPI trace which will show you the nodes you’re connected to. If this errors out, please PM me. It could be the system is down, but we want to make sure.
4.Ok, now your account is ready to rock! We need to make sure you can get to the scheduler, your account has an example ready to go so execute the following commands:

cd example
qsub pbs


At this point, a parallel Pi calculator is run on 8 nodes. It takes about 10 seconds and then you should have a file named ‘pbs.oXXX’ where the ‘XXX’ is a number. Do a cat or a less on that file to see the output, it should show check-ins from the nodes, the value of Pi, and the time elapsed. If you got this, you’re good to go! Otherwise get in touch with me about what erred out. Oh, and that pbs script you submitted with qsub is a great jumping off point for making your own scripts.

Note: The single cores have hyperthreading on, and the manager will allot 2 processes to each unless you specify. Nodes 10, 11, and 12 are dual cores without HT so keep this in mind if you are looking to maximize performance. Basically the resource manager sees all nodes as having 2 CPUs.



I’m not including a tutorial on how to use the job manager or MPI etc, because there are thousands of good pages out there explaining their use, and I couldn’t write anything better. If you’re just starting, look at the example pbs script and google the following:
1.Programming with MPI
2.MPI examples
3.MPICH2 (check out the user’s guide)
4.PBS scripting (very important to get things running)


REMEMBER; DO NOT RUN JOBS OUTSIDE OF THE SCHEDULER. AKA DO NOT EXECUTE A MPIEXEC COMMAND UNLESS IT’S INSIDE A PBS SCRIPT YOU SUBMIT. THIS IS FOR EVERYONE’S BENEFIT INCLUDING YOURS!



IF IN DOUBT, JUST ASK ME! I’d rather you ask then bomb everyone’s jobs Don’t worry, there really isn’t anything you can screw up though, it’ll just bog down the system.

Cluster Information:
Ubuntu Lucid Lynx 10.04

10x: Pentium 4 540, 3.2Ghz, Hyperthreading is on and accounted for in the manager.
3x: Pentium D 3.4Ghz, no HT.
All nodes: 1GB ram, 100mbps network (eventually upgradeable to 1Gbps)

Headnode: Pentium 4 3.6Ghz, 1.5GB ram, 160GB disk, dedicated NICs for both internet and the cluster subnet.

Flat network topology, the nodes are on the 10.11.12.xxx subnet with 10.11.12.1 being the headnode’s eth1. The nodes cannot access the internet. This shouldn’t be an issue for most people, but I’m working on it. It will work for a while then magically stop *shrugs*. The WAN is 128.210.164.159 on the headnode’s eth0.

Naming: The server is ‘cluster-master’, the nodes are ‘node0’ through ‘node12’. Nodes 0 through 9 are single cores, node 10, 11, and 12 are the duals.

Job manager: Torque (open source Portable Batch System (PBS))

Software: MPICH2, gcc, g++, etc. If you need something I don't have, I'll put it on as long as I can apt-get it. Just email or PM me.

Security: fail2ban will ban your IP if you mess up your password more than 5 or 6 times.



Edited by X-Ravin - 6/12/10 at 10:53pm
    
CPUMotherboardGraphicsRAM
11x P4s, 3x PDs (all 3.2 to 3.4Ghz) 14x HP mobos Why? 14x 1GB 
Hard DriveOptical DriveOSMonitor
1x 160GB SATA Yeah Ubuntu Lucid SSH 
PowerCase
9x 300W HP supplies 5 Threaded rods 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
11x P4s, 3x PDs (all 3.2 to 3.4Ghz) 14x HP mobos Why? 14x 1GB 
Hard DriveOptical DriveOSMonitor
1x 160GB SATA Yeah Ubuntu Lucid SSH 
PowerCase
9x 300W HP supplies 5 Threaded rods 
  hide details  
Reply
post #2 of 37
Thread Starter 
Cluster News, Updates, and Work in Progress

We are currently DOWN Blew the breaker. Found a new room, but it will take time to get the power and network run into it.
Edited by X-Ravin - 9/14/10 at 9:27am
    
CPUMotherboardGraphicsRAM
11x P4s, 3x PDs (all 3.2 to 3.4Ghz) 14x HP mobos Why? 14x 1GB 
Hard DriveOptical DriveOSMonitor
1x 160GB SATA Yeah Ubuntu Lucid SSH 
PowerCase
9x 300W HP supplies 5 Threaded rods 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
11x P4s, 3x PDs (all 3.2 to 3.4Ghz) 14x HP mobos Why? 14x 1GB 
Hard DriveOptical DriveOSMonitor
1x 160GB SATA Yeah Ubuntu Lucid SSH 
PowerCase
9x 300W HP supplies 5 Threaded rods 
  hide details  
Reply
post #3 of 37
so... Can I play Crys.... Nevermind.
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
post #4 of 37
Thread Starter 
Quote:
Originally Posted by DuckieHo View Post
so... Can I play Crys.... Nevermind.
Duckie I'm going to smack you! But really if you could figure out how to get it to run over a cluster, be my guest
    
CPUMotherboardGraphicsRAM
11x P4s, 3x PDs (all 3.2 to 3.4Ghz) 14x HP mobos Why? 14x 1GB 
Hard DriveOptical DriveOSMonitor
1x 160GB SATA Yeah Ubuntu Lucid SSH 
PowerCase
9x 300W HP supplies 5 Threaded rods 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
11x P4s, 3x PDs (all 3.2 to 3.4Ghz) 14x HP mobos Why? 14x 1GB 
Hard DriveOptical DriveOSMonitor
1x 160GB SATA Yeah Ubuntu Lucid SSH 
PowerCase
9x 300W HP supplies 5 Threaded rods 
  hide details  
Reply
post #5 of 37
Lol, Crysis only needs 4 cores, and any recent Quadcore is going to rape some Pentium 4s.
Logic
(14 items)
 
  
CPUMotherboardGraphicsRAM
i7 2600K ASRock P67 Extreme4 Gen3 ASUS HD 7950 TWIN FROZR 3GB Corsair Vengeance 16GB DDR3-1600 
Hard DriveCoolingOSMonitor
Intel 330 120GB SSD + Spinpoint F3 1TB Corsair H60 Windows 7 Ultimate x64 Samsung 24.6" P2570HD 
KeyboardPowerCaseMouse
Filco Ninja Majestouch-2 Tenkeyless Antec EarthWatts 650W Corsair 600T Silver Logitech G500 
  hide details  
Reply
Logic
(14 items)
 
  
CPUMotherboardGraphicsRAM
i7 2600K ASRock P67 Extreme4 Gen3 ASUS HD 7950 TWIN FROZR 3GB Corsair Vengeance 16GB DDR3-1600 
Hard DriveCoolingOSMonitor
Intel 330 120GB SSD + Spinpoint F3 1TB Corsair H60 Windows 7 Ultimate x64 Samsung 24.6" P2570HD 
KeyboardPowerCaseMouse
Filco Ninja Majestouch-2 Tenkeyless Antec EarthWatts 650W Corsair 600T Silver Logitech G500 
  hide details  
Reply
post #6 of 37
Quote:
Originally Posted by wuddersup View Post
Lol, Crysis only needs 4 cores, and any recent Quadcore is going to rape some Pentium 4s.
Crysis also needs a GPU and won't run on cluster.... it's joke....
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
post #7 of 37
Quote:
Originally Posted by wuddersup View Post
Lol, Crysis only needs 4 cores, and any recent Quadcore is going to rape some Pentium 4s.
parallelism
Quote:
Originally Posted by DuckieHo View Post
Crysis also needs a GPU and won't run on cluster.... it's joke....
swiftshader?


Thank you very much, x-ravin for this oppertunity!!!
Longshot r4.2
(14 items)
 
  
CPUMotherboardGraphicsRAM
i7 2600k Maximus IV GENE-Z eVGA GTX780 Ti 3GB Sniper 8GB 1600 1.25V 
Hard DriveOptical DriveOSMonitor
M4 256GB + C300 128GB + WD Red 4TB LG Blu-ray RW external Windows 10 U2711 
KeyboardPowerCaseMouse
Ducky Shine 3 AX850 FT03b Logitech G9x 
Mouse PadAudio
Bubbles Pad NR1604 + RB-51 II 
  hide details  
Reply
Longshot r4.2
(14 items)
 
  
CPUMotherboardGraphicsRAM
i7 2600k Maximus IV GENE-Z eVGA GTX780 Ti 3GB Sniper 8GB 1600 1.25V 
Hard DriveOptical DriveOSMonitor
M4 256GB + C300 128GB + WD Red 4TB LG Blu-ray RW external Windows 10 U2711 
KeyboardPowerCaseMouse
Ducky Shine 3 AX850 FT03b Logitech G9x 
Mouse PadAudio
Bubbles Pad NR1604 + RB-51 II 
  hide details  
Reply
post #8 of 37
16 cores, 24 threads... what a beast!!!
You should call some universities up and ask them if they want to rent it for light cluster computation
    
CPUMotherboardGraphicsGraphics
i7 2600k Asrock Extreme 4 Gen 3 Evga GTX560Ti 448 FTW Evga GTX560Ti 448 FTW 
RAMHard DriveHard DriveCooling
Corsair Vengeance DDR3 Crucial M4 SSD 256GB Samsung Spinpoint F3 1TB Alphacool NexXxos 480 
CoolingOSMonitorKeyboard
XSPC Raystorm Windows 7 Ultimate Iiyama ProLite E2273HDS Corsair Vengeance K90 
PowerCaseAudio
Silverstone Strider+ 850 Silverstone Temjin TJ07 Asus Xonar Essence STX 
  hide details  
Reply
    
CPUMotherboardGraphicsGraphics
i7 2600k Asrock Extreme 4 Gen 3 Evga GTX560Ti 448 FTW Evga GTX560Ti 448 FTW 
RAMHard DriveHard DriveCooling
Corsair Vengeance DDR3 Crucial M4 SSD 256GB Samsung Spinpoint F3 1TB Alphacool NexXxos 480 
CoolingOSMonitorKeyboard
XSPC Raystorm Windows 7 Ultimate Iiyama ProLite E2273HDS Corsair Vengeance K90 
PowerCaseAudio
Silverstone Strider+ 850 Silverstone Temjin TJ07 Asus Xonar Essence STX 
  hide details  
Reply
post #9 of 37
Thread Starter 
Quote:
Originally Posted by wuddersup View Post
Lol, Crysis only needs 4 cores, and any recent Quadcore is going to rape some Pentium 4s.
For floating point performance yes, but you maybe surprised with integer performance. 4 P4 3.2's will outperform a lower frequency quad core if you are using a ton of integers. Though you would have to factor in if processes needed a lot of IPC, that would be faster on silicon fabric rather than a LAN of course.


Quote:
Thank you very much, x-ravin for this oppertunity!!!
My pleasure, I figure it's a waste to let it sit idle.


Quote:
16 cores, 24 threads... what a beast!!!
You should call some universities up and ask them if they want to rent it for light cluster computation
Purdue has a huge cluster (>10,000 cores) that they rent out like that. Mine is pretty puny comparatively. I wouldn't ever charge money for it's use because I don't want to have to guarantee uptime. My cluster has a lot of random downtime as I'm always trying to make it better.
    
CPUMotherboardGraphicsRAM
11x P4s, 3x PDs (all 3.2 to 3.4Ghz) 14x HP mobos Why? 14x 1GB 
Hard DriveOptical DriveOSMonitor
1x 160GB SATA Yeah Ubuntu Lucid SSH 
PowerCase
9x 300W HP supplies 5 Threaded rods 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
11x P4s, 3x PDs (all 3.2 to 3.4Ghz) 14x HP mobos Why? 14x 1GB 
Hard DriveOptical DriveOSMonitor
1x 160GB SATA Yeah Ubuntu Lucid SSH 
PowerCase
9x 300W HP supplies 5 Threaded rods 
  hide details  
Reply
post #10 of 37
Quote:
Originally Posted by X-Ravin View Post
For floating point performance yes, but you maybe surprised with integer performance. 4 P4 3.2's will outperform a lower frequency quad core if you are using a ton of integers. Though you would have to factor in if processes needed a lot of IPC, that would be faster on silicon fabric rather than a LAN of course.
Quick question.... what kind of parallel workload today needs interger performance?
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Coding and Programming
Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › Free Cluster Access for OCN!