Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › Tasking for a Clustered System.
New Posts  All Forums:Forum Nav:

Tasking for a Clustered System.

post #1 of 10
Thread Starter 
I have a TON of data that needs to be processed, so I was going to create a cluster of Raspbery Pi 3's to process all of the data. So, the question is ... whats the best approach for telling the 50 or so nodes what to do? I can have them query a DB and pick from items in a table, or I can setup some Master\slave relationship where 1 of the devices tells the others what to do.

I am leaning towards them being independent and querying the DB for their tasks. Add a heartbeat and the nodes identify by their assigned DHCP IP addressed. My issue with this approach is how do you update all the nodes if the server DB IP changes? How can I nicely update all the nodes? I guess thats probably another service\application. So, this will all be on linux, not sure what distro yet... so feel free to toss in your 2 cents.
Zev's Comp
(15 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5-2500K Sandy Bridge 3.3GHz GIGABYTE GA-Z68X-UD3H-B3 LGA 1155 Intel Z68 HDM... GeForce GTX 750 Ti G.SKILL Ripjaws X Series 8GB 
Hard DriveHard DriveHard DrivePower
1TB HDD 64GB SSD (Used for SRT) 500 GB. Antec BP550 Plus 550W Continuous Power ATX12V V... 
Case
COOLER MASTER ELITE 335 RC-335-KKN1-GP Black S... 
  hide details  
Reply
Zev's Comp
(15 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5-2500K Sandy Bridge 3.3GHz GIGABYTE GA-Z68X-UD3H-B3 LGA 1155 Intel Z68 HDM... GeForce GTX 750 Ti G.SKILL Ripjaws X Series 8GB 
Hard DriveHard DriveHard DrivePower
1TB HDD 64GB SSD (Used for SRT) 500 GB. Antec BP550 Plus 550W Continuous Power ATX12V V... 
Case
COOLER MASTER ELITE 335 RC-335-KKN1-GP Black S... 
  hide details  
Reply
post #2 of 10
A single Xeon would be faster, cheaper, and way more boring. But if you are dead set on a cluster for lab purposes, you're gonna want static IP addresses for each node to start. Better do 48 instead of 50 so that you can use a reasonably priced switch for your backbone. No gigabit neccessary, should be cheap enough for a 48 port with gigabit uplink. For that many nodes you're gonna want MPI or Hadoop, or some such. Hadoop should be easier to manage, but the performance loss due to Java will be noticeable in this case. Make sure to use gpu acceleration, as the gpu is actually way more powerful than the CPUs in a raspberry. You're gonna have a metric ton of cables to manage. Better plan ahead. Google raspberry pi cluster for some ideas.
post #3 of 10
It really depends on how you want to .. 'do' it. Kafka would be an excellent way to introduce messaging in the master / slave paradigm. However, I'm not sure how it would scale down to the size of the Pi3's architecture.

Gamester3333 seems to think the performance hit of java on the Pi3 would be extensive, and I don't know from experience if the lack of hardware acceleration or other support would validate his claims, or if he is basing them solely on old anecdotes about the efficiency of java compared to native code (it's now almost unilaterally within 3% or less).

A binary vs character based protocol would suffer a much heavier penalty with character encoding coming in roughly 35% less performant due to encoding and decoding operations in the protocol.

I am not sure if anyone has ever put these kind of tiny systems in a virtual cluster, but one way to manage the networking and other issues would be to use 1-1 mappings of vm's to hosts allowing the VM tool to control the cluster.

If you have a known dataset size, you could always partition the DB id records with modulus to select incremental sets, and paginate through the resultsets via SQL.

Otherwise, you could use Kakfa as stated to be 'the database' and have the messages partitioned via a numbering mechanism in the kafka partition name, and directly associating that with a single node on one node. This could be useful for creating the database, but you could have an unknown number of records and by managing the inserts to each partition the round robin inserts would distribute the load quasi evenly.

This would require an external kafka host on your network, as well as a machine to 'insert' the dataset into the queues, which could be one in the same if it was strong enough... however you may run into issues where network capacity or IO at this service is not sufficient to feed the swarm.

I don't think the Pi3 would be a good option to try to run hadoop on, or any of its services, but this just an anecdotal guess based on memory and other consumption habits. But you most likely could utilize the queue as an extremely effective distribution platform outside of the cluster.
Nightrider
(17 items)
 
Commodore 64
(10 items)
 
 
CPUMotherboardGraphicsRAM
3930k x79 gd45 PLUS GTX Titan Crucial Ballistix Sport VLP  
Hard DriveHard DriveHard DriveCooling
HyperX 3k Intel 320 Seagate Barracuda Swifttech H220 
CoolingCoolingOSOS
Swifttech 220QP Corsair SP120 Windows 8.1 Pro Windows 10 Pro 
OSOSMonitorMonitor
Windows 7 Home Ubuntu 15.4 QNIX 2710 Catleap 2B 
Keyboard
Ducky - Cherry MX Red 
CPUMotherboardGraphicsRAM
3570k DZ77GA - 70K GTX670-DC2-4GD5  MV-3V4G3D/US 
Hard DriveCoolingOSOS
HyperX 3k CM 212 + Win 7 64 ubuntu 
PowerCase
Seventeam 850w modular CS-NT-ZERO-2  
  hide details  
Reply
Nightrider
(17 items)
 
Commodore 64
(10 items)
 
 
CPUMotherboardGraphicsRAM
3930k x79 gd45 PLUS GTX Titan Crucial Ballistix Sport VLP  
Hard DriveHard DriveHard DriveCooling
HyperX 3k Intel 320 Seagate Barracuda Swifttech H220 
CoolingCoolingOSOS
Swifttech 220QP Corsair SP120 Windows 8.1 Pro Windows 10 Pro 
OSOSMonitorMonitor
Windows 7 Home Ubuntu 15.4 QNIX 2710 Catleap 2B 
Keyboard
Ducky - Cherry MX Red 
CPUMotherboardGraphicsRAM
3570k DZ77GA - 70K GTX670-DC2-4GD5  MV-3V4G3D/US 
Hard DriveCoolingOSOS
HyperX 3k CM 212 + Win 7 64 ubuntu 
PowerCase
Seventeam 850w modular CS-NT-ZERO-2  
  hide details  
Reply
post #4 of 10
Thread Starter 
I will be transcoding 500+ TB of video, and I was able to get a build working with codecs that utilize the h264 onboard chip which makes them pretty fast and cheap. Its possible that I can create a PC with multiple GPUS and work that way, but I dont think it will be as fast. I will give it a bit more testing and do a bit more googling just in case.

Since I already have a SQL DB on a server, I probably should just use that. I guess my main issue would then be how to update all of the nodes. So, like in my example above, if the IP for the sql db changes, how do i tell all the nodes to switch over? Since their IPs will be DHCPed, I would probably have a table setup with the last update from them, the IP they are using, versions of my software , config version, and their MAC address as a unique identifier. From there, if the SQL db changes, I will have a list of IPs of all the devices, but is there a nice way to bulk send a file? Perhaps write a seperate application to place in the init.d that would check a server and backup server to see if it needs to update the application or config that runs before the main application. So, if i need to update the device I would then just put a flag in the DB for them to reboot.
Zev's Comp
(15 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5-2500K Sandy Bridge 3.3GHz GIGABYTE GA-Z68X-UD3H-B3 LGA 1155 Intel Z68 HDM... GeForce GTX 750 Ti G.SKILL Ripjaws X Series 8GB 
Hard DriveHard DriveHard DrivePower
1TB HDD 64GB SSD (Used for SRT) 500 GB. Antec BP550 Plus 550W Continuous Power ATX12V V... 
Case
COOLER MASTER ELITE 335 RC-335-KKN1-GP Black S... 
  hide details  
Reply
Zev's Comp
(15 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5-2500K Sandy Bridge 3.3GHz GIGABYTE GA-Z68X-UD3H-B3 LGA 1155 Intel Z68 HDM... GeForce GTX 750 Ti G.SKILL Ripjaws X Series 8GB 
Hard DriveHard DriveHard DrivePower
1TB HDD 64GB SSD (Used for SRT) 500 GB. Antec BP550 Plus 550W Continuous Power ATX12V V... 
Case
COOLER MASTER ELITE 335 RC-335-KKN1-GP Black S... 
  hide details  
Reply
post #5 of 10
Those questions are exactly why I was pointing you to perhaps putting a hypervisor on the Pi3s and having chef / puppet / vagrant manage the creation of the VM's because one of the things that can do is give you access to host and namespace resolution.

Vagrant has solved this particular problem with the hosts plugin, almost completely.

The other option is to just give you DB a known hostname, and put a DNS server in front of the Pi3's so that they resolve the IP, instead of directly access a raw IP. I'm not sure why A. the ip would change on you if the stuff is a local network or B. each node would need to know about each other.
Nightrider
(17 items)
 
Commodore 64
(10 items)
 
 
CPUMotherboardGraphicsRAM
3930k x79 gd45 PLUS GTX Titan Crucial Ballistix Sport VLP  
Hard DriveHard DriveHard DriveCooling
HyperX 3k Intel 320 Seagate Barracuda Swifttech H220 
CoolingCoolingOSOS
Swifttech 220QP Corsair SP120 Windows 8.1 Pro Windows 10 Pro 
OSOSMonitorMonitor
Windows 7 Home Ubuntu 15.4 QNIX 2710 Catleap 2B 
Keyboard
Ducky - Cherry MX Red 
CPUMotherboardGraphicsRAM
3570k DZ77GA - 70K GTX670-DC2-4GD5  MV-3V4G3D/US 
Hard DriveCoolingOSOS
HyperX 3k CM 212 + Win 7 64 ubuntu 
PowerCase
Seventeam 850w modular CS-NT-ZERO-2  
  hide details  
Reply
Nightrider
(17 items)
 
Commodore 64
(10 items)
 
 
CPUMotherboardGraphicsRAM
3930k x79 gd45 PLUS GTX Titan Crucial Ballistix Sport VLP  
Hard DriveHard DriveHard DriveCooling
HyperX 3k Intel 320 Seagate Barracuda Swifttech H220 
CoolingCoolingOSOS
Swifttech 220QP Corsair SP120 Windows 8.1 Pro Windows 10 Pro 
OSOSMonitorMonitor
Windows 7 Home Ubuntu 15.4 QNIX 2710 Catleap 2B 
Keyboard
Ducky - Cherry MX Red 
CPUMotherboardGraphicsRAM
3570k DZ77GA - 70K GTX670-DC2-4GD5  MV-3V4G3D/US 
Hard DriveCoolingOSOS
HyperX 3k CM 212 + Win 7 64 ubuntu 
PowerCase
Seventeam 850w modular CS-NT-ZERO-2  
  hide details  
Reply
post #6 of 10
Thread Starter 
Each node would not need to know about each other, but the reason why the IP could change is more along the lines of targeting a different DB (same schema), but on a different server.

I have always had bad luck with Virtualization where I always run into a conflict with something. Some API no longer works or something. I have not tried any virtualization on a Pi, but I will give it a shot. I have not used Chef before, but one of my friends has, so I will bug him about it.
Zev's Comp
(15 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5-2500K Sandy Bridge 3.3GHz GIGABYTE GA-Z68X-UD3H-B3 LGA 1155 Intel Z68 HDM... GeForce GTX 750 Ti G.SKILL Ripjaws X Series 8GB 
Hard DriveHard DriveHard DrivePower
1TB HDD 64GB SSD (Used for SRT) 500 GB. Antec BP550 Plus 550W Continuous Power ATX12V V... 
Case
COOLER MASTER ELITE 335 RC-335-KKN1-GP Black S... 
  hide details  
Reply
Zev's Comp
(15 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5-2500K Sandy Bridge 3.3GHz GIGABYTE GA-Z68X-UD3H-B3 LGA 1155 Intel Z68 HDM... GeForce GTX 750 Ti G.SKILL Ripjaws X Series 8GB 
Hard DriveHard DriveHard DrivePower
1TB HDD 64GB SSD (Used for SRT) 500 GB. Antec BP550 Plus 550W Continuous Power ATX12V V... 
Case
COOLER MASTER ELITE 335 RC-335-KKN1-GP Black S... 
  hide details  
Reply
post #7 of 10
You might be wading into uncharted territory, but short of creating a single linux image, which you remotely install headlessly with your own local DNS configuration set up, I'm not sure exactly how this wouldn't be a nightmare to manage. You're setting up 48 + a control server boxes.. this type of workload is why enterprise management software exists.

It doesn't matter that your boxes are all relatively low powered, they all have the same network provisioning concerns.

You could simply write a bash script to create the installation process in 48 different threads on a host matching.. naming everything node[1-48] and giving its own domain for DNS resolution, in that way you could push this resolution (with static IP's) to the known hosts and /etc/hosts files to speed up the process.

You are going to have an issue still with the database, because you don't have a C&C server for the nodes in regards to the database you want to access. However you could simply have the script parse the database information + credentials from the filesystem, and using a similar script to the installation distribute this script to all the nodes.

If the virtualization doesn't immediately work, you might consider dropping it as I don't know if the Pi3 comes with any accelerators like VT-x or AMD-V.

In that case, you're gonna be stuck with rolling your own small version of a resource pool management in the fashion described above.
Nightrider
(17 items)
 
Commodore 64
(10 items)
 
 
CPUMotherboardGraphicsRAM
3930k x79 gd45 PLUS GTX Titan Crucial Ballistix Sport VLP  
Hard DriveHard DriveHard DriveCooling
HyperX 3k Intel 320 Seagate Barracuda Swifttech H220 
CoolingCoolingOSOS
Swifttech 220QP Corsair SP120 Windows 8.1 Pro Windows 10 Pro 
OSOSMonitorMonitor
Windows 7 Home Ubuntu 15.4 QNIX 2710 Catleap 2B 
Keyboard
Ducky - Cherry MX Red 
CPUMotherboardGraphicsRAM
3570k DZ77GA - 70K GTX670-DC2-4GD5  MV-3V4G3D/US 
Hard DriveCoolingOSOS
HyperX 3k CM 212 + Win 7 64 ubuntu 
PowerCase
Seventeam 850w modular CS-NT-ZERO-2  
  hide details  
Reply
Nightrider
(17 items)
 
Commodore 64
(10 items)
 
 
CPUMotherboardGraphicsRAM
3930k x79 gd45 PLUS GTX Titan Crucial Ballistix Sport VLP  
Hard DriveHard DriveHard DriveCooling
HyperX 3k Intel 320 Seagate Barracuda Swifttech H220 
CoolingCoolingOSOS
Swifttech 220QP Corsair SP120 Windows 8.1 Pro Windows 10 Pro 
OSOSMonitorMonitor
Windows 7 Home Ubuntu 15.4 QNIX 2710 Catleap 2B 
Keyboard
Ducky - Cherry MX Red 
CPUMotherboardGraphicsRAM
3570k DZ77GA - 70K GTX670-DC2-4GD5  MV-3V4G3D/US 
Hard DriveCoolingOSOS
HyperX 3k CM 212 + Win 7 64 ubuntu 
PowerCase
Seventeam 850w modular CS-NT-ZERO-2  
  hide details  
Reply
post #8 of 10
Thread Starter 
I did some more encoding with the Pi3, and the encoder that used the onboard h264 chip was a bit finicky. I could encode to 720p no problem, but if i tried to encode a file to 1080p it would stall. I adjusted the dedicated memory to no avail. CPU would work fine though... granted it was 6x slower. One of my co-workers decided to try out Azures Batch process and it was able to encode pretty quick while still being cheap, so we will probably go that direction. It should be easier to run, maintain, and scale as well.
Zev's Comp
(15 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5-2500K Sandy Bridge 3.3GHz GIGABYTE GA-Z68X-UD3H-B3 LGA 1155 Intel Z68 HDM... GeForce GTX 750 Ti G.SKILL Ripjaws X Series 8GB 
Hard DriveHard DriveHard DrivePower
1TB HDD 64GB SSD (Used for SRT) 500 GB. Antec BP550 Plus 550W Continuous Power ATX12V V... 
Case
COOLER MASTER ELITE 335 RC-335-KKN1-GP Black S... 
  hide details  
Reply
Zev's Comp
(15 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5-2500K Sandy Bridge 3.3GHz GIGABYTE GA-Z68X-UD3H-B3 LGA 1155 Intel Z68 HDM... GeForce GTX 750 Ti G.SKILL Ripjaws X Series 8GB 
Hard DriveHard DriveHard DrivePower
1TB HDD 64GB SSD (Used for SRT) 500 GB. Antec BP550 Plus 550W Continuous Power ATX12V V... 
Case
COOLER MASTER ELITE 335 RC-335-KKN1-GP Black S... 
  hide details  
Reply
post #9 of 10
Yea, it was always kinda likely you were going to run into issues with nodes at that memory capacity, and age of processors. A docker swarm of small runlets or similar would probably be the cheapest way if you aren't talking about doing it yourself on your own hardware.

I avoid M$ like the plague, so I'm not sure about azures performance and cost.
Nightrider
(17 items)
 
Commodore 64
(10 items)
 
 
CPUMotherboardGraphicsRAM
3930k x79 gd45 PLUS GTX Titan Crucial Ballistix Sport VLP  
Hard DriveHard DriveHard DriveCooling
HyperX 3k Intel 320 Seagate Barracuda Swifttech H220 
CoolingCoolingOSOS
Swifttech 220QP Corsair SP120 Windows 8.1 Pro Windows 10 Pro 
OSOSMonitorMonitor
Windows 7 Home Ubuntu 15.4 QNIX 2710 Catleap 2B 
Keyboard
Ducky - Cherry MX Red 
CPUMotherboardGraphicsRAM
3570k DZ77GA - 70K GTX670-DC2-4GD5  MV-3V4G3D/US 
Hard DriveCoolingOSOS
HyperX 3k CM 212 + Win 7 64 ubuntu 
PowerCase
Seventeam 850w modular CS-NT-ZERO-2  
  hide details  
Reply
Nightrider
(17 items)
 
Commodore 64
(10 items)
 
 
CPUMotherboardGraphicsRAM
3930k x79 gd45 PLUS GTX Titan Crucial Ballistix Sport VLP  
Hard DriveHard DriveHard DriveCooling
HyperX 3k Intel 320 Seagate Barracuda Swifttech H220 
CoolingCoolingOSOS
Swifttech 220QP Corsair SP120 Windows 8.1 Pro Windows 10 Pro 
OSOSMonitorMonitor
Windows 7 Home Ubuntu 15.4 QNIX 2710 Catleap 2B 
Keyboard
Ducky - Cherry MX Red 
CPUMotherboardGraphicsRAM
3570k DZ77GA - 70K GTX670-DC2-4GD5  MV-3V4G3D/US 
Hard DriveCoolingOSOS
HyperX 3k CM 212 + Win 7 64 ubuntu 
PowerCase
Seventeam 850w modular CS-NT-ZERO-2  
  hide details  
Reply
post #10 of 10
Thread Starter 
My company has an agreement with them so I tend to get awesome support. As for pricing, their Media Encoding service is expensive, but not many people doing that as is. Their VM pricing is on par with everyone else. As for manageability and administration, Amazon > Microsoft > Google. The batch processing is pretty cool in that it basically simplifies the whole creating VM instances, bringing them up and taking them down for specific tasks. I personally have not used it, but my coworkers finished up a project with it, and after they got trough the hurdle of understanding the nuances of it, they were really pleased with it.
Zev's Comp
(15 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5-2500K Sandy Bridge 3.3GHz GIGABYTE GA-Z68X-UD3H-B3 LGA 1155 Intel Z68 HDM... GeForce GTX 750 Ti G.SKILL Ripjaws X Series 8GB 
Hard DriveHard DriveHard DrivePower
1TB HDD 64GB SSD (Used for SRT) 500 GB. Antec BP550 Plus 550W Continuous Power ATX12V V... 
Case
COOLER MASTER ELITE 335 RC-335-KKN1-GP Black S... 
  hide details  
Reply
Zev's Comp
(15 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5-2500K Sandy Bridge 3.3GHz GIGABYTE GA-Z68X-UD3H-B3 LGA 1155 Intel Z68 HDM... GeForce GTX 750 Ti G.SKILL Ripjaws X Series 8GB 
Hard DriveHard DriveHard DrivePower
1TB HDD 64GB SSD (Used for SRT) 500 GB. Antec BP550 Plus 550W Continuous Power ATX12V V... 
Case
COOLER MASTER ELITE 335 RC-335-KKN1-GP Black S... 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Coding and Programming
Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › Tasking for a Clustered System.