Overclock.net › Forums › Overclockers Care › Overclock.net BOINC Team › Project "Headless Linux CLI Multiple GPU Boinc Server" - Ubuntu Server 12.04.4/14.04.1 64bit - Using GPU's from GeForce GT610/GT640/GTX750ti/+ to crunch data.
New Posts  All Forums:Forum Nav:

Project "Headless Linux CLI Multiple GPU Boinc Server" - Ubuntu Server 12.04.4/14.04.1 64bit - Using GPU's from GeForce GT610/GT640/GTX750ti/+ to crunch data. - Page 12

post #111 of 343
Thread Starter 
Hi Guys,


I'm kind of "crashing" my own thread here, but I have an issue I'm pretty sure you can help me solve!

For the next "generation" of "Headless Linux Multiple GPU Boinc Servers" I've just bought a graphic card to test. Since there's no news regarding Low Profile cards and that I haven't found any Low Profile cards which is larger/better than Asus GT640, I decided that the next "round" of tests/building affordable crunchers, will be with normal size cards. And therefore I bought this Asus GTX 770 to test on a windows rig, just to see how it handless heat, how it does compared to the other cards and how much power it eats. If it's possible at all, to get a PSU that can supply 4 of those cards.
But, there's an issue. I'm really not very familiar with these "larger" graphic cards and especially not the newer ones. So I'm having trouble finding information on how to install these. Not on the mobo of course, but regarding the power connectors on the graphic card. There's 2 connectors, 1x 6 pin and 1x 8 pin and according to what I've read and learned, I must plug in PCIe power in both of them. Both the 6 pin and the 8 pin!?!? I'm using a Corsair 750 watts PSU which has separate wires/connectors and I have 2x wires with a 8 pin connector in one end and 2x 6 + 2 pin in the other end. Those 2x 6 + 2 pin connectors I can see is made so that you can connect them to both a 6 pin and a 8 pin connector.
I just would like, very much to get some kind of a confirmation on this, and that I'm suppose to connect power to both connectors on the card !? It says so here, but I would just hate to make a big mistake, and ruin a card which coursed an empty fridge in this month wink.gif

Here's the connectors in question:





Hope to hear from you soon wink.gif


***************************************************
No need for assistance - got it running now wink.gif
***************************************************
Edited by DanHansenDK - 10/2/14 at 7:44am
post #112 of 343
Yep, gotta plug in both connectors!

And dont worry, it wont run at full speed or wont boot if you dont plug in the pci-e power connectors, so you wont ruin the card.

As for my 750 comment, I was wondering if you were going to start replacing those 640's with something like a GTX 750. They have low-profile variants but they are dual slot.
Edited by Finrond - 10/2/14 at 10:57am
i5
(16 items)
 
  
CPUMotherboardGraphicsGraphics
i5 2500k MSI z77a-g43 MSI RX 480 GAMING X 4G MSI RX 480 GAMING X 4G 
RAMHard DriveHard DriveHard Drive
G.Skill Ripjaw X series 16GB OCZ agility 3 90GB Sandisk Ultra II 240GB Seagate ST4000DM000 
Optical DriveOSCase
None Windows 10 Pro Thor 
  hide details  
Reply
i5
(16 items)
 
  
CPUMotherboardGraphicsGraphics
i5 2500k MSI z77a-g43 MSI RX 480 GAMING X 4G MSI RX 480 GAMING X 4G 
RAMHard DriveHard DriveHard Drive
G.Skill Ripjaw X series 16GB OCZ agility 3 90GB Sandisk Ultra II 240GB Seagate ST4000DM000 
Optical DriveOSCase
None Windows 10 Pro Thor 
  hide details  
Reply
post #113 of 343
Thread Starter 
Hi Finrond,

Quote:
As for my 750 comment, I was wondering if you were going to start replacing those 640's with something like a GTX 750. They have low-profile variants but they are dual slot.
SAY WHAT!?!? It can't be true!?!? I've been looking and searching for a pretty g.. d... long time now!! Haven't managed to find anything which even compares to the GT640 from Asus !!! It endures heat very well (95 degrees C.) and is very very stable. Well I have some trouble, the system is not perfect, yet, but this is due to OS issues or at least I'm pretty sure it is. Please show me these cards!! If it's possible to get a low profile edition we can make a very special cruncher come together wink.gif The problem will be the 2U PSU, but that's just a thing to fix. No big deal wink.gif
My head is already spinning. Thoughts flying away here!!! Regarding the dual slot issue, do enyone know the space which is between the sockets?? If they will fit into a 4x SLI board, we can only use 2 of course. But they will still be much faster than the 4x GT640 cards together..

That was great news!! I hope Asus has made a version of these LP cards wink.gif

Looking forward to hear from you...


I've noticed that a SETI job done by the GTX770 only takes about 3-4 minutes, some even less. The same job (same size) takes 1 hour and 30+ minutes for one of the CPU cores to do. I'm not sure about the GT640 yet. But I guess they are faster than one CPU core (i5-4670K - i5-4690K)

Here's a few samples:

I've currently joined Asteroid@Home & SETI@Home and in this sample this is the samples shown. And it's the GTX770 we are looking at right now, not GT640 from test system 1-4.


The two Asteroid jobs is about 1.380.000 Gflops each. One is done by GPU/Cuda55. The other by one of the CPU cores on i5-4690K. As you can see both jobs is about 1.380.000 Gflops. The GPU/Cuda55 does this job in 22 minutes. The CPU core takes 1 hour and about 25 minutes to do the "same" job. With that in mind, let's look at the next two jobs, this time SETI.



These two jobs is smaller, but the difference between the time it takes the CPU cores to do the jobs is not that big. But, when it comes to the GPU there's a huge difference! This GPU does the "same" job as the CPU core in 7 minutes!!! I know that it's estimated jobsizes etc. etc. That some use Cuda55, some Cuda32/Cuda42 and that there's a lot of other reasons, that this is difficult to compare, but anyway. My G..!! 7 minutes!!! And I've even seen jobs been "chewed" in 3 minutes thumb.gif




OK, I'm looking forward to hear some noise, because I'm not no professor on these matters, that's for sure. If anyone knows about these things, don't hesitate to "teach" me a lesson biggrin.gif


BTW, system 5 "blenheim" I should be able to put together on thursday or friday!! D... ! just remembered one thing I forgot! Forgot the AeroCool Fan Control/Sensor!?!?
Is there any of you guys who uses these fan controllers? I need to say that I've found a possible "weakness" regarding the AeroCool X-Vision Fan-Controller. Sadly the display gets dimmer along the time. When used 24/7 it seems that there's an issue with the display. I guess it's a component which can't take the heat (heat from its own components).
So, if you know a fan-controller with temp. sensors, capable of controlling 5 fans, please let me know. One you have used for a longer period of time wink.gif

.
Edited by DanHansenDK - 10/4/14 at 4:36pm
post #114 of 343
Thread Starter 
Hello wink.gif


And then something like this happens! Suddenly it takes GTX770 almost 4 hours to do a single Asteroid job. Same size as before. And the CPU core does this in about 1 hour 50 minutes. So in this case the GPU is twice as slow as a CPU core. I just don't get this. There must be an error in the readings or those job sizes are very "estimated". Can't see what else can be wrong. It looks like, it's no good idea to crunch Asteroid jobs using GPU's in a windows environment. In Linux I see no such issues!

What say you guys smile.gif

GPU GTX770:


CPU core (i5-4690K):




Hi Finrond,
Quote:
As for my 750 comment, I was wondering if you were going to start replacing those 640's with something like a GTX 750. They have low-profile variants but they are dual slot.
SAY WHAT!?!? It can't be true!?!? I've been looking and searching for a pretty g.. d... long time now!! Haven't managed to find anything which even compares to the GT640 from Asus !!! It endures heat very well (95 degrees C.) and is very very stable. Well I have some trouble, the system is not perfect, yet, but this is due to OS issues or at least I'm pretty sure it is. Please show me these cards!! If it's possible to get a low profile edition we can make a very special cruncher come together. The problem will be the 2U PSU, but that's just a thing to fix. No big deal.
My head is already spinning. Thoughts flying away here!!! Regarding the dual slot issue, do enyone know the space which is between the sockets?? If they will fit into a 4x SLI board, we can only use 2 of course. But they will still be much faster than the 4x GT640 cards together..

That was great news!! I hope Asus has made a version of these LP cards wink.gif

Looking forward to hear from you... thumb.gif


.
Edited by DanHansenDK - 10/5/14 at 11:28am
post #115 of 343
There is only 1 low profile 750 that I can find at the moment

http://www.newegg.com/Product/Product.aspx?Item=9SIA5751ZT1529

Seems a lil pricey to me, but hey, low profile!
i5
(16 items)
 
  
CPUMotherboardGraphicsGraphics
i5 2500k MSI z77a-g43 MSI RX 480 GAMING X 4G MSI RX 480 GAMING X 4G 
RAMHard DriveHard DriveHard Drive
G.Skill Ripjaw X series 16GB OCZ agility 3 90GB Sandisk Ultra II 240GB Seagate ST4000DM000 
Optical DriveOSCase
None Windows 10 Pro Thor 
  hide details  
Reply
i5
(16 items)
 
  
CPUMotherboardGraphicsGraphics
i5 2500k MSI z77a-g43 MSI RX 480 GAMING X 4G MSI RX 480 GAMING X 4G 
RAMHard DriveHard DriveHard Drive
G.Skill Ripjaw X series 16GB OCZ agility 3 90GB Sandisk Ultra II 240GB Seagate ST4000DM000 
Optical DriveOSCase
None Windows 10 Pro Thor 
  hide details  
Reply
post #116 of 343
Here's a few more:

GTX 750
Galaxy
Gigabyte

GTX 750ti
KFA2
Gigabyte
post #117 of 343
Thread Starter 
Hi Finrond & Tictoc,

Thanks a lot guys!! Looks promising!
But, how come the GTX770 card from Zotac only have 500+ cuda cores? Is this because it's a Low Profile card perhaps???

Looking for a card which endures heat the same way Asus GT640 Low Profile does it wink.gif Must look a little better into the links you send me. Thanks for that my friends!

BTW! Got the hardware for the new test system14 days ago, but haven't had the time to put it together wink.gif University stuff! Trying hard to be "accepted" (what's the word?) into this final bit of education, which I really need!! I'm so G.. d.... nervous!! Well, what I wanted to say was, that I'm putting this 5 test system together right now! And this will be with the new 4x SLI mobo from ASRock with the Z97 chipset (Devils canyon) And we are going to deploy the "WatchDog*-scripts" on this system as well. No hardware fan & temperature controller on this system! We learned that there's to much difference between the onboard/sensors data/temp. and the output from these hardware fan/temp. controllers (in my case it was AeroCool X-vision). I rather preferred to have a hardware controller just to have the analogue output. An visible controller with alerts/sounds and blinking displays etc.
Anyway, there's some advantages using the mobo t control the fans. This mobo has 4 extra fan controls/connectors which works with onboard temperature sensors. This means we are able to control the fans with my "WatchDog*-scripts" and have the temperature of the system/onboard temperature "watched". This way it all comes together, CPU temperature control, warning, logging and safety shutdowns, GPU temperature control, warning, logging and safety shutdowns, Chassis temperature control, warning, logging and safety shutdowns, (HD temperature control, warning, logging and safety shutdowns - not in my case because I'm using SSD's) I'll make the script accessible of course, for those of you who use standard drives wink.gif

Let's go!!!
Edited by DanHansenDK - 10/27/14 at 4:23am
post #118 of 343
Thread Starter 
Here's the hardware specs for test system 5:

1x ASRock OC Formula Z97
Intel i5-4690K
Industrial 2U Cooler from JAC
Industrial 2U PSU ATX300W
4x Industrial Fan's Pabst
1x 4Gb Kingston HyperX Genesis X2 Grey S.
Asus GT640-1GD5-L PCIe 2.0 x16
Asus GT640-1GD5-L PCIe 2.0 x16
Asus GT640-1GD5-L PCIe 2.0 x16
Asus GT640-1GD5-L PCIe 2.0 x16
GPU specs: http://www.asus.com/Graphics_Cards/GT6401GD5L/specifications/


YES!!! We are ready thumb.gif




Power consumption all together for this system running at 100%. (1 x CPU at 100% & 4 GPU's at 100%):
Watts: 217 thumb.gif
Here's a tool you can use when trying to find the right size PSU. It's not completely hopeless! It's actually OK. It works "on the safe site", that's for sure, but you can use it to be guided in the right direction. Click here



Stuff's getting better:
1. This board is smaller! It's a standard ATX size and not as large as the Z87 edition. I guess it's because of the "missing" internal LCD display (which to me was a total flop). It may also be due to the fact that this mobo hasn't got the "water cooling kit" on the onboard GPU. But we don't need it, so no big deal biggrin.gif

Issues still to be solved:
1. Implementation of my "WatchDog* shell-scripts"
2. Ubuntu Server 14.04.1 issues (crashing&rebooting) Temperature or software issues??
3. Mobo Chassis Fan- & Temperature control.
4. 2U Industrial CPU Cooler bracket issue!

Links to the steps, which is not going to be changed. The way we install the fans, makes room for air (breathing holes) etc. etc.:
http://www.overclock.net/t/1467918/ubuntu-server-12-04-4-64bit-boinc-using-gpu-from-geforce-gt610-to-crunch-data/90#post_22734366
http://www.overclock.net/t/1467918/ubuntu-server-12-04-4-64bit-boinc-using-gpu-from-geforce-gt610-to-crunch-data/90#post_22741815









STATUS 28.10.14 09:15

Great news!! Using this new mobo has unintentionally solved one of the issues from earlier on! Remember the "bracket" for the "2U Industrial CPU Cooler" !?!? I had to cut, grind and remove a lot of the material, to make room for the subsided components or component pins!! This is no longer an issue! It's PERFECT!! I hated to grind the bracket and make small metal spurs*. It's just not the best stuff to have in this kind of an environment wink.gif

Here's how it looked on one of the earlier mobo's:







As you can see below, highligthed in red boxes, the component pins are not even close to the bracket! Happy days!! I have high hopes for this mobo, I really do! I really really hope that the 4 GPU's wouldn't course any problems! And that the "crash & reboot issue" from test system 4, is a software based problem!?!?






Allrighty-then wink.gif Let's continue this...
Edited by DanHansenDK - 10/28/14 at 9:26am
post #119 of 343
Thread Starter 
*****A question for you guys *****

Linux FileServer with RAID1 - double backup to 2 extra disks:
I've been building a Linux FileServer with RAID1 (Ubuntu Software RAID) and double backup function to 2 extra disks (NTFS & EXT4). This way backups are readable from windows boxes and Linux as well. All kind of system monitoring and warning systems. Hot-swap bays (green box) for RAID'ed disks, quick fix of a degraded array wink.gif
If 1 disk "dies" you'll be warned about it. Then just plug in a new disk and the system gets "updated" System runs even though 1 disk is dead. In case of a double breakdown, the data will not be accessible of course, but data will be on the 2 backup disks. Both in NTFS and EXT4 file-system, so that data is accessible right away wink.gif
Total step-by-step ToDo, from clean disks to system ready! Is this maybe something anybody inhere could use? It's more a ToDo and issue solving than a graphical "how to do it" !?!? Just let me know if someone needs a thing like this wink.gif





Edited by DanHansenDK - 10/28/14 at 5:00am
post #120 of 343
Thread Starter 
STATUS 28.10.14 1:24pm


Issues still to be solved:
1. Implementation of my "WatchDog* shell-scripts"
2. Ubuntu Server 14.04.1 issues (crashing&rebooting) Temperature or software issues??
3. Mobo Chassis Fan- & Temperature control --> x4 Papst Industrial Fans
4. 2U Industrial CPU Cooler bracket issue! [SOLVED]
5. System mail & warnings --> only SMTP! [SOLVED]
6. Front-filter! A swap-able filter in front of the 2U chassis! Replace bolt/screw, to large. Stops the Rack chassis/from closing properly.
7. Rack-box Fan expansion!? Install another 2 large 220volts fans either in the top or in the bottom!!
8. Make a bracket for the 2U industrial PSU. (If you don't want to make one yourself, then write me and I'll send you one for free. Details later on!)


STATUS 28.10.14 3:36pm
Task 6. Front-filter! These 2U Server Chassis are always "full of holes". That's the way they are designed! To have as much air as possible flow in the chassis, these 2U cases are full of ventilation holes. To keep a steady flow of air in the system, you'll place all fans in one direction. Usually you'll then install some kind of a filter in the front of the case. This wonderful 2U case has a special bracket in the front which can be fitted with a filter. Sadly there's not room for the bracket, because I'm using a 60cm Rack box and the case is 55cm. Not much space to work with wink.gif I'll find a way to mount a new bolt or I'll just use a screw to mount it. Then I'll just have to use a screwdriver when changing the filter, no big deal! We just have to be sure, that the airflow will keep on be as good as it is right now. It will be a problem, if there's not "access" to enough air. Then it wouldn't work, no matter how fantastic fans you install!! wink.gif




STATUS 28.10.14 4:29pm
Fitting the 4 Asus GT640 graphic cards with the low profile bracket! BTW! Asus doesn't deliver your "low profile" graphic card with the brackets needed for using the card as a low profile card!! (Red box). This you'll have to order by itself. If only you were told about this, it wouldn't have been an issue. But when you'll have to open the box first, to know about this "missing" part, then it's a problem! I hate that kind of sales tricks. Because, that's what it is! Both Zotac ans MSI version of the GT640/610 comes along these brackets!




Finish fitting stuff....
Finish mounting CPU, memory and 2U CPU cooler on to the mobo! And please notice we doesn't need to grind and cut into the 2U CPU cooler bracket, which is a great thing! Hate to use powertools when putting computers together biggrin.gif




Fitting the Fans from Papst....
Here we are going to fit the special industrial fans from Papst. For further details, look here please: Mounting the new fan's in the 2U chassis/case




STATUS 5:02
I'm looking so much forward to building this 5 system, because I think we've found the right mobo, the right set up and the right pieces! I'm really looking forward to implimenting my "WatchDog*-scripts" and make the chassis fans run using the 4 onboard chassis fanconnectors. It's perfect. 4 chassis fan connectors on this mobo and 4 is what we need!
Let's go! thumb.gif
Edited by DanHansenDK - 10/28/14 at 9:11am
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Overclock.net BOINC Team
Overclock.net › Forums › Overclockers Care › Overclock.net BOINC Team › Project "Headless Linux CLI Multiple GPU Boinc Server" - Ubuntu Server 12.04.4/14.04.1 64bit - Using GPU's from GeForce GT610/GT640/GTX750ti/+ to crunch data.