Originally Posted by SHNS0;12277695
Something like that.
Or, 4 of something like this together:
Xeons use quad channel right? That means You need to find a block for 4 dimms and get 4 of em. But that would be kind of expensive, I think.
It is running quad channel memory but I actually have 16 FB-DIMMs installed (4 GB each).
I did some more Internet research into overclocking this system and in one of the threads found an interesting FB-DIMM water cooling setup. You can check it out here:http://forums.overclockers.com.au/showthread.php?t=714778
I also found several threads where SkullTrail owners overclocked their 5400 series systems. It looks like the e5420 processors are good for about 3.2-3.5 GHz and they like to run at stock voltage levels. My 1333 MHz memory is likely my weakest link for overclocking.
So far I've cranked the base clock up to 387.2 MHz which bumps the CPU up to 2.9 GHz. It is still 100% stable running Prime95 on all 8 cores.
According to HWMonitor the (stabilized) temperatures haven't moved much either (CPUs are cooler if anything), but the fans are definitely a bit louder.
One thing that did concern me a little was starting with an idling system, when I ran Prime95 on all 8 cores, the CPUs climbed up to 68°C before the fans start to rev up.
Once the fans rev up and everything stabilizes, the CPUs are now down to a reasonable 60°C.
The way it actually went, the fans started screaming when the CPUs got to 68°C and the CPU temps very quickly dropped to about 57°C. After about 15 seconds the fans slowed down to a more reasonable level and the CPU temperatures slowly climbed back to 60°C and stayed there. The fans are still definitely louder than at stock speeds. The rear FB-DIMM fans (the stock ones, not the ones I added) are spinning at 3160 rpm and the internal case fans are all spinning at 2188 rpm. The CPU fans are at 2250 rpm. At stock speeds most of the fans don't go much over 2000 rpm ever.
68°C is pushing it a bit since thermal throttling starts to kick in at anything over 67°C (although I didn't see CPU-Z reporting any drop in clock rate).
The memory temperatures are still OK. High by most of your standards in the low 90°C range, but totally stock (without the extra fans) they run in the 110-115°C and I've seen them get as high as 120°C in the summer. Without the overclock, but with my extra fans the FB-DIMMs typically don't go above 90°C, even in the summer.
I'm guessing I'll be able to push this system to at least 3.0 GHz (400 MHz base clock), which is the stock speed for the e5450 processors (the e5450's are identical to the e5420's I have except they have a 400 MHz base clock instead of 333 MHz).
EDIT: Correction-the e5450 runs a 1333 FSB with a 9x multiplier. I'm actually currently running close to the stock speeds for the X5472 which runs at 3.0GHz with a 1600 FSB and a 7.5 multiplier (the e5420's that I have run stock at 2.5 GHz/1333FSB with a 7.5 multiplier).
Prime95 has been running on all 8 cores for about an hour at 2.9 GHz so far. The CPU temperatures for some wierd reason are actually running cooler at 2.9GHz than at 2.5GHz (they typically run just under or at 65°C at 2.5GHz).
BTW, I've searched the web as much as I know how and I have not been able to find a single other person who has overclocked the motherboard I have (Tyan Tempest i5400PW otherwise known as the Tyan S5397).
I'm surprised to be honest. With all of the interest in the Skull Trail platform (there is quite a bit if you search), I'm surprised no one else decided to build a system using Tyan's top of the line motherboard. There are many discussions where people "think about" going with this board, but they always end up going with Intel's actual Skull Trail or Asus's equivalent instead for their 8-core gaming rigs.
Unlike a typical "server" motherboard, this particular board includes dual PCIe x16 slots and supports dual ATI video cards in Crossfire mode. With the quad channel memory the memory bandwidth was pretty good in its day (not as good as Intel's i5/i7 quick path interface though). It also has a decent PCIe/PCI slot layout that allows a high end audio card to be installed (such as the Asus Xonar DX http://www.asus.com/product.aspx?P_ID=R3E8hhLCEgwyAOt9
that I went with).
The main reason I went with this board was the 16 memory slots so I could get my 64 GB without spending a fortune (8GB FB-DIMMS cost a fortune, but 4GB ones are relatively cheap). I also wanted dual video card support (using high bandwidth PCIe 2.0 x16 slots) so I could have both a decent gaming and decent CAD video card installed. I also liked the fact that it has a decent RAID controller built in to support the 8 hot swappable drive bays in my case (I get a sustained read and write rate of above 450 MB/s over the entire 4-drive RAID 0 "short stroked" array of 500 GB single platter Seagate drives - which is better than most solid state drives). Finally the onboard dual gigabit NIC works well with my Cisco SLM2008 switch using LACP teaming for 2Gbps throughput (our P55A-UD4P based system is also running a dual gigabit LACP team to the same switch and the dual RAID 0 drives provide over 250 MB/s throughput so we can actually take advantage of the extra performance - files transfer extremely quickly).
Oh well, I guess I'm on my own... Prime95 has been running for several hours now. Time to shut it down and see how well GTA IV plays with the CPUs at 2.9 GHz...
GTA IV was a little better, but still not fantastic at 2.9GHz.
I turned down the view distance and detail settings to about 70% and the frame rate looked to be about 25-40 depending on how much complexity was in the view. The system was perfectly stable during both several hours of Prime95 on all 8 cores and during a couple of hours of GTA IV.
I've now cranked up the base clock to 400 MHz. My CPU is now running at stock speeds for the X5472 or 3.0 GHz with a 1600 MHz FSB (stock for my e5420(s) is 2.5 GHz with a 1333 MHz FSB) and my memory is now running at stock settings for 1600 MHz RAM (stock is 1333 MHz). This is a modest 20% overclock for both the CPU and RAM, but the motherboard and chipset are just running at top rated speeds (not overclocked).
Prime95 has so far been running for about 15 minute as I type this. This time the fans started to kick in when the CPUs got to 63°C. Again the fans overcompensated at first, but not as badly this time. CPU0 is currently 57°C and CPU1 is currently 59°C.
One of the FB-DIMMs has just hit 101°C (3 others are at 95-97°C), but the rest are all in the high 70°C to 85°C range. I find it greatly depends on which region of memory that Prime95 is running in as to which FB-DIMM(s) become hot. If I stop it, run something else, and then start Prime95 again, it will typically be a different FB-DIMM that gets the hottest. Prime95 tends to be extremely hard on four FB-DIMMs due to how it pounds on only 4 FB-DIMMs at a time. When running Ansys and using all 64 GB the FB-DIMMs tend to spread out the heat amongst them all. Prime95 just started the 8k test and all of the FB-DIMMs dropped into the 70°C to 90°C range.
Without additional cooling for the FB-DIMMs this is about the fastest I'm comfortable with. The system is still Prime95 stable (so far) - about 30 minutes so far. For some reason the CPUs are running cooler at 3.0GHz than they did at 2.9GHz, but the fans are also running a little faster (2500 rpm is the slowest fan, while the stock FB-DIMM fans are spinning at 3600 rpm). To be honest the fan noise is getting to the point where I would not want to sit beside the workstation for hours on end, but the only thing that will push it this hard is solving a large finite element model (not running any game). I'm sure it will be quieter playing GTA IV.
With Prime95 still running the CPUs have peaked at 60°C and 61°C.
Here are a few images:
I have to say I'm very pleased with the results. The system is 100% stable running at the speed as if I had a pair of 3.0 GHz X5472 processors along with 64 GB of 1600MHz FB-DIMMs.
I suspect that the "Extreme series" X5472 processors are actually 100% identical to the E5420's except they are just binned and labelled as X5472's. Even the TDP ratings make sense:
- The E5420's are rated at: 80W, 2.5GHz, 12MB cache, 1333FSB and 7.5x multiplier.
- The X5472's are rated at: 120W, 3.0GHz, 12MB cache, 1600FSB and 7.5x multiplier.
I haven't checked the prices today, but when I built this system the X5472 processors were selling for something like $2000 (Canadian $s) each and 64 GB of 1600MHz FB-DIMMs (16x 4GB modules) would have cost something like $20,000 (or more) for the RAM alone.
I only wish I'd done this earlier. I could have saved a lot of time when running some of those big FEA jobs (I'll have to keep Prime95 running for 24 hours or more to make certain the computational accuracy is still there first though).
EDIT: Actually I did just discover there is one other difference between the X5472s and the E5420s. The X5472's are rated to run at 63°C and the E5420's are rated to run at 67°C. Mine are running (just) below 63°C according to HWMonitor (see images above with Prime95 running on all 8 cores).
Here are the spec's:http://ark.intel.com/Compare.aspx?ids=33927,34447,
One thing I am puzzled about. When running at stock speeds (2.5GHz), the automatic thermal control system in the motherboard that runs the fans keeps the processors at 65°C during 100% utilization for a long time. When running at 3.0GHz, instead of adjusting the fans to keep the processors at 65°C, they are now being kept at 61°C instead. Why would the fan controller keep the CPUs 4°C cooler when overclocked to 3.0GHz? I would have expected the CPUs, if anything, to run a little hotter.
Edit: I think I figured out the reason why the CPUs are running cooler when overclocked (although this is really a guess). The Northbridge produces quite a bit more heat and this causes the chassis fans to run allot faster. I suspect this means that the air inside the case ends up being a fair bit cooler which results in a cooler CPU temperature.
I had a look at Intel's design guidelines for the 5400 series processors (http://www.intel.com/Assets/en_US/PDF/designguide/318611.pdf
) to get a better understanding on what are considered safe CPU operating temperatures for the X5472 (assuming I should now comply with them instead of the E5420 temperatures).
It turns out that the 63°C and 67°C Tcase maximum temperatures for the X5472 and E5420 respectively are actually external temperatures, measured at the geometric center on the outside of the CPU heat spreader.
The digital thermal sensor that HWMonitor is accessing is inside the case, physically inside the processing core itself so it will report considerably higher temperatures than what the outside Tcase temperature is. Also, since there is a certain amount of thermal resistance between the inside and outside of the CPU, even with identical internal chip temperatures, the specified external "Tcase maximum" temperature would need to be lower with the X5472 due to the 50% higher TDP. In other words it is quite possible that Intel's thermal ratings actually result in the same internal core temperature and that internal allowable temperature is likely above 70°C. I wish Intel published a maximum allowable internal temperature. So far I have not been able to find one.
This explains why with the original stock Intel coolers that the reported internal temperatures would get as high as 72°C before thermal throttling would actually kick in (in fact I'm not sure, but I recall thermal throttling might not have actually kicked in until HWMonitor reported temperatures as high as 75°C).
Also, the way the temperatures are actually reported is in degrees Celsius "below" thermal throttling temperature. Once the temperature is above this threshold, regardless of how high the temperature actually climbs it is always reported as just 0. I don't know what HWMonitor is using for the temperature calculation, but since I've seen temperatures in the 72-75°C range, it seems it is calculating the temperature as something like 75°C-reported temperature (HWMonitor may even be using a value higher than 75°C for all I know).
Unfortunately all of this says I don't really know what the actual CPU Tcase temperature is and the only way to truly find it is to measure it directly. What I do know is since I've seen reported temperatures of at least 72°C that when 62°C is reported the CPU is at least 10°C away from the thermal throttling temperature and according to the specification, thermal throttling should kick in when Tcase (external center of heat spreader) hits 67°C with the CPU running at 100% and a standard Intel spec'd cooler in place (so the internal temperature would likely be 72°C or more).
Based on this, my best estimate on the Tcase temperature is roughly 5°C less than HWMonitor's reported internal temperature. In this case it is reporting 62°C max so the Tcase is probably 57°C which is 10°C below the maximum allowed for the stock E5420 and 6°C below the maximum allowed for the X5472. It seems odd for Intel to specify a maximum allowable EXTERNAL temperature, instead of internal, but the way the spec is written Intel's intent is to rely on thermal throttling when required and to just specify an external maximum since this is something that can be directly measured by 3rd parties designing systems around the Xeon 5400 series hardware.
The specification also states the following:
By taking advantage of the Thermal Monitor features, system designers may reduce thermal solution cost by designing to the Thermal Design Power (TDP) instead of maximum power. TDP should be used for processor thermal solution design targets. TDP is not the maximum power that the processor can dissipate. TDP is based on measurements of processor power consumption while running various high power applications. This data set is used to determine those applications that are interesting from a power perspective. These applications are then evaluated in a controlled thermal environment to determine their sensitivity to activation of the thermal control circuit. This data set is then used to derive the TDP targets published in the processors datasheet. The Thermal Monitor can protect the processors in rare workload excursions above TDP. Therefore, thermal solutions should be designed to dissipate this target power level. The thermal management logic and thermal monitor features are discussed in extensive detail in the Quad-Core Intel® Xeon® Processor 5400 Series Datasheet.
In other words the stock Xeon cooler is designed to allow the processor to run at the maximum Tcase temperature when running a typical application and at times it is expected that thermal throttling will kick in to keep the temperature under control (which is what Intel told me when I first contacted them when I first built the system and I felt the CPU temperatures were too high). When running something like Prime95, which produces maximum heat, it sounds like thermal throttling would actually be expected to kick in to keep the temperature under control when using the stock Intel cooler. With the stock Intel coolers this is exactly what I used to experience (so I installed the 6 heatpipe Supermicro 150W Xeon coolers instead of the Intel ones and this dropped the temperatures by about 10°C).
There is also a section in the specification that covers fan failure. The processor is spec'd to operate for 360 hours per year with a failed fan, while being guaranteed to not be damaged in any way (as long as the thermal throttling capability is enabled in the BIOS).
Anyway, although all of this is not as clear as I would like, it sounds like the temperatures I am seeing right now are about right as far as Intel's design guidelines are concerned and are not a problem (even if I actually was using X5472's instead of E5420's). One bit of good news is the CPUs actually run a good 5 or more °C cooler when running heavy duty application (such as Ansys) compared to when running Prime95 so Prime95 produces an absolute worst case CPU load (at least when running a "torture test").