Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › CPU Load and Memory Load
New Posts  All Forums:Forum Nav:

CPU Load and Memory Load

post #1 of 8
Thread Starter 
Hey guys,

So first off, this isn't for school or anything like that. Haha.

So I am relatively new to python and I am wanting to create a script for a unix based box to test another program I am working on out. Essentially I want to stress the CPU and memory, ideally in a smooth, sine like pattern.

Doesn't need to be anything complex. Ideally one script to load (stress) the CPU and another script to load (fill up) the memory. But I want to have some control over it. I don't want to just stress the CPU to 100%, and same for the memory.

Anyone have ideas on how to go about doing this in Python?

For the memory I was thinking of just creating massive arrays of strings....
Then the cpu I am not sure, I know how to stress it (lots of calculations) but I am not really sure how to go about doing it in a sine like fashion so the load is smooth and somewhat controllable.

Any ideas and thoughts are welcomed and greatly appreciated. Right now for the CPU I have a primefactors generator... but the load doesn't get very high.

Thanks. smile.gif
Edited by deafboy - 11/16/11 at 3:56pm
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
post #2 of 8
Thread Starter 
So I got the memory to load up and unload gracefully (gracefully enough anyways).

But everything I have tried for the CPU has only been able to get up to 40% load where it hung around 5% most of the time.

Anyone have thoughts for the CPU stuff?
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
post #3 of 8
Thread Starter 
So I got the CPU to get up to 100%... trying to use sleeps to create a gradual incline/decline of the processor usage. Incline, easy enough, decline it kind of a pain. Ideas?
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
post #4 of 8
Thread Starter 
Alright, well I got it more or less. Got some help from Google, AKA, found something similar to what I wanted and tweaked it to fit my needs.

Thought I would post it just in case anyone else was looking for something similar.



For memory,
Code:
import threading
import math
import time

class MyThread ( threading.Thread ) :
        def run(self):
                a=[]
                amountOfRam = 2000                                              #amount of ram in MB
                oneMB = 1000000
                
                while(1):
                        print('Filling the Memory . . .')
                        
                        for i in range(amountOfRam):            #build an array of specified MB size
                                #a.append(bytearray(oneMB))     #array with byte array in 1MB chunks
                                a.append(' ' * oneMB)                   #array with byte array in 1MB chunks
                                time.sleep(0.04)                                #sleep for 40msec
                        
                        print('MEMORY FILLED WITH ' + str(amountOfRam) + 'MB')
                        print('Clearing the Memory . . .')
                                
                        for i in range(amountOfRam):
                                a.pop()                                                 #pop to start shrinking array
                                time.sleep(0.04)                                #sleep for 40msec
                                
                        print('MEMORY CLEARED')


MyThread().start()




For CPU, relatively a sinusoidal usage, from low to high to low ...etc
Code:
import itertools
import math
import time
import sys
import threading


class MyThread ( threading.Thread ) :
        def run(self):
                time_period = 60   # seconds
                time_slice  = 0.04 # seconds

                N = int(time_period / time_slice)

                for i in itertools.cycle(range(N)):
                        busy_time = time_slice / 2 * (math.sin(2*math.pi*i/N) + 1)
                        t = time.clock() + busy_time
                        while t > time.clock():
                                pass
                        time.sleep(time_slice - busy_time);


MyThread().start()



For CPU, full load... no transitions
Code:
import threading
import math

def primefactors(x):
        factorlist=[]
        loop=2
        while loop<=x:
                if x%loop==0:
                        x/=loop
                        factorlist.append(loop)
                else:
                        loop+=1
        return factorlist

class MyThread ( threading.Thread ) :
        def run(self):  
                count = 2
                while(1):
                        primefactors(count)
                        count = count + 1

MyThread().start()
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
post #5 of 8
Thread Starter 
So my goal for this weekend is to specify the min and max load values so it's not always alternating between 0-100% usage.

I think I have a good idea what to do, but as always, feel free to chime in if you'd like. Once it's done and working I will be sure to post it up.
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
post #6 of 8
Not sure if these suggestions are too late, but CPU loads are generally tested with prime number generators. You'll probably want to run several threads as most CPUs these days are multi-core - so a single threaded arithmetical routine might only utilise one core and thus your load averages wouldn't peak in the expected way.

As for testing memory, you would be best using classes (not sure how well Python handles classes - if at all), that way you can assign chunks of RAM to each class object and have an array of said objects. Then, when you want to free up memory, it's a simple process of destroying the class objects.

[edit]

oops, just spotted you are threading primes:D
post #7 of 8
Thread Starter 
smile.gif

As for reaching specific usage numbers, I figure for the CPU I will make a run through checking the usage and once I reach the usage I will record what sleep value is need and assure that I dont exceed it (for max load) or go beneath it (for minimum load).

For memory, very easy as I only have to assign certain number of blocks, no real calculations needed.
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
post #8 of 8
Thread Starter 
Memory was easy enough but I still can't get the CPU one to do what I would like...

I have a bash script that a user uses to give the percentages they want then that bash scripts fires off however many instances of python it needs and passes those values off to python.

Issues is, I can't seem to get the current CPU usage in python, I wrote one but it wasn't very accurate (at all). I then have another batch script to get the cpu usage and that works, very fast, very accurate but I have no way of getting that value into python where I need it.

So I am thinking of scrapping everything (unless anyone has ideas) and starting over....but I am not exactly sure what the best plan of attack would be. Anyone have any thoughts?
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
 
Play Server
(10 items)
 
NAS
(14 items)
 
CPUCPUMotherboardRAM
Intel Xeon L5640 Intel Xeon L5640 Dell R710 Mobo Hynix 144GB DDR3 ECC PC3L-10600R 
Hard DrivePowerCaseOther
Intel X25-M G2 80GB Redundant 840W PSU Dell 710 2.5" - Google Search Appliance Dell Perc H700 Raid Controller 
OtherOther
Dell iDRAC6 Enterprise Mellanox Connectx-2 10Gb SFP+ NIC 
CPUCPUMotherboardRAM
Intel Xeon E5-2670 v1 Intel Xeon E5-2670 v1 SuperMicro X9DRL-iF 64GB Kingston ECC (kvr16r11d4/16HA) 
Hard DriveHard DriveHard DriveCooling
12x 4TB HGST 5K4000 SanDisk SSD 960GB (cache) 6x 8TB WD Red Scythe AP-29 
CoolingOSPowerCase
2x Noctua U9DXi4 FreeNAS Corsair AX750 Norco RPC-4224 
OtherOther
3x LSI 9211-8i (HBA) Mellanox Connectx-2 10Gb NIC 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Coding and Programming
Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › CPU Load and Memory Load