Overclock.net › Forums › Industry News › Hardware News › [PCPer] Frame Rating: A New Graphics Performance Metric
New Posts  All Forums:Forum Nav:

[PCPer] Frame Rating: A New Graphics Performance Metric

post #1 of 69
Thread Starter 
card3zplo.jpg

Quote:
A change is coming in 2013

If the new year will bring us anything, it looks like it might be the end of using "FPS" as the primary measuring tool for graphics performance on PCs. A long, long time ago we started with simple "time demos" that recorded rendered frames in a game like Quake and then played them back as quickly as possible on a test system. The lone result was given as time, in seconds, and was then converted to an average frame rate having known the total number of frames recorded to start with.

More recently we saw a transition to frame rates over time and the advent frame time graphs like the ones we have been using in our graphics reviews on PC Perspective. This expanded the amount of data required to get an accurate picture of graphics and gaming performance but it was indeed more accurate, giving us a more clear image of how GPUs (and CPUs and systems for that matter) performed in games.

And even though the idea of frame times have been around just a long, not many people were interested in getting into that detail level until this past year. A frame time is the amount of time each frame takes to render, usually listed in milliseconds, and could range from 5ms to 50ms depending on performance. For a reference, 120 FPS equates to an average of 8.3ms, 60 FPS is 16.6ms and 30 FPS is 33.3ms. But rather than average those out by each second of time, what if you looked at each frame individually?

Scott over at Tech Report started doing that this past year and found some interesting results. I encourage all of our readers to follow up on what he has been doing as I think you'll find it incredibly educational and interesting.

Through emails and tweets many PC Perspective readers have been asking for our take on it, why we weren't testing graphics cards in the same fashion yet, etc. I've stayed quiet about it simply because we were working on quite a few different angles on our side and I wasn't ready to share results. I am still not ready to share the glut of our information yet but I am ready to start the discussion and I hope our community find its compelling and offers some feedback.

Source


Full article at the source and comment section.


Sample Images (Click to show)
heaven_nocolord0rkt.jpg

heaven_colorjkqd9.jpg

sleepingdogs_colora5rxd.jpg


PC Perspective - Frame Rating: A New Graphics Performance Metric




Frame Rating Part 2: Finding and Defining Stutter
Part 2 YouTube Video (Click to show)
d318ab95.jpg
Quote:
Today we are taking that a step further and looking at a couple of captured videos that demonstrate a "stutter" and walking you through, frame by frame, how we can detect, visualize and even start to measure them.

dis171p09.jpg
Quote:
This video takes a couple of examples of stutter in games, DiRT 3 and Dishonored to be exact, and shows what they look like in real time, at 25% speed and then finally in a much more detailed frame-by-frame analysis.

Frame Rating Part 2: Finding and Defining Stutter

Quote:
Obviously this is just a couple instances of what a stutter is and there are often times less apparent in-game stutters that are even harder to see in video playback. Not to worry - this capture method is capable of seeing those issues as well and we plan on diving into the "micro" level as well shortly.

We aren't going to start talking about whose card and what driver is being used yet and I know that there are still a lot of questions to be answered on this topic. You will be hearing more quite soon from us and I thank you all for your comments, critiques and support.

Edited by Metric - 1/17/13 at 3:38am
post #2 of 69
It's good to see that reviewers will be giving us tests like this. Gives a better idea of the overall experience, instead of just the pure fps.
 
Benching
(17 items)
 
 
CPUMotherboardGraphicsRAM
[i7 5960X @ 4.8GHz] [Rampage V Extreme] [Titan 1400MHz (1500MHz bench)] [Various] 
Hard DriveCoolingCoolingCooling
[250GB 840EVO +2x SpinpointF3 1TB RAID0] [LD PC-V2 SS Phase Change] [XSPC X2O 750 pump/res] [Monsta 360 full copper + EK XT 360 + XT 240] 
MonitorPowerCaseAudio
[Crossover 27Q LED-P 1440p+ASUS 1200p+LG 1080p] [Corsair AX1200] [Dimastech Easy v3.0] [Sennheiser HD558s] 
CPUCPUMotherboardGraphics
FX 8320, FX 8350, Phenom II x2 555BE i7 3930K, i7 860, i7 4770K, 68x Celeron D CVF, commando, 2x RIVE, Z87X-OC Asus 4870x2, Sapphire 4870 
GraphicsGraphicsGraphicsGraphics
2x 5870, 5850, 5830, 5770 2x 3870x2, 3870 GTX Titan, GTX 480, GTX 590 GTX 285, GTX 260, 4x 9800GT, 8800GTX 
RAMHard DriveCoolingCooling
4x4GB vengeance, 2x4GB predatorX, 2x1GB OCZ DDR2 Intel X25-M 80GB LD PC-V2 SS Phase Change OCN Marksman 
CoolingCoolingOSPower
2x old tek slims (GPU) Various watercooling stuff win7, winxp AX1200 
Case
test bench / cardboard box 
  hide details  
Reply
 
Benching
(17 items)
 
 
CPUMotherboardGraphicsRAM
[i7 5960X @ 4.8GHz] [Rampage V Extreme] [Titan 1400MHz (1500MHz bench)] [Various] 
Hard DriveCoolingCoolingCooling
[250GB 840EVO +2x SpinpointF3 1TB RAID0] [LD PC-V2 SS Phase Change] [XSPC X2O 750 pump/res] [Monsta 360 full copper + EK XT 360 + XT 240] 
MonitorPowerCaseAudio
[Crossover 27Q LED-P 1440p+ASUS 1200p+LG 1080p] [Corsair AX1200] [Dimastech Easy v3.0] [Sennheiser HD558s] 
CPUCPUMotherboardGraphics
FX 8320, FX 8350, Phenom II x2 555BE i7 3930K, i7 860, i7 4770K, 68x Celeron D CVF, commando, 2x RIVE, Z87X-OC Asus 4870x2, Sapphire 4870 
GraphicsGraphicsGraphicsGraphics
2x 5870, 5850, 5830, 5770 2x 3870x2, 3870 GTX Titan, GTX 480, GTX 590 GTX 285, GTX 260, 4x 9800GT, 8800GTX 
RAMHard DriveCoolingCooling
4x4GB vengeance, 2x4GB predatorX, 2x1GB OCZ DDR2 Intel X25-M 80GB LD PC-V2 SS Phase Change OCN Marksman 
CoolingCoolingOSPower
2x old tek slims (GPU) Various watercooling stuff win7, winxp AX1200 
Case
test bench / cardboard box 
  hide details  
Reply
post #3 of 69
After reading that post from the Intel guy about how the game itself adjusts to frame time discrepancies, I'm wondering if measuring it at the output is really the best way. It'll show the effect of any frame "smoothing" they are doing, but if the game itself is inserting skips because of frame delays, I don't know if this method will show it. Hopefully they'll use a combination of both to show the difference between the FRAPS times and the display times.
Quote:
Remember, the CPU is timing the rate at which it is allowed to fill the input queue and using that as the rate at which it updates the game simulation. Thus a long frame like the one at B causes it to conclude that the rendering pipeline has slowed all-of-a-sudden, and to start updating the game simulation by 38ms each frame instead of 16ms. i.e. onscreen objects will move more than twice as far on the frame following B as they did previously. In this case though, it was just a spike and not a change in the underlying steady state rate, so the following frames (C) effectively have to move less (10ms each) to resynchronize the game with the proper wall clock time. This sort of "jump ahead, then slow down" jitter is extremely visible to our eyes, and demonstrated well by Scott's follow-up video using a high speed camera. Note that what you are seeing are likely not changes in frame delivery to the display, but precisely the affect of the game adjusting how far it steps the simulation in time each frame.
Quote:
3) A spike anywhere in the pipeline will cause the game to adjust the simulation time, which is pretty much guaranteed to produce jittery output. This is true even if frame delivery to the display (i.e. rendering pipeline output) remains buffered and consistent. i.e. it is never okay to see spikey output in frame latency graphs.

http://forum.beyond3d.com/showthread.php?t=62775
Edited by Forceman - 1/4/13 at 11:36pm
post #4 of 69

I like the idea behind this. Hopefully more reviewers will decide to adopt or least try out this approach.

post #5 of 69
I also like this approach, even if it overemphasizes what may or may not be a perceptible performance issue.

By forcing manufacturers to look at metrics other than raw frame rate, we will ensure that frame rate optimizations do not take precedence over a good gaming experience.
Primary
(15 items)
 
Secondary
(13 items)
 
In progress
(10 items)
 
CPUMotherboardGraphicsRAM
5820K @ 4.2/3.5GHz core/uncore, 1.175/1.15v Gigabyte X99 SOC Champion (F22n) Gigabyte AORUS GTX 1080 Ti (F3P) @ 2025/1485, 1... 4x4GiB Crucial @ 2667, 12-12-12-28-T1, 1.34v 
Hard DriveHard DriveHard DriveCooling
Plextor M6e 128GB (fw 1.06) M.2 (PCI-E 2.0 2x) 2x Crucial M4 256GB 4x WD Scorpio Black 500GB Noctua NH-D15 
OSMonitorKeyboardPower
Windows 7 Professional x64 SP1 BenQ BL3200PT Filco Majestouch Tenkeyless (MX Brown) Corsair RM1000x 
CaseMouseAudio
Fractal Design Define R4 Logitech G402 Realtek ALC1150 + M-Audio AV40 
CPUMotherboardGraphicsRAM
X5670 @ 4.4/3.2GHz core/uncore, 1.36 vcore, 1.2... Gigabyte X58A-UD5 r2.0 w/FF3mod10 BIOS Sapphire Fury Nitro OC+ @ 1053/500, 1.225vGPU/1... 2x Samsung MV-3V4G3D/US @ 2000, 10-11-11-30-T1,... 
RAMHard DriveHard DriveHard Drive
1x Crucial BLT4G3D1608ET3LX0 @ 2000, 10-11-11-3... OCZ (Toshiba) Trion 150 120GB Hyundai Sapphire 120GB 3x Hitachi Deskstar 7k1000.C 1TB 
CoolingOSPowerCase
Noctua NH-D14 Windows 7 Pro x64 SP1 Antec TP-750 Fractal Design R5 
Audio
ASUS Xonar DS 
CPUMotherboardGraphicsRAM
i7-6800K @ 4.3/3.5GHz core/uncore, 1.36/1.2v ASRock X99 OC Formula (P3.10) GTX 780 (temporary) 4x4GiB Crucial DDR4-2400 @ 11-13-12-28-T2, 1.33v 
Hard DriveHard DriveCoolingOS
Intel 600p 256GB NVMe 2x HGST Travelstar 7k1000 1TB Corsair H55 (temporary) Windows Server 2016 Datacenter 
PowerCase
Seasonic SS-860XP2 Corsair Carbide Air 540 
  hide details  
Reply
Primary
(15 items)
 
Secondary
(13 items)
 
In progress
(10 items)
 
CPUMotherboardGraphicsRAM
5820K @ 4.2/3.5GHz core/uncore, 1.175/1.15v Gigabyte X99 SOC Champion (F22n) Gigabyte AORUS GTX 1080 Ti (F3P) @ 2025/1485, 1... 4x4GiB Crucial @ 2667, 12-12-12-28-T1, 1.34v 
Hard DriveHard DriveHard DriveCooling
Plextor M6e 128GB (fw 1.06) M.2 (PCI-E 2.0 2x) 2x Crucial M4 256GB 4x WD Scorpio Black 500GB Noctua NH-D15 
OSMonitorKeyboardPower
Windows 7 Professional x64 SP1 BenQ BL3200PT Filco Majestouch Tenkeyless (MX Brown) Corsair RM1000x 
CaseMouseAudio
Fractal Design Define R4 Logitech G402 Realtek ALC1150 + M-Audio AV40 
CPUMotherboardGraphicsRAM
X5670 @ 4.4/3.2GHz core/uncore, 1.36 vcore, 1.2... Gigabyte X58A-UD5 r2.0 w/FF3mod10 BIOS Sapphire Fury Nitro OC+ @ 1053/500, 1.225vGPU/1... 2x Samsung MV-3V4G3D/US @ 2000, 10-11-11-30-T1,... 
RAMHard DriveHard DriveHard Drive
1x Crucial BLT4G3D1608ET3LX0 @ 2000, 10-11-11-3... OCZ (Toshiba) Trion 150 120GB Hyundai Sapphire 120GB 3x Hitachi Deskstar 7k1000.C 1TB 
CoolingOSPowerCase
Noctua NH-D14 Windows 7 Pro x64 SP1 Antec TP-750 Fractal Design R5 
Audio
ASUS Xonar DS 
CPUMotherboardGraphicsRAM
i7-6800K @ 4.3/3.5GHz core/uncore, 1.36/1.2v ASRock X99 OC Formula (P3.10) GTX 780 (temporary) 4x4GiB Crucial DDR4-2400 @ 11-13-12-28-T2, 1.33v 
Hard DriveHard DriveCoolingOS
Intel 600p 256GB NVMe 2x HGST Travelstar 7k1000 1TB Corsair H55 (temporary) Windows Server 2016 Datacenter 
PowerCase
Seasonic SS-860XP2 Corsair Carbide Air 540 
  hide details  
Reply
post #6 of 69
Seems like there is a lot of focus on stuttering as of late. Essentially what I'm picking up from this is you can average 80fps, but if every 5th frame takes 33ms (or even more I suppose) to render then you're going to have stutter, while a bench of 16.6ms on every frame sequentially leading to an average of 60fps wouldn't have the stutter and would, in fact, be better overall.
Lawl Mark II
(12 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5 @ 5ghz (air) Z77 Extreme6 SLI GTX 660 Ti SC 3GB (1228/1734) Patriot Viper 3 16GB 2133mhz 
Hard DriveCoolingOSKeyboard
OCZ Vertex 3 128GB XIGMATEK Dark Knight II Windows 7 Ultimate Razer Blackwidow Ultimate 
PowerMouseMouse PadAudio
CM Silent Pro 1000W Cyborg RAT 7 Razer Goliathus Xonar DG 
  hide details  
Reply
Lawl Mark II
(12 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i5 @ 5ghz (air) Z77 Extreme6 SLI GTX 660 Ti SC 3GB (1228/1734) Patriot Viper 3 16GB 2133mhz 
Hard DriveCoolingOSKeyboard
OCZ Vertex 3 128GB XIGMATEK Dark Knight II Windows 7 Ultimate Razer Blackwidow Ultimate 
PowerMouseMouse PadAudio
CM Silent Pro 1000W Cyborg RAT 7 Razer Goliathus Xonar DG 
  hide details  
Reply
post #7 of 69
I am glad that we are moving in this direction
facing tough times for companies that want to increase sales in pre-Christmas period with questionable methods ... will also contribute to a better general education of users
My System
(13 items)
 
  
Reply
My System
(13 items)
 
  
Reply
post #8 of 69
so we might start looking more at quality, rather than just concentrating on quantity. thats interesting.
Unnamed Build
(23 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i7 3770k MAXIMUS V GENE NVIDIA GeForce GTX 780 Hydro Copper Vulcan 
Hard DriveCoolingCoolingCooling
Samsung 1TB 840 SSD Koolance CPU-380i XSPC Dual Bay Reservoir Phobya Xtreme 200mm Radiator 
CoolingCoolingCoolingOS
XSPC EX120 120mm Radiator Silverstone AP181 Fan Gentle Typhoon AP15 Windows 8 Professional x64 
MonitorKeyboardPowerCase
HP LP3065c DasKeyboard Ultimate Model S Seasonic X650 Silverstone TJ08B-E 
MouseMouse PadAudioAudio
Logitech G700 Mionix Propus 380 Audioengine D1 USB DAC Parasound 2125 Amplifier 
AudioAudioAudio
Beyerdynamic DT 770 Pro-80 Closed Studio Headph... Audio-Technica ATH-M50 Definitive Technology Studiomonitor 450 
  hide details  
Reply
Unnamed Build
(23 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i7 3770k MAXIMUS V GENE NVIDIA GeForce GTX 780 Hydro Copper Vulcan 
Hard DriveCoolingCoolingCooling
Samsung 1TB 840 SSD Koolance CPU-380i XSPC Dual Bay Reservoir Phobya Xtreme 200mm Radiator 
CoolingCoolingCoolingOS
XSPC EX120 120mm Radiator Silverstone AP181 Fan Gentle Typhoon AP15 Windows 8 Professional x64 
MonitorKeyboardPowerCase
HP LP3065c DasKeyboard Ultimate Model S Seasonic X650 Silverstone TJ08B-E 
MouseMouse PadAudioAudio
Logitech G700 Mionix Propus 380 Audioengine D1 USB DAC Parasound 2125 Amplifier 
AudioAudioAudio
Beyerdynamic DT 770 Pro-80 Closed Studio Headph... Audio-Technica ATH-M50 Definitive Technology Studiomonitor 450 
  hide details  
Reply
post #9 of 69
Quote:
Originally Posted by Forceman View Post

After reading that post from the Intel guy about how the game itself adjusts to frame time discrepancies, I'm wondering if measuring it at the output is really the best way. It'll show the effect of any frame "smoothing" they are doing, but if the game itself is inserting skips because of frame delays, I don't know if this method will show it. Hopefully they'll use a combination of both to show the difference between the FRAPS times and the display times.
http://forum.beyond3d.com/showthread.php?t=62775
Personally I think measuring the output is the best possible method to give someone perspective on what they'll be viewing at the monitor. However using this method and FRAPs/Afterburner should make it easier to find problems in latency, trying to target where the problem is and making it easier for both AMD and Nvidia to fix said problems. However what truly matters is the end experience, what we're seeing at the monitor. So being able to record that data will be a huge leap for latency testing in general.
Edited by Mazel - 1/5/13 at 3:55am
post #10 of 69
Must have been an epic amount of work to design their own piece of hardware and software for it. Just hope they will figure out how to properly analyze the data now.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Hardware News
Overclock.net › Forums › Industry News › Hardware News › [PCPer] Frame Rating: A New Graphics Performance Metric