Overclock.net banner

1 - 18 of 18 Posts

·
Premium Member
Joined
·
10,020 Posts
Discussion Starter #1
I've been seeing a lot of review sites posting this information and on the surface , it appears to be quite useful. However after playing around with frame time analyzers using my own data it became apparent to me that it's not nearly as meaningful as it had appeared.<br><br>
The case in point<br><br><a class="H-lightbox-open" href="http://www.overclock.net/content/type/61/id/3146888/"><img alt="" class="lightbox-enabled" data-id="3146888" data-type="61" src="http://www.overclock.net/content/type/61/id/3146888/width/350/height/700/flags/LL" style="; width: 350px; height: 194px"></a><br><br><br>
166 fps minimum vs showing a .1% low of 100 because of single frames spread out over the entirety of a sample, ( out of 33,000 in 3 minutes of play) doesn't seem to be very representative of perceptible differences in performance.<br><br>
Would any of you smart fellows out there have any thoughts on how to make better use of frametime data ? I was wondering how to use those 33000 data points to find any one second of gameplay with the lowest performance?<br><br>
Any excel or open office wizards out there care to make a tool for doing this?<br><br>
Perhaps fraps already does this to produce the 166 number? I've always assumed that it simply polls the fps during a given second and reports this instead.<br><br>
Thanks in advance for any and all replies.
 

·
Iconoclast
Joined
·
30,216 Posts
FRAPS is counting the time of every frame.<br><br>
166fps minimum is clearly incorrect, and is probably a 1s average. A ~20ms frame is a ~50fps minimum frame rate. The 1% and .01% frame rates are there to cull extreme outliers like that.<br><br>
I'd argue, that in this case, the 0.1% is far more useful than that misleading average figure, or the single frame out of ~33k frames that falls significantly outside the rest.
 
  • Rep+
Reactions: cssorkinman

·
Premium Member
Joined
·
10,020 Posts
Discussion Starter #3
<div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434017" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>Blameless</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434017"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br>
FRAPS is counting the time of every frame.<br><br>
166fps minimum is clearly incorrect, and is probably a 1s average. A ~20ms frame is a ~50fps minimum frame rate. The 1% and .01% frame rates are there to cull extreme outliers like that.<br><br>
I'd argue, that in this case, the 0.1% is far more useful than that misleading average figure, or the single frame out of ~33k frames that falls significantly outside the rest.</div>
</div>
<br>
The reason I think it isn't a good measure is because you would have to have 100 of those highest frametime images occur in the same one second segement of the bench in order to even come close to the threshold of human perception. If I interpret this correctly , there are only 33 such instances of frames taking that long to render across nearly a 3 minute benchmark.<br><br><br><br>
If I recall the data comes from this section of BF1's single player mode , on a later run.<br><br>
<a href="https://youtu.be/YfhL08s4IMw" target="_blank">https://youtu.be/YfhL08s4IMw</a><br><br>
FPS is slightly higher whilst not recording and I recall a later run with the cpu cranked up a bit averaging 198 fps with the 200 fps cap.<br><br>
I appreciate the reply.
 

·
Premium Member
Joined
·
6,351 Posts
<div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434017" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>Blameless</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434017"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br>
FRAPS is counting the time of every frame.<br><br>
166fps minimum is clearly incorrect, and is probably a 1s average. A ~20ms frame is a ~50fps minimum frame rate. The 1% and .01% frame rates are there to cull extreme outliers like that.<br><br>
I'd argue, that in this case, the 0.1% is far more useful than that misleading average figure, or the single frame out of ~33k frames that falls significantly outside the rest.</div>
</div>
<br><br>
I think GamersNexus does a good job at explaining it here: <a href="https://www.youtube.com/watch?v=uXepIWi4SgM" target="_blank">https://www.youtube.com/watch?v=uXepIWi4SgM</a>. I also have the g3258 and have seen what he's describing.
 
  • Rep+
Reactions: cssorkinman

·
Iconoclast
Joined
·
30,216 Posts
<div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434055" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>cssorkinman</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434055"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br>
The reason I think it isn't a good measure is because you would have to have 100 of those highest frametime images occur in the same one second segement of the bench in order to even come close to the threshold of human perception.</div>
</div>
<br>
100 of those frames would <i>be</i> a full second or more.<br><br>
Given the frame time distribution of your graph (where every several frames you get a frame that double the frame time) I'd consider those 100fps and 112fps figures way more useful than the 166fps or 193fps figures.<br><br>
The 0.1% frame rate lets the true outliers be culled and gives you a useful minimum performance figure.<br><div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434064" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>Mr.N00bLaR</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434064"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br>
I think GamersNexus does a good job at explaining it here: <a href="https://www.youtube.com/watch?v=uXepIWi4SgM" target="_blank">https://www.youtube.com/watch?v=uXepIWi4SgM</a>. I also have the g3258 and have seen what he's describing.</div>
</div>
<br>
Yes.<br><br>
In cssorkinman's case, he's not getting any severe dips like that, but if you had a true constant/fixed frame time 193fps output vs. a true 100fps output, the later would be closer to representing the smoothness he'd see than the former.<br><br>
Now, at 100+fps it's smooth enough regardless and many people may well not be able to tell the difference between 193/166 apparent fps and 112/100 apparent fps, but that doesn't deminish the usefulness of the 0.1% lows as a measurement.
 

·
Premium Member
Joined
·
10,020 Posts
Discussion Starter #6
<div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434064" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>Mr.N00bLaR</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434064"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br><div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434017" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>Blameless</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434017"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br>
FRAPS is counting the time of every frame.<br><br>
166fps minimum is clearly incorrect, and is probably a 1s average. A ~20ms frame is a ~50fps minimum frame rate. The 1% and .01% frame rates are there to cull extreme outliers like that.<br><br>
I'd argue, that in this case, the 0.1% is far more useful than that misleading average figure, or the single frame out of ~33k frames that falls significantly outside the rest.</div>
</div>
<br><br>
I think GamersNexus does a good job at explaining it here: <a href="https://www.youtube.com/watch?v=uXepIWi4SgM" target="_blank">https://www.youtube.com/watch?v=uXepIWi4SgM</a>. I also have the g3258 and have seen what he's describing.</div>
</div>
<br>
I understand what they are trying to express and I thank you for the link.<br><br>
I guess it comes down to the way fraps is measuring it - does it keep track of the fps produced for every second and find the lowest and show that value as the min or is it just sampling every so often.<br><br>
After looking at the min max ave spreadsheet produced it would seem that it is at least averaging across complete seconds ( possibly 10 from the look of it) and considers those .1% lows because it has the average fps being the same as the frame time analyzer.
 

·
Iconoclast
Joined
·
30,216 Posts
<div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434095" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>cssorkinman</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434095"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br>
I guess it comes down to the way fraps is measuring it - does it keep track of the fps produced for every second and find the lowest and show that value as the min or is it just sampling every so often.</div>
</div>
<br>
FRAPS times every single frame. It's less than ideal as it doesn't pull info from the very end of the display chain (pure software solutions cannot do this), but should be accurate enough.<br><br><img alt="ATPipeline_575px.png" class="bbcode_img" src="https://images.anandtech.com/doci/6862/ATPipeline_575px.png"><br><br>
Hardware based capture and measurement solutions grab from the very end of the display chain, at the physical display output itself, so can measure what is actually seen.
 

·
Premium Member
Joined
·
10,020 Posts
Discussion Starter #8
<div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434100" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>Blameless</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434100"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br><div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434095" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>cssorkinman</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434095"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br>
I guess it comes down to the way fraps is measuring it - does it keep track of the fps produced for every second and find the lowest and show that value as the min or is it just sampling every so often.</div>
</div>
<br><span style="text-decoration:underline;">FRAPS times every single frame. It's less than ideal as it doesn't pull info from the very end of the display chain</span> (pure software solutions cannot do this), but should be accurate enough.<br><br><img alt="ATPipeline_575px.png" class="bbcode_img" src="https://images.anandtech.com/doci/6862/ATPipeline_575px.png"><br><br>
Hardware based capture and measurement solutions grab from the very end of the display chain, at the physical display output itself, so can measure what is actually seen.</div>
</div>
<br><br>
Would that account for frametimes that are lower than the 200 fps limit should allow?
 

·
Iconoclast
Joined
·
30,216 Posts
<div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434111" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>cssorkinman</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434111"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br>
Would that account for frametimes that are lower than the 200 fps limit should allow?</div>
</div>
<br>
It could.
 

·
Premium Member
Joined
·
10,020 Posts
Discussion Starter #10
<div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434137" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>Blameless</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434137"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br><div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434111" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>cssorkinman</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless#post_26434111"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br>
Would that account for frametimes that are lower than the 200 fps limit should allow?</div>
</div>
<br>
It could.</div>
</div>
<br>
I appreciate your time, thank you. <img alt="thumb.gif" class="bbcode_smiley" src="http://files.overclock.net/images/smilies/thumb.gif"><br><br>
Now I'm wondering about the cap's effect on frametimes.<br><br>
I'll keep playing with it
 

·
Premium Member
Joined
·
10,020 Posts
Discussion Starter #11
Bumping into the frame cap is producing the " seismic" activity on frafs <img alt="tongue.gif" class="bbcode_smiley" src="http://files.overclock.net/images/smilies/tongue.gif"><br><br>
120 fps cap medium graphics settings<br><a class="H-lightbox-open" href="http://www.overclock.net/content/type/61/id/3147316/"><img alt="" class="lightbox-enabled" data-id="3147316" data-type="61" src="http://www.overclock.net/content/type/61/id/3147316/width/350/height/700/flags/LL" style="; width: 350px; height: 159px"></a><br><br>
200 fps cap medium settings.<br><br><a class="H-lightbox-open" href="http://www.overclock.net/content/type/61/id/3147317/"><img alt="" class="lightbox-enabled" data-id="3147317" data-type="61" src="http://www.overclock.net/content/type/61/id/3147317/width/350/height/700/flags/LL" style="; width: 350px; height: 158px"></a><br><br><br>
Something else is bugging me about the reviewers - they keep claiming that they aren't gpu limited using a 1080ti at utlra settings @ 1080 res - if that's true , they should be clobbering my Fury regardless of lod settings.
 

·
Premium Member
Joined
·
6,351 Posts
Also another GN feference, but Steve shows in a few titles that the game engines have a maximum frame rate despite having an even faster gpu, especially at lower resolutions. Seems like it matters which games you are comparing at 1080 when you say clobbered.
 

·
Premium Member
Joined
·
10,020 Posts
Discussion Starter #13
<div class="quote-container" data-huddler-embed="/t/1641492/frametime-analysis-1-lows-seem-meaningless/10#post_26435606" data-huddler-embed-placeholder="false"><span>Quote:</span>
<div class="quote-block">Originally Posted by <strong>Mr.N00bLaR</strong> <a href="/t/1641492/frametime-analysis-1-lows-seem-meaningless/10#post_26435606"><img alt="View Post" class="inlineimg" src="/img/forum/go_quote.gif"></a><br><br>
Also another GN feference, but Steve shows in a few titles that the game engines have a maximum frame rate despite having an even faster gpu, especially at lower resolutions. Seems like it matters which games you are comparing at 1080 when you say clobbered.</div>
</div>
<br>
I think something is really odd about the 1080ti's reported usage % Need to get one so I can figure out what kind of a$$hattery is going on with them.
 

·
Premium Member
Joined
·
10,020 Posts
Discussion Starter #14
Demonstration of why I feel the way I do about these measurements.

Very similar numbers generated , no noticeable fps slow down in one , in the other the dip was as obvious as a hippo riding shotgun in your Prius.
 

Attachments

·
Premium Member
Joined
·
10,020 Posts
Discussion Starter #16
Ideally you want output that has as little variance as possible.
Minimum can show you absolute lowest spike.
.1% a part of minimums, 1% and so on, ...

At best times, fps viewed as histogram.
Ideally as long as the minimum output is above the threshold of perception . If that is true, the absolute minimum seems meaningless.
 

·
Overclocker
Joined
·
11,339 Posts
An output that jumps between 10fps and 300fps fast and constantly, will look pretty bad despite having 155fps average fps. What is a threshold of perception, it differs for each observer.

Min max has always been fairly useless metric. Hence some use .1% and 1% lows instead eventhough having precise time plot and a histogram is the way to go. It is far easier for people to understand and for reports to show a single number than to try and cram 40 plots onto one page/video frame. Some reviewers offer plots for some tests. Guru3D.

.1% and 1% suffer the same issue as min/max as well in that the time length of spikes is important as the longer it is the more perceptible it will be. All in all a custom metric would have to be agreed upon by reviewers that would return a quality of the measured output, the longer the low dips the worse, the higher the overall variance the worse, ... Not a single reviewer uses such metric and people in general don't want to agree upon such things. Best bet would be to offer a frame time measurement application that offers this metric as it's output, since most reviewers use commonly available (often poor) measuring apps and do not develop their own. Such as FRAPS that will only measure something but not everything, plus it's hooking (as any other hooking/injecting app) can break an application.

At best a complex, expensive solution that captures GPU output is needed. There is FCAT but it's Nvidia's solution = not trustworthy and as far as I know it also hooks/injets to add colored borders/markers and what not.
 

·
Premium Member
Joined
·
10,020 Posts
Discussion Starter #18
An output that jumps between 10fps and 300fps fast and constantly, will look pretty bad despite having 155fps average fps. What is a threshold of perception, it differs for each observer.

Min max has always been fairly useless metric. Hence some use .1% and 1% lows instead eventhough having precise time plot and a histogram is the way to go. It is far easier for people to understand and for reports to show a single number than to try and cram 40 plots onto one page/video frame. Some reviewers offer plots for some tests. Guru3D.

.1% and 1% suffer the same issue as min/max as well in that the time length of spikes is important as the longer it is the more perceptible it will be. All in all a custom metric would have to be agreed upon by reviewers that would return a quality of the measured output, the longer the low dips the worse, the higher the overall variance the worse, ... Not a single reviewer uses such metric and people in general don't want to agree upon such things. Best bet would be to offer a frame time measurement application that offers this metric as it's output, since most reviewers use commonly available (often poor) measuring apps and do not develop their own. Such as FRAPS that will only measure something but not everything, plus it's hooking (as any other hooking/injecting app) can break an application.

At best a complex, expensive solution that captures GPU output is needed. There is FCAT but it's Nvidia's solution = not trustworthy and as far as I know it also hooks/injets to add colored borders/markers and what not.
Appreciate your well thought out replies.

Seems like we agree there should be a better way .

It does bother me to see the " experts" award the performance crown based upon those figures.

On the other end of the spectrum I've also seen scenarios where maximum fps in a benchmark skews the average fps so badly it makes them less useful as well.

Could also filter out those absolute high fps numbers of questionable value and see how that affects cpu comparisons for average fps.

Putzing around with excel spreadsheets is such a crappy winter's day project..... it may be a while before I tackle that.
 
1 - 18 of 18 Posts
Top