As some of you are aware, The Tech Report have been using a new type of analysis on FPS to analyse frame latencies (see http://techreport.com/review/21516/i...e-benchmarking).

To do this yourself I have devised a methodology (which could be automated) using Doom 3.

Step 1:
Record frame timings (I have used the command from the OpenBenchmarking Doom3 Test Profile). The important bit is the com_speeds flag:

Code:
./doom3 +exec doom3-pts.cfg +set sys_VideoRam 512MB +set r_mode -1 +timedemoquit demo1 +set com_speeds 1 > doom3.log
Step 2:
Parse log file using this Python script:
Code:
import re

# Compile reg exp
re1='.*?'	    # Non-greedy match on filler
re2='(\\d+)'	# Integer Number 1
r = re.compile(re1+re2+re1+re2+re1+re2+re1+re2+re1+re2,re.IGNORECASE|re.DOTALL)

# Open log file
frames = []
f = open('doom3.log')
for line in f.readlines():
    if line.startswith('frame:'):
        m = r.search(line)
        if m:
            frames.append( {'com_frameNumber': int(m.group(1)),
                            'com_frameMsec':   int(m.group(2)), 
                            'time_gameFrame':  int(m.group(3)), 
                            'time_frontend':   int(m.group(4)), 
                            'time_backend':    int(m.group(5))} )
f.close()

# Write to data file
f = open('data.csv', 'w')
f.write('Frame,FrameTime\n')
for frame in frames:
    # Skip first frame
    if frame['com_frameNumber'] == 0:
        continue
    f.write('%i,%i\n' % (frame['com_frameNumber'], frame['com_frameMsec']))
f.close()
Step 3:
Use this R script on resulting csv file:
Code:
dat <- read.csv("data.csv",header=T,colClasses=c("integer","numeric"))
summary(dat)

# Progression of frame times
plot(dat, type='n', xlab="Frame number", 
    ylab="Frame time in ms (lower is better)")
lines(dat)

# Average fps
max(dat$Frame)*1000/sum(dat$FrameTime)

# Percentiles
# It's the point below which 99% of all frames have been rendered. 
# We're simply excluding the last 1% of frames, many of them potential outliers, 
# to get a sense of overall smoothness. 
quantile(dat$FrameTime, .99) 

# Frame latencies by percentile
p <- seq(0, 1, length.out=1000)^(1/3)
quan <- data.frame(q = quantile(dat$FrameTime, probs = p), prob = p)
plot(quan$prob, quan$q, type='n', xaxt='n', xlim=c(0.5,1),
  xlab="Proportion of frames rendered", ylab="Frame time in ms (lower is better)")
axis(1, at=seq(0,1,by=.05), labels=paste(100*seq(0,1,by=.05), "%") )
lines(quan$prob, quan$q)

# Time spent beyond 50 ms
sum(subset(dat,FrameTime>50)$FrameTime)

# Time spent beyond 16.7 ms
sum(subset(dat,FrameTime>16.7)$FrameTime)

Example of resulting images: