Defining The Phoronix Test Suite [Archive]

View Full Version : Defining The Phoronix Test Suite

phoronix

02-07-2008, 04:00 PM

Phoronix: Defining The Phoronix Test Suite

It's going on four years that we have been providing benchmarks under Linux at Phoronix, and through this time the process has evolved as more applications have come about that gauge Linux system performance and meet our criteria for use in hardware reviews. While our internal process is continually refined, we have received requests for creating a Linux benchmarking manifest so that interested parties are able to easily reproduce our tests for comparative purposes and know everything that's involved. With that said, we are now working to establish the Phoronix Test Suite that will consist both of detailed documentation defining our hardware benchmarking protocols and free software that automates much of this work.

http://www.phoronix.com/vr.php?view=11813

Regenwald

02-07-2008, 04:29 PM

really, really great idea. i'm going to think about several components for such a benchmark, as soon as i've such results, i'm going to write them here :)

hmmm

02-07-2008, 05:54 PM

what licence will this be - gpl?

Michael

02-07-2008, 06:05 PM

what licence will this be - gpl?

GPLv3 likely (chance it could be MIT/X11 but most likely GPL)

joshuapurcell

02-08-2008, 01:45 AM

Way to go, this is a great idea. I'm sure this will go far to create a standard benchmarking tool for Linux.

SavageX

02-08-2008, 09:34 AM

Are there any free games in there? OpenArena, Cube2, Nexuiz (shameless plug)? What about e.g. oggenc?

Michael

02-08-2008, 09:37 AM

oggenc is in there.

SavageX: Last I checked out the Nexuiz benchmarking capabilities, I wasn't too impressed to be honest. Have they improved as of late? Last time I looked into it was about a year ago. If you're able to shed anymore light on it I'd be more than willing to investigate it further.

SavageX

02-08-2008, 10:30 AM

SavageX: Last I checked out the Nexuiz benchmarking capabilities, I wasn't too impressed to be honest. Have they improved as of late? Last time I looked into it was about a year ago. If you're able to shed anymore light on it I'd be more than willing to investigate it further.

As basically all Quake-engine based games it has the nice timedemo command. Try e.g. "timedemo demos/demo1". I think it should even work directly at game startup: "./nexuiz-linux-whatever-binary +timedemo demos/demo1"

Or you can create a benchmark config file (e.g. "benchmark.cfg" in the Nexuiz "data" directory):

timedemo/demo1
timedemo/demo1
timedemo/demo1
timedemo/demo2
timedemo/demo2
timedemo/demo2

and do a "./nexuiz-linux-bla-bla +exec benchmark.cfg" to automate testing.

Each run will print to console (ingame and to stdout) statistics like min/avg/max framerates.

We'll release a new version of Nexuiz hopefully later this month (see http://emptyset.endoftheinternet.org/~polzer/nexuiz/builds/ for SVN builds) which will introduce (optional) rendering features like reflective water with refraction - this will make sure even fast cards are kept busy ;-)

(Plus we finally have a much nicer menu ;-) )

curaga

02-08-2008, 12:15 PM

Where did you find RAMspeed? I can't locate it anywhere, google doesn't find it, nor is it in Debian's repos..

Michael

02-08-2008, 12:23 PM

Where did you find RAMspeed? I can't locate it anywhere, google doesn't find it, nor is it in Debian's repos..

http://www.alasir.com/software/ramspeed/

Vadi

02-09-2008, 01:48 AM

Awesome. I've been using Hardinfo for benchmarking, but their benchmarks are a bit weird (author said they'll need to be re-designed - but maybe they could just include yours. Hardinfo also has a central server so you can compare your benchmarks to other setups anonymously).

Michael

02-09-2008, 11:42 AM

I suggest you a crossplatform test. Not all the benchmark - better if they are -, but at least a part of it.

If the same test can be run over windows / Linux / MacOS / and others with a single number score, It would be better.

Then the driver development across the time and systems would be better, at least to watch.

For example a KDE4 test (it is cross-platform), and some similar as VLC tests for videos, Quake tests for games etc.

Opengl test would be a great tool to compare between drivers and systems.

We will find that, best sysmark - only windows - results may be not best with Linux or Mac OS.

And perhaps in a future some systems under Linux score high than under windows with driver improvements, especially for games.

I should put office tests with openoffice, browser tests with Firefox and opera, and of course on that tests Linux and MacOS should win windows, HD and RAM ones probably too. The lack are graphic drivers, and ATI and NVIDIA are working well now opening their drivers, but it is not enough.

I wish this benchmark become a world reference one for non directx benchmarks for every OS.
I would like to see your scores at any other computer analitics web page, as anandtech, tom's hardware, extremetech, 4gamers and similars.

To address this question of multi-platform compatibility, the Phoronix Test Suite software is compatible with Solaris and I'd anticipate it even working on the *BSDs.

However, the Phoronix Test Suite specification will be specifically designed and tested for Linux. Sure, you can apply the same testing practices to Windows or any other OS for that matter, but the chosen benchmarks and accompanying documentation will be all geared for Linux testing.

christian_frank

02-10-2008, 01:10 PM

Michael

02-10-2008, 01:35 PM

Hi Michael,

i got a new 8800gt this week and i needed some benchmark to compare it with my 7800gt. And i found a wonderfu OpenGl Benchmark (very gpu heavy, shader heavy).

Its called "Fur" from ozone3d and its freeware:

http://www.ozone3d.net/benchmarks/fur/

Its made for windows, but it runs 100% and with 0%, and i really mean 0% performance loss in wine-0.9.55.

My windows score: 2339
My wine score: 2344

I think this benchmark would be great for the testsuite.

What do you think ?

Best regards,
Christian

Hi Christian,

Thanks for the link regarding Fur. However, at this point no WINE-dependent benchmarks will be included with the Phoronix Test Suite specification itself. It could be supported by the PTS software though. Right now with new WINE releases coming out every two weeks, it's quickly changing and really wouldn't present itself as a good candidate in everyday benchmarking until it settles down in the future.

Then relying upon a Windows test in a Linux test suite would also need to be thought about and debated.

Regenwald

02-11-2008, 03:03 PM

hm, perhaps sauerbraten2 is the way to go? http://en.wikipedia.org/wiki/Sauerbraten_%28video_game%29
at least for graphic tests/benchmarks, this game seems to be a good choice, it uses actual techniques such as hdr-rendering etc.
the licence is also gpl-compatible. don't know if they do have a benchmark or a timedemo built-in, but you could ask the devs. it is written using sdl, so it is platform independent. have a look at it :)

another idea is to use superpi. this benchmark is really popular all around the world^^. of course it is primary for cpus, but better than nothing. you could calculate the first 20mio decimals with it and measure the time..

SavageX

02-12-2008, 02:52 AM

Personally I don't consider superpi to be a meaningful benchmark. Yes, it gives something like a score, but what is calculated is very very special. Superpi is nothing one "works" or "games" with and its results cannot easily be used to judge how this CPU fares with any different workload.

Same goes for basically most low-level benchmarking tools (e.g. memory speed tests) - what matters more IMO is "real" application performance.

Tillin9

02-12-2008, 11:13 AM

Might I suggest some developer help towards GL O.B.S. http://globs.sourceforge.net/about.php

It would be nice to be able to test individual OpenGL features more or less independently. Not only for user benchmarking but for driver development (see the Intel EXA talk from linux.conf.au? http://linux.conf.au/programme/detail?TalkID=167). Breaking down and benchmarking individual features is a great way to find trouble spots.

I know Phoronix knows of GL O.B.S. as v0.1 was used in an old X300 vs. Intel Q965 comparison. Of course, right now its highly alpha so it would need a great deal of work before inclusion.

bogdanbiv

02-12-2008, 09:11 PM

GPLv3 likely (chance it could be MIT/X11 but most likely GPL)

Totally awesome! First i had thought the Test Suite will be released under a proprietary license, and I came to the forum to ask if this is right. And now I see GPLv3.

I'm totally knocked off! :)

Can't wait to see distros shipping up packages with Phoronix Testing Suite.

deanjo

02-13-2008, 01:53 AM

Why not add in benches of

Blender3D rendering, an alternative for the 3dsMax benches suites use (heck even use the project files from Elephants Dream)

GIMP running though a series of filters and scripts much like other bench suites use Photoshop

POVRay benches

MySQL(or postgresql) benchmarks

x264 encoding benches (perhaps utilizing various filters in an app like avidemux2)

These apps would be a far more accurate representation of performance then any lame RAMspeed or hdparm test.

As well they should be compiled in 32-bit and 64-bit with all applicable flags.

(Oh gee look at that, all free software and all crossplatform)

I would stay away from propriatary apps like Quakewars. Keep the suite free of non OSS. Use one of the many free fps's instead so everybody can use it and it can be included with linux distributions.

georgia_tech_swagger

02-14-2008, 11:38 AM

Michael

02-14-2008, 11:47 AM

Will this benchmark suite have a cumulative scoring system, similar to PCMark?

The ONLY THING out there for Linux similar to PCMark is Geekbench. Wahoo Geekbench is cross platform and covers a multitude of tests. Booooo it's proprietary and you have to pay to run 64-bit versions.

I am so disgusted with the lack of a PCMark-like benchmark on Linux... I'm seriously considering developing a cross platform PCMark-lite suite if the Phoronix suite doesn't do score indexing.

If it doesn't, would the Phoronix guys be interested in helping develop such a suite?

Right now there's nothing on the road-map for coming up with a composite scoring system, however, that's not to say it can't be done. Once the framework for the software is in place, it can then be planned and implemented, since that's where we're at right now.

You're more than welcome to start coming up with ideas and most likely could be integrated into the Phoronix Test Suite.

Michael

02-22-2008, 03:27 AM

I hoped to start pushing out some things this week, but was busier than expected... Should have the start of it being pushed soon though, just some code cleaning to do and finishing off various bits and the graphics profiles.

Vadi

02-22-2008, 08:52 AM

That's very quick anyhow - but thanks for the update.

Michael

03-31-2008, 11:56 AM

As basically all Quake-engine based games it has the nice timedemo command. Try e.g. "timedemo demos/demo1". I think it should even work directly at game startup: "./nexuiz-linux-whatever-binary +timedemo demos/demo1"

Or you can create a benchmark config file (e.g. "benchmark.cfg" in the Nexuiz "data" directory):

timedemo/demo1
timedemo/demo1
timedemo/demo1
timedemo/demo2
timedemo/demo2
timedemo/demo2

and do a "./nexuiz-linux-bla-bla +exec benchmark.cfg" to automate testing.

Each run will print to console (ingame and to stdout) statistics like min/avg/max framerates.

We'll release a new version of Nexuiz hopefully later this month (see http://emptyset.endoftheinternet.org/~polzer/nexuiz/builds/ (http://emptyset.endoftheinternet.org/%7Epolzer/nexuiz/builds/) for SVN builds) which will introduce (optional) rendering features like reflective water with refraction - this will make sure even fast cards are kept busy ;-)

(Plus we finally have a much nicer menu ;-) )

SavageX: Is there a command for Nexuiz similar to timedemoquit on the Quake games? Where when benchmarking is completed and results written to stdout that the game will immediately quit? I'm writing a Nexuiz 2.4 profile for the initial PTS release right now.

Kano

03-31-2008, 12:34 PM

Michael

03-31-2008, 12:44 PM

nexuiz -benchmark demos/demo1

result shown or in

.nexuiz/data/benchmark.log

PS: Als long as you run free games - best those which are packaged in Debian I could even create a live cd for testing... nexuiz (without music) + gl2benchmark fits on one cd iso.

The benchmark argument doesn't work in Nexuiz 2.4. The timedemo argument works, but it doesn't quit automatically. Aside from that, the profile is working.

Matthew Tippett has already recommend creating a Live CD/DVD, so maybe your offer will be used once the GUI for PTS is in place, etc.

Kano

03-31-2008, 12:49 PM

It works with nexus 2.4 - 100%. I am using my sid backport on etch:

ii nexuiz 2.4-1 A fast-paced 3D first-person shooter

Michael

03-31-2008, 12:58 PM

Doh, I did +benchmark instead of -benchmark... Yeah, it's now working. Thanks Kano.

SavageX

03-31-2008, 04:15 PM

I'd just like to add that it makes sense to run the (Nexuiz) benchmark more than once to make sure everything is cached and whatnot - but I guess this is already done.

chem

05-07-2008, 01:23 PM

To address this question of multi-platform compatibility, the Phoronix Test Suite software is compatible with Solaris and I'd anticipate it even working on the *BSDs.

However, the Phoronix Test Suite specification will be specifically designed and tested for Linux. Sure, you can apply the same testing practices to Windows or any other OS for that matter, but the chosen benchmarks and accompanying documentation will be all geared for Linux testing.

I am currently installing the 0.51 PTS on my Fedora 8 desktop. However, I would love to compare its results (on either the "universe" suite or even just a subset of tests) to my MacBook Pro laptop, running OS X 10.5. As OS X is certified Unix '03, BSD-based, and has a variety of GNU/Linux programs available via Macports or Fink, I imagine it would be not too much trouble to get PTS working (at least in part!) on OS X.

I think this would be an incredibly valuable contribution. You could even compare benchmark scores on the same hardware (macbook pro, in my case) booted to Linux vs OS X. Currently, I do not know of a good suite to do this.

While I understand that Phoronix makes their money from Linux, I hope that Phoronix reconsiders and at least makes a perfunctory effort to make their PTS compatible with OS X 10.5. It may even be a wise business or PR move. :)

Thanks.

Michael

05-07-2008, 01:26 PM

chem

05-07-2008, 01:34 PM

chem,

I really wouldn't mind porting it to Mac OS X, but right now I have no Mac setups at all... Therefore inhibiting that work. Perhaps when I have the time and extra resources I'll pick up a Mac and start that work.

Thanks for the quick reply. If you ever need someone to test a release on OS X 10.5 (Intel), just send me a message. Good luck!