Phoronix Test Suite 0.4.0 Released

[phoronix]

Phoronix: Phoronix Test Suite 0.4.0 Released

Following last week's release of Phoronix Test Suite 0.3.0 and the 0.3.1 hot-fix, a surprising amount of changes have wound up in this week's development release known as version 0.4.0. In fact, there are 21 major changes in this new release not counting general development work and minor bug fixes. Among the changes are new test profiles, a number of new PTS options, PTS External Dependencies support for new Linux distributions, reworking how results are saved, and the graphs are now rendered locally when saving the results.

http://www.phoronix.com/vr.php?view=12268

[Michael]

Test away... take note PHP GD is now required if you wish to see the graphs. Aside from that, hopefully no major regressions in this release...

[lordmozilla]

Wow progress is really fast on this, good to see, however do you really have to bump up the numbers so quickly? I mean are you trying to get to 1.0 as quickly as possible? By my accounts you should be on like 0.1.4 and not 0.4. Admittedly this is unimportant, but it seems silly to just jump so fast when there are not really all that many changes between some of the releases...

Ether way keep up the good work, just about to run a 0.4 universe benchmark...

Also how about a .deb package? and maybe .rpm, but it would be cool to have this product in the debian repo's! also it would help so people would be able to upgrade version easily... maybe your own apt repo?

[deanjo]

Also how about a .deb package? and maybe .rpm, but it would be cool to have this product in the debian repo's! also it would help so people would be able to upgrade version easily... maybe your own apt repo?

I have already suggested this in another thread. I suggested that Michael look into the openSuSE build service as it can build packages for:


SUSE
openSUSE 10.3
openSUSE 10.2
SUSE Linux 10.1
SUSE Linux Enterprise 10
SUSE Linux Enterprise 9
openSUSE Factory
Debian
Debian Etch
Red Hat
Fedora 8
Fedora 7
Fedora 6 + Extras
Red Hat Enterprise Linux 5
CentOS 5
Mandriva Linux
Mandriva 2008
Mandriva 2007
Mandriva 2006
xUbuntu
xUbuntu 6.06
xUbuntu 7.04
xUbuntu 7.10 xUbuntu 8.04With more being added soon.

[puntarenas]

Test away... take note PHP GD is now required if you wish to see the graphs. Aside from that, hopefully no major regressions in this release...

Very minor issue, but is there a way to have the graphs generated on a saved result (I had one universe run without PHP GD installed)?

PTS really improves very fast and I think you are working hard to stabilize the main features in the first place. However there are some suggestions I would love to see included some day:

Option: ignore-first-run

On some tests (e.g. audio-encoding) the first run really suffers from disk activity and counting it into average seems a bit odd to me.

3D settings detection

I don't know if it is possible to detect the driver settings of nvidia-glx or fglrx, but it would be nice to have AA and AF settings displayed in the results.


higher 4:3 resolutions

At least 1600x1200 would be nice, 1280x960 also seems to make sense.


additional run to measure the CPU limit of game tests

I would love to have an initial run of the game tests (maybe optional) with the lowest possible quality settings at 640x480 to determine the CPU limit of a game test. This should be displayed along with the final result (the "real" benchmark run).

:)

[Michael]

I mean are you trying to get to 1.0 as quickly as possible?

Nope... 1.0 will have all of the features as planned aside from the GUI. 1.0 should be very stable and offer a great platform for writing Linux tests. The GUI will likely be postponed from the 1.0 release since it's not being requested much as it's already very easy to use from the CLI. The target has been to have 1.0 out in late May or early June and we're still on schedule for reaching that mark.[/quote]

Also how about a .deb package? and maybe .rpm, but it would be cool to have this product in the debian repo's! also it would help so people would be able to upgrade version easily... maybe your own apt repo?

There will be a .deb package of the Phoronix Test Suite in Ubuntu's Universe repository for 8.10 Intrepid Ibex, but a Debian package may be available from Phoronix-Test-Suite.com starting with v0.5 or v0.6. There will be packages available too in the next version of Mandriva and likely many other distributions to come.

[Michael]

Very minor issue, but is there a way to have the graphs generated on a saved result (I had one universe run without PHP GD installed)?

There is no feature for that right now. Though one way to see it would be by just running phoronix-test-suite UPLOAD saved-result and then viewing it on PTS Global where you can see a graph generated there.

PTS really improves very fast and I think you are working hard to stabilize the main features in the first place.

Yep, thanks.

Option: ignore-first-run

On some tests (e.g. audio-encoding) the first run really suffers from disk activity and counting it into average seems a bit odd to me.

I've been meaning to add an option where a test maintainer can add this option in the XML profile for a test and it will ignore the first run. I'll try to get this done before the next release.

I don't know if it is possible to detect the driver settings of nvidia-glx or fglrx, but it would be nice to have AA and AF settings displayed in the results.

With NVIDIA, yes, that information can be easily polled. On the fglrx driver, I don't know of a clean method.


higher 4:3 resolutions

At least 1600x1200 would be nice, 1280x960 also seems to make sense.

Yep, another minor item that will be addressed shortly.



additional run to measure the CPU limit of game tests

I would love to have an initial run of the game tests (maybe optional) with the lowest possible quality settings at 640x480 to determine the CPU limit of a game test. This should be displayed along with the final result (the "real" benchmark run).

Yep, coming soon.

[puntarenas]

You rule :)

[lordmozilla]

I have to say i'm very impressed with how quick you are dealing with all these issues. A gui is pointless though, don't bother with it yet ;-)

Thanks for replying to all of our questions so quickly!

[Michael]

I have to say i'm very impressed with how quick you are dealing with all these issues.

That's just the rate at which I take care of business. :D

[lordmozilla]

That's just the rate at which I take care of business. :D

I just wish vmware support where half as fast as you, and we pay them!!

[Melcar]

pts-test1 still fails to upload. And anyone having problems running the Wine test?

[Michael]

I think I may know what's causing the pts-test1 failure and that's due to the size of what's being uploaded, but that requires writing a better way to upload the results.

[Michael]

Option: ignore-first-run

On some tests (e.g. audio-encoding) the first run really suffers from disk activity and counting it into average seems a bit odd to me.

With the latest commit today, now if a test maintainer adds <IgnoreFirstRun>TRUE</IgnoreFirstRun> to their test profile, it will automatically ignore the first result and will not use that in calculating the average.

[deanjo]

How about adding a video encoding test suite with mencoder since we compile mplayer anyways?

[Michael]

Following PTS 0.5.0/0.6.0, a majority of the focus will turn to cleaning up the existing profiles and adding in new tests along with the PCQS tests... (Phoronix Certification & Qualification Suite)

[deanjo]

Following PTS 0.5.0/0.6.0, a majority of the focus will turn to cleaning up the existing profiles and adding in new tests along with the PCQS tests... (Phoronix Certification & Qualification Suite)

So going at the rate your going I guess I can expect that in 3 or 4 days then :P

[Michael]

3D settings detection

I don't know if it is possible to detect the driver settings of nvidia-glx or fglrx, but it would be nice to have AA and AF settings displayed in the results.

New feature... Now if you're using a NVIDIA graphics card, and you run a graphics test from either a suite or individually, if you over-rode the graphics setting from the nvidia-settings utility, the overrode Antialiasing and Anisotropic Filtering levels will be reported under the test notes. This is now in git.

PTS now also can monitor the die temperature, but I haven't decided how/where to use that yet.

[puntarenas]

Amazing!

On the GPU temperature, maybe it would be enough to have another optional test note. It would be very useful to compare different graphics cards (think of cards with the same GPU and clock rates, but different cooling solutions). Minimum and maximum value would be adequate IMHO, I don't think average calculation or visualization in a graph would make much sense.

On the other hand, this could also be optional for other sensor readings (CPU temperature, mainboard temperature, fan speeds...). Having an overall command line option for reading availiable sensor Information would be great, but I have no idea how to present the results in a sophisticated way within every single graph or test note.

There is another thing that came to my mind which I am a bit concerned about. PTS generates a lot of additional traffic on external sites (think of all the Nexuiz downloads). Wouldn't it be fair to have those dependencies bundled in an archive and distributed throug bittorrent or something similar? Maybe an included bittorrent downloader would help to reduce traffic impact and be more gentleman like.

BTW, I wrote a german tutorial at wiki.ubuntuusers.de (http://wiki.ubuntuusers.de/Phoronix_Test_Suite) to spread the word. :)

[Michael]

MONITOR=cpu.temp,sys.temp,gpu.temp is supported now too, for CPU and system if you have LM_Sensors installed.

[Michael]

Emerging today in git is line graphing support of these monitor results too....

http://www.phoronix-test-suite.com/misc/1209308384-0.png

There are some minor bugs to work out in the line graphing and some other things to clean up, but it's working in git now. I also haven't decided whether to integrate any of these thermal monitor measurements into PTS Global. At this point I am thinking no.

[Michael]

Amazing!

...
There is another thing that came to my mind which I am a bit concerned about. PTS generates a lot of additional traffic on external sites (think of all the Nexuiz downloads). Wouldn't it be fair to have those dependencies bundled in an archive and distributed throug bittorrent or something similar? Maybe an included bittorrent downloader would help to reduce traffic impact and be more gentleman like.

Possibly... Though most of the big downloads (i.e. Nexuiz) are going through SourceForge and not little bootstrapped servers. In fact, it's the Phoronix-Test-Suite server that's already getting crunched.

BTW, I wrote a german tutorial at wiki.ubuntuusers.de (http://wiki.ubuntuusers.de/Phoronix_Test_Suite) to spread the word. :)

Thanks!

[Michael]

http://www.phoronix-test-suite.com/misc/1209325782-0.png

A few more monitor improvements... ACPI temperatures are also tapped into now too.

[puntarenas]

Will it generate such a curve for every single test? If it is an overall view, it would be nice to associate the readings with the corresponding test in some way.

Any chance to see the values as numbers in addition? As it is now, it is hard to read off the exact values?

(I'm sorry, but I cannot try out the GIT version right now, so I have to ask stupid questions :))

[Michael]

It will generate a graph as an overview of whatever you ran.... if you did phoronix-test-suite benchmark multicore it will do it for multicore, if you did it for phoronix-test-suite benchmark doom3, it will do it for just doom3. I can easily make it associate the readings, but I am not sure how to best put the information with the graph. For a long test, there is limited room between the points so all of the text would get scrunched together. It wouldn't be easy to read. If there were just like two tests that would be fine. Even if using time as in like the number of seconds in the test, that still might be too big.

Values could easily be added somewhere if the test isn't too long.

[abuster]

The file pts/benchmark-resources/build-imagemagick/install.sh contains a hardlink to ImageMagick download.

"wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick-6.4.0-7.tar.bz2 -O image-magick.tar.bz2", but ImageMagick-6.4.0-8.tar.bz2 is no longer in archive on FTP. Since the folder contains ImageMagick.tar.bz2 which always is linked to latest source, why not use it?

If you're worried by different test results, maybe a FTP with longer/older archive should be used.