Phoronix Test Suite 1.0 Release Blockers

[Michael]

In this thread propose features and bug fixes you would like to see addressed in Phoronix Test Suite by 1.0 "Trondheim" stage...

[deanjo]

I would consider moving the project to the opeensuse build service where it could be built for many distro's, many architectures and you would then not have to worry about bandwidth for the larger stuff.

[Michael]

I would consider moving the project to the opeensuse build service where it could be built for many distro's, many architectures and you would then not have to worry about bandwidth for the larger stuff.

It's PHP, it's a script... Or are you referring to some other aspect of it? How would the OpenSuSE Build Service address the bandwidth needs for larger stuff?

[deanjo]

It's PHP, it's a script... Or are you referring to some other aspect of it? How would the OpenSuSE Build Service address the bandwidth needs for larger stuff?

Most of the projects that you use already have repo's there compiled for various distro's and arch's. Using the build service would allow you to make installable packages for all the major distro's so all they have to do is add the repo to their favorite package manager and download.

[Michael]

The problem with using the distro's package for the test itself is that there is no control over any compile-time optimizations or any other changes made by that package maintainer that could skew the results compared to other distro packages or the vanilla source. However, if you look at the latest development work for trondheim-pts you'll see the "External Dependencies" feature, where the distribution's packaging system is now being used in some cases for obtaining GLUT for instance as well as some other dependencies for the benchmark but not the benchmark itself.

[deanjo]

The problem with using the distro's package for the test itself is that there is no control over any compile-time optimizations or any other changes made by that package maintainer that could skew the results compared to other distro packages or the vanilla source. However, if you look at the latest development work for trondheim-pts you'll see the "External Dependencies" feature, where the distribution's packaging system is now being used in some cases for obtaining GLUT for instance as well as some other dependencies for the benchmark but not the benchmark itself.


There is nothing stopping you from popping the source from your favorite project and setting up the optimization flags you want on the build service. You can set up a complete repo of all the packages you use just for use with the test suite. You do not have to depend on other people's tweaks and mods at all. Making a live distro is a snap as well if you want with kiwi. I would really recommend you look into the power of the service that they are offering.

[Michael]

There is nothing stopping you from popping the source from your favorite project and setting up the optimization flags you want on the build service. You can set up a complete repo of all the packages you use just for use with the test suite. You do not have to depend on other people's tweaks and mods at all. Making a live distro is a snap as well if you want with kiwi. I would really recommend you look into the power of the service that they are offering.

It does sound more advanced than I had thought it was... I'll look into it.

[hwertz]

In this thread propose features and bug fixes you would like to see addressed in Phoronix Test Suite by 1.0 "Trondheim" stage...

I'd put a note about NUM_CPU_JOBS in the README, or rig something up to set it from /proc/cpuinfo or the like. Per this thread on mplayer compilation test crashing (http://www.phoronix.com/forums/showthread.php?t=8914) the default NUM_CPU_JOBS of just plain leaving it blank uses too much RAM for some systems. If the test is meant to simulate juggling around dozens of active processes, this would change the test too much though.

[Michael]

I'd put a note about NUM_CPU_JOBS in the README, or rig something up to set it from /proc/cpuinfo or the like. Per this thread on mplayer compilation test crashing (http://www.phoronix.com/forums/showthread.php?t=8914) the default NUM_CPU_JOBS of just plain leaving it blank uses too much RAM for some systems. If the test is meant to simulate juggling around dozens of active processes, this would change the test too much though.

This is fixed in git. It was just a simple regression when I accidentally renamed the NUM_CPU_JOBS to SYS_CPU_JOBS but didn't think about the profiles :( NUM_CPU_JOBS contains the number of cores (as parsed by /proc/cpuinfo) plus one. I'll likely push out PTS 0.3.1 today due to this crashing bug.

[Michael]

Bump for 1.0 feature requests / blocker bugs.

[apaige]

Is there a fixed sound file?

[Michael]

Yes, should be one of the original NIN WAV files. Found in git.

[apaige]

I assume <Description>time lame -h audio.wav audio.mp3 (45.0MB)</Description> is no longer accurate. Also, making a tarball is not necessary. Just gzip it, or even use FLAC for increased compression (since FLAC gets compiled anyway).

[apaige]

You should credit Nine Inch Nails with the proper license mention in LICENSE or something. Btw, an IRC channel to talk about that kind of stuff would be nice. FreeNode seems appropriate. http://freenode.net/

[Michael]

I assume <Description>time lame -h audio.wav audio.mp3 (45.0MB)</Description> is no longer accurate.

Fixed in git.

[Michael]

You should credit Nine Inch Nails with the proper license mention in LICENSE or something. Btw, an IRC channel to talk about that kind of stuff would be nice. FreeNode seems appropriate. http://freenode.net/

If you look at the new file that's downloaded, they are credited in a file dedicated to it. Though does it need to be mentioned in the PTS LICENSE file since it's necessarily not being downloaded every time and not a critical part? But it is indeed mentioned when you download it.

[apaige]

Oh no, the new WAV file is the fake 24-bit/96 kHz version that is going to be fixed: http://www.hydrogenaudio.org/forums/index.php?showtopic=63086&st=50&p=564079&#entry564079

Please use the 16-bit/44.1kHz copy: like I said, audio encoders are best tuned for CD-DA content, and that's what most people deal with on a day-to-day basis. If you want a ~100MB file, just join two songs together.

[apaige]

I'm in #phoronix on irc.freenode.net if you want to chat about it.

[apaige]

The correct license URL is http://creativecommons.org/licenses/by-nc-sa/3.0/us/, its title "Attribution-Noncommercial-Share Alike 3.0", and http://creativecommons.org/wired doesn't have anything to do with it.

[Michael]

Is the new copy of it available yet?

[apaige]

No, it hasn't been fixed yet. Anyway, that would be the 24-bit/96kHz version. The 16-bit/44.1kHz FLAC copy is fine. That's the one you should download.

[apaige]

Any news on this?

[Michael]

Are you proposing the FLAC is then downloaded every time? I'd prefer not to do that because then they need to convert it back to WAV so more stuff to download. Granted, in most cases PTS will already have downloaded the needed packages.

[apaige]

Well you're already compiling FLAC in the encode-flac profile. You could use the compiled binary to decode the FLAC file. But mostly I want to know if you finally have a correct 16-bit/44.1kHz audio file, WAV or otherwise. Have you downloaded the 16-bit/44.1kHz FLAC files of The Slip yet?

[Michael]

Yes, but it's encoding in half the time of the current file being used.

[apaige]

I don't understand: is the file smaller? If so, just join two tracks together! What's the hold-up here?

[Michael]

What's the hold-up here?

Just really low on time... will get it done soon.

[Michael]

Any last minute 1.0 requests?

[apaige]

Not really a blocker, but the openssl benchmark runs all RSA sizes (512, 1024, 2048 and 4096 bits), while PTS only parses the 4096-bit result. The test takes a lot more time to run than necessary. Replacing the benchmark command with the following would run only the 4096-bit test:
./apps/openssl speed rsa4096 -multi $NUM_CPU_CORES" > openssl

[Michael]

This will be fixed in git, thanks.