What Shall We Benchmark Next? Let Us Know!
Phoronix: What Shall We Benchmark Next? Let Us Know!
With Phoromatic we make it easy to build a test farm for benchmarking and automated regression management and to prove it we began monitoring the Linux kernel performance on a daily basis where we track the performance of the latest Linux kernel code on a daily basis using multiple systems. This has been going on for months and then yesterday we announced that Phoromatic reached a 1.0 status and that we have launched Phoromatic Ubuntu Tracker, a way to monitor the performance of Ubuntu Linux as a whole on a daily basis by benchmarking the most recent development packages. A day has passed so now we are already thinking of what next to add to our test farm for continuous performance tracking...
Why not do some testing of popular applications on Wine on a daily basis? Wine's git HEAD is rather stable (Alexandre does unit tests + compile test before he commits anything) so that should work fairly well.
Wine for sure, I use it every day for Warcraft 3, Starcraft one and two. And I think I'm not the only one
Both opengl and directx may be very interesting on different cards/drivers.
i guess anything that might help in finding regressions in any way. but it ll be hard to find anything that evolves as fast as the kernel, so that daily snapshots would make sense.
what im missing personally is to see other hardware drivers being benchmarked, like WLAN cards, or chipsets. i would like to see if there are differences in reliability and performance, and if theres any development.
maybe different programs: wicd vs. network manager
that would be rather ideas for testing once in a while.
But a transfer rate/data loss/signal strengh tests for instance would be interesting.
i have the feeling its being neglected a bit.
Few weeks ago, I asked if it was possible to bench KDE4 vs KDE3 as it was said in the early devs of KDE4 that it would consume less memory than KDE3.And then no news about it.
I'm just curious
My post: http://www.phoronix.com/forums/showthread.php?t=22401
FreeBSD and Networking
Why don't you track the performance of another kernel, such as FreeBSD? It will be interesting to compare the development approach taken by the different camps. Does FreeBSD experience performance regressions (as Linux recently did with ext4fs)? Does FreeBSD get faster or slower with every release?
I wouldn't expect UFS2 to change much as it is tried and tested (although the introduction of Soft Updates + Journaling [SUJ] might change that). ZFS, on the other hand, may experience significant fluctuations as fixes and improvements are implemented. The kernel is also slowly moving away from the GIANT lock (similar to the python GIL) so that may result in performance increases (or decreases).
Another area of benchmarking that I do not recall Phoronix doing is on the network stack. How does Linux, FreeBSD and other OS's compare in TCP/IP(6) throughput, load balancing, firewalls, etc. How resilient are the kernels to DOS attacks and other maliciousness.
With clang maturing and FreeBSD taking a close look at adopting it that might be another area for consideration. I would expect the compilation time, using clang, to increase as more features (especially C++) are implemented yet the runtime (of the compiled binaries) should decrease as more optimizations are implemented. This could also be compared to GCC 4.x progression.
I'm amazed that you have got a tool to do this. way to go!
I''d like to see how Wine and gfx is improving, but that would be a bi-weekly event if it's remains released each fortnight.
Is this request viable?
A wine benchmark would be cool, if you could base it on a native game, which also has an openGL and a Direct3D renderer for the Windows version.
(ie Unreal 2004)
So you could then test with lower spec cards from AMD and NVidia, and whatever graphics Intel have, and see what it looks like with the native performance, and the wine performance in the drivers with Direct3D and openGL.
Unlikely to do all of that for a tracker unless there is significant upstream interest in the tracker and also an increase in use of the affiliate shopping links / Premium subscribers / anything to justify the electricity for a number of systems dedicated to that task.
Originally Posted by poofyyoda
The question here really is to find something with an open development model, regular updates and relevance to people and companies.
Once thing that comes to mind would be to benchmark android.
I don't know if xorg-eddgers ppa is being benchmarked along with the newest kernel.
Tags for this Thread