Page 1 of 4 123 ... LastLast
Results 1 to 10 of 34

Thread: Why Software Defaults Are Important & Benchmarked

  1. #1
    Join Date
    Jan 2007
    Posts
    14,795

    Default Why Software Defaults Are Important & Benchmarked

    Phoronix: Why Software Defaults Are Important & Benchmarked

    Almost every time benchmarks are published on Phoronix, there's always at least a handful of people - or more - that will right away say the benchmarks are flawed, meaningless, or just plain wrong. Why? Because the software configuration is tested with its default (stock) settings. These users then go on to say that the defaults are not optimized for performance and that "everyone else knows better" to use a particular set of options, etc. But it's my firm belief that it's up to the upstream maintainer -- whether it be the project itself developing the software in question or the distribution vendor that's packaging and maintaining the given component -- to choose the most sane and reliable settings, and that's what most people use. In addition, with open-source software, there's endless possibilities for how a given piece of software can be tuned and tweaked. Here's some numbers confirming these beliefs of testing software at its defaults...

    http://www.phoronix.com/vr.php?view=OTE4OQ

  2. #2
    Join Date
    May 2007
    Location
    Third Rock from the Sun
    Posts
    6,584

    Default

    One thing that I do ask Michael is that when you bench openSUSE in your articles you list the exact kernel package installed. The installer for openSUSE will make a decision during install based on the test systems hardware configuration and it could install the "generic -default" kernel packages or the " -desktop" kernel packages.

  3. #3

    Default

    Quote Originally Posted by deanjo View Post
    One thing that I do ask Michael is that when you bench openSUSE in your articles you list the exact kernel package installed. The installer for openSUSE will make a decision during install based on the test systems hardware configuration and it could install the "generic -default" kernel packages or the " -desktop" kernel packages.
    It will be said on OpenBenchmarking.org always, now that all results are to be hosted there.

  4. #4
    Join Date
    Aug 2007
    Location
    Europe
    Posts
    401

    Default

    Maybe you could write an article about xorg.conf settings and how they affect performance, with actual data.

    I think the black magic of xorg.conf could need that.

  5. #5
    Join Date
    Jun 2006
    Posts
    311

    Default

    Quote Originally Posted by sabriah View Post
    Maybe you could write an article about xorg.conf settings and how they affect performance, with actual data.

    I think the black magic of xorg.conf could need that.
    Well, xorg.conf is supposed to be gone already (interesting when you think about the topic of the post .

    Michael and I have been pondering ways of doing that. It's not technically openbenchmarking specific, but effectively fuzzing the configuration options to find a miximal performance is an incredibly interesting idea.

    It applies equally well to compiler flag optimization and kernel option configuration too...

    I am sure Michael and I will have something baked towards the end of the year .

    Matthew

  6. #6
    Join Date
    Jul 2009
    Posts
    416

    Default

    As a Gentoo user, I can see the advantage to doing benchmarks with the default options.

    I use a AMD Athlon II X4, which doesn't get tested at all on Phoronix, so any benchmark that has Core i7 optimizations won't necessarily apply to me. I'd rather see what the base performance is, rather than have to figure out if something is fast merely because of i7 optimizations.

  7. #7
    Join Date
    Aug 2009
    Posts
    2,264

    Default

    For doing an Ubuntu benchmark I would agree. I'm also not going to spend a lot of time tweaking Windows.

    But when you are talking about looking at how a particular Linux patch/feature affects performance and then just benchmark the kernel with some apps in a serialised fashion (BKL removal), I do not agree as that benchmark would be pointless to the article. Using the results as a Kernel snapshot benchmark would be a different story.

    Michael it is your job as a journalist to investigate and not test. Testing might be a part of investigation, but it is not always set up to fullfill the investigation role.

    Just saying. I still like the website

  8. #8
    Join Date
    Jul 2009
    Posts
    258

    Default

    hmm telling micheal what his job is might be a bit rude i think
    anyway, this whole discussion id think leads to the interests.
    ones would like to see the topmost performance of developed software whether stable or not. just to see what is possible, how developers are doing. and as far as i remember phoronix defined itself as a regression finding site to help the devs right? so thats one point.
    a different one is to look at stock, for usability. testing a well known system thats reliable - mostly important for new users and well people in need of reliability. probably companies/vendors running linux not wanting to perform one liner patches on hundreds of desktops and/or laptops.
    those 2 groups just want totally differnt things. from the devs and regresion finding viewpoint "flaw in testing" seems correct to me, whereas the same would not apply from the general/noob user.
    id say one would need tests as done in the past. ones for default stack and ones for highly specialized tests on different machines. (like the ones enabling avm or whatever the name was - the vector graphics thing)

  9. #9
    Join Date
    Nov 2010
    Location
    Stockholm, Sweden
    Posts
    403

    Default

    Quote Originally Posted by sabriah View Post
    Maybe you could write an article about xorg.conf settings and how they affect performance, with actual data.

    I think the black magic of xorg.conf could need that.
    Well, I'll be damned, I was just about to say exactly what you said. A guide to xorg.conf settings (or xorg.conf.d/*) for

    fglrx
    nvidia-blob
    noveau
    intel, nvidia and ati xorg-drivers.

    and perhaps also some generic settings. I've been a Linux user since 2002 and I still use the default, except EXA (is that even the good thing to use anymore) and SoftwarePointer "No".
    Do we need to enable Composite in the extensions anymore? Etc etc etc.

    Most documentation is from the days before Compiz reached a fairly stable point (I still don't think they're there) and Compositing was done via XGL and other weird stuff.

    Pleaaaase, Master, teaaach us!

  10. #10
    Join Date
    Sep 2008
    Posts
    989

    Default

    It's true; stock configurations are all that 90% of the users will see. Even 80-90% of Phoronix readers are newbies, as taken from the previous years' Phoronix Linux Graphics Survey. And Phoronix readers are more savvy than the general populace.

    There should be a huge challenge afoot in the distribution marketplace to create optimized distros that, while able to run on old hardware, can adapt to new hardware and run fast there as well. I'm not just talking about SSE, but maybe, things like having certain programs (very performance sensitive programs) specially compiled on install? Not everything is performance sensitive, but certainly the 3d graphics related stuff is, and certainly the kernel is.

    I've seen a few binary distros around that purport to focus on optimization, but I think I'd rather have the mainstream distros focus on optimization. The mainstream distros are where most of the casual users are going, and we can afford to give them a better out of the box experience.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •