Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: Gosh, Our Test Farm Already Finds Big Regression

  1. #1
    Join Date
    Jan 2007
    Posts
    14,919

    Default Gosh, Our Test Farm Already Finds Big Regression

    Phoronix: Gosh, Our Test Farm Already Finds Big Regression

    Back in October we made it possible to autonomously find performance regressions in a project's code-base by leveraging the Phoronix Test Suite atop Git's bisect command. A new Phoronix Test Suite module was created that would automatically run a binary search on a defined code-base and keep running the specified test(s) on each revision and then traverse to the next using Git bisect until the lone commit causing the defined regression was located...

    http://www.phoronix.com/vr.php?view=Nzc5NA

  2. #2
    Join Date
    Jul 2009
    Posts
    12

    Default

    I'm curious if you could compare older versions with current kernels sometimes in the future. Just to see if 2.6.32 is really faster than 2.6.12/22/etc. (if possible - btrfs-performance in 2.6.12 wouldn't be very useful ;-) )

  3. #3
    Join Date
    Dec 2007
    Location
    Edinburgh, Scotland
    Posts
    580

    Default

    Seriously why cant you folk just raise a bug!!

  4. #4
    Join Date
    Jul 2008
    Posts
    69

    Default

    And you are holding off disclosing this because???? Seriously, what do you have to gain by not submitting this. Every day you wait brings the release date for .33 closer, and makes it less likely that there will be time to create/test patches.

    I don't even know what to call this move. It's not selfish because there's not really anything to gain/

  5. #5
    Join Date
    Jun 2006
    Location
    Portugal
    Posts
    530

    Default

    Quote Originally Posted by jbrown96 View Post
    And you are holding off disclosing this because???? Seriously, what do you have to gain by not submitting this. Every day you wait brings the release date for .33 closer, and makes it less likely that there will be time to create/test patches.

    I don't even know what to call this move. It's not selfish because there's not really anything to gain/
    .33 should still be 3 months away or something -- the merge window is still open and -rc1 hasn't even come out yet.
    And if it's a regression, then a patch fixing it would still be accepted outside the merge window, and basically anytime until the final release.

  6. #6
    Join Date
    Sep 2006
    Location
    PL
    Posts
    910

    Default

    And if it's a regression, then a patch fixing it would still be accepted outside the merge window, and basically anytime until the final release.
    waiting and not disclosing security bugs is bad.

    performance regressions are not as critical, but i think not disclosing them during merge window is a fatal mistake, because it might be very difficult to revert the problematic merge.

    I don't even know what to call this move. It's not selfish because there's not really anything to gain/
    publicity stunt.
    Last edited by yoshi314; 12-11-2009 at 05:06 AM.

  7. #7
    Join Date
    May 2008
    Posts
    99

    Default Compier-related suggestions

    Here are some suggestins:

    1) It would be nice to have tests on both gcc and icc (and LLVM?). Gcc-generated code quality messure over time, on real world SW, is missing.

    2) Tests for different versions of gcc, inluding ancient (3.x).

    3) Tests for default compiler settings vs. CPU type specific options.

  8. #8
    Join Date
    Oct 2007
    Posts
    912

    Default

    Quote Originally Posted by yoshi314 View Post
    waiting and not disclosing security bugs is bad.

    performance regressions are not as critical, but i think not disclosing them during merge window is a fatal mistake, because it might be very difficult to revert the problematic merge.

    publicity stunt.
    I'm normally quiet about this sort of thing, but I agree with the publicity stunt opinion. And this probably isn't the place to say it, but I do think Phoronix needs to have better wording in the articles to avoid that kind of impression - if I might exaggerate a little, less of "look aren't we great, here's sensationalist news" and more "here's something interesting".
    Don't get me wrong, a solid automated testing framework to discover regressions would be very useful - just the article reads too much like an opinionated blog.
    Of course, feel free to disagree with me, I won't be offended.

  9. #9
    Join Date
    Sep 2006
    Location
    PL
    Posts
    910

    Default

    i have to applaud phoronix for kernel test farm. this will definitely be helpful. also, we need more people concerned with linux desktop performance, as contrary to most kernel developers (as proven by Con Kolivas' experience).

    the only good reason for not disclosing a performance regression is that it has to verified in more test environments.

    and this probably isn't the place to say it, but I do think Phoronix needs to have better wording in the articles to avoid that kind of impression - if I might exaggerate a little, less of "look aren't we great, here's sensationalist news" and more "here's something interesting".
    the one in this article sounds to me this way - "here's some interesting performance regression in the linux kernel, but you have to wait one week when our next great product comes out to learn about it." . there's pretty much a marketing decision.


    phoronix should generally disclose the regression as quickly as possible. if it's real - kernel test farm will already prove a valuable tool, regardless of its release date - today, tomorrow or maybe even never (Coverity is extremely useful, but it was never released to the public)
    Last edited by yoshi314; 12-11-2009 at 12:14 PM.

  10. #10

    Default

    This delay is just not so the public interface can be finished up but to also ensure the issue can be reproduced on other platforms and that the commit that's isolated is indeed correct. This will hopefully be all completed by Monday so it's really not much of a delay.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •