Results 1 to 5 of 5

Thread: How to account for "time taken to complete a single run" while comparing the results?

  1. #1
    Join Date
    Jul 2012
    Posts
    4

    Default How to account for "time taken to complete a single run" while comparing the results?

    I have two machines that I would like to compare each other with using PTS. Say, machine1 gives the result 80 transaction/sec for Postmark benchmark and the time taken to run is 52 minutes and machine2 gives 110 transactions/sec but the time taken to run is 1hr and 10 minutes. My question is if I don't consider the time taken, machine2 is better but what happens otherwise? Do we have to take the time into consideration or not? Why?

    Thanks!

  2. #2
    Join Date
    Feb 2008
    Location
    Linuxland
    Posts
    5,179

    Default

    Isn't the fluctuation there about the standard deviation?

    Your machine2 had more variation in the result, so to get reliable results PTS ran the test more times on there, is my guess.

    You shouldn't take the runtime into account.

  3. #3
    Join Date
    Jul 2012
    Posts
    4

    Default

    The standard deviation is there only for the scores and not for the time right?

    I'm not sure what you mean in your second line. I ran the test several times and I constantly get the same results I posted in my earlier thread oth on machine1 and machine2.

    So you are saying in my example even though machine1 took less time, machine2 is better because it has higher transactions/sec, am I right?

  4. #4
    Join Date
    May 2007
    Location
    Third Rock from the Sun
    Posts
    6,584

    Default

    The standard deviation is there only for the scores and not for the time right?
    Yes the standard deviation only applies to the scores of the test. Now if it is "better" or "worse" is debatable but it shows that for some reason the results were not consistent enough to make a determination to somewhat accurately give you a result representative of the performance. Ideally when benchmarking you would have little deviation otherwise with a large deviation outside influences can be effecting your score and masking the true performance of the system.

  5. #5
    Join Date
    Jul 2012
    Posts
    4

    Default

    Thanks for the reply. I do undertand the purpose of standard deviation. The scores from both machine have std. dev. less than 5%. So I don't have a problem with that. My only problem is the time taken for the benchmark to run. It takes longer on one machine and shorter on the other. Should I just neglect it?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •