RFC: A Preview Of The Phoronix Graphs With Iveland

[Khudsa]

It's nice, really nice and elegant.

[ilm1]

when first viewing the plots I would use simply the extrema of the data or make that easily accessible, instead of the STD or SE. It's easier to understand for most readers, when more advanced metrics are needed you can use STD or SE.

Though if you want to go completely off the charts with information, use boxplots. Including median, mean, extrema and all other quantiles... very useful, but easily bloated. E.g., you can easily see if the data is distributed uniformely between the extrema or are more biased to one side...

[ilm1]

now that i think of it (couldn't edit my message). You could go with the current plots now (or perhaps a boxplot) and when the user clicks on a single bar a histogram or kernel density estimation is plotted to provide detailed information for that single experiment.

[xav_19]

now that i think of it (couldn't edit my message). You could go with the current plots now (or perhaps a boxplot) and when the user clicks on a single bar a histogram or kernel density estimation is plotted to provide detailed information for that single experiment.

maybe something simpler, like if you plot the median just go with min/max and if you plot the mean, just use the std.
in my mind, it's clearer this way, something like raw data vs. aggregated data, but i'm not a statistician...
my 2 cents

[Xanikseo]

What about a way to order the results, eg with the best at the top going towards the worst at the bottom? That way it would be much easier to compare similar results, as with the current system we have to rely on our eyesight to be able to tell us which bar is smaller/bigger.

[Michael]

What about a way to order the results, eg with the best at the top going towards the worst at the bottom? That way it would be much easier to compare similar results, as with the current system we have to rely on our eyesight to be able to tell us which bar is smaller/bigger.

That would get rather confusing as each bar graph then would have a different order to them..

[ilm1]

maybe something simpler, like if you plot the median just go with min/max and if you plot the mean, just use the std.
in my mind, it's clearer this way, something like raw data vs. aggregated data, but i'm not a statistician...
my 2 cents

yes, sure that's what i meant. show something simple first, like you said mean/average + std/extrema (99% of the time this will be enough).

But it is also good to have the more advanced, powerful plots nearby when something is awry. Histogram/kde/boxplots are good (statistical valid) methods to investigate further (e.g., outliers are easily spotted).

ps: This is just my opinion of course. I'm no statistician either
ps2: I believe statisticians also promote the use of the median rather than the mean, as it is less sensitive to outliers.

[Xanikseo]

That would get rather confusing as each bar graph then would have a different order to them..

That's what I meant though :P. By looking at the label second to the top, you would see immediately that it was second. You wouldn't even have to look at the bars to compare, just the order of the labels. The length of the bars will then show how close the results were.

Surely the graphs want to show which results do better than others, and the new feature which turns the top result a red colour does this, but it only shows what is first. We then have to put the effort into working out what came second and third and so on. This would be made easier if the results are already ordered.

If your concern is that a user would want to see the performance of one particular system configuration in every graph, and that it would therefore be harder to find that configuration if the order is different every time - that is the whole point. Instead of looking at the bar on the right every time and comparing the length of the bar to the others, you find the configuration you are interested in in the labels, and you would immediately be able to see that it came second or third without looking at the bar. You would then look at the bars to see how close it was to the other results.

Perhaps giving each configuration a different colour would make it easier to find each label when the order keeps changing?

[Xanikseo]

No edit button? Anyway..

Perhaps you wouldn't have to re-order the graphs, just put position numbers next to the labels or something; but it would be much nicer, easier and neater if the graphs were ordered.

[Michael]

That's what I meant though :P. By looking at the label second to the top, you would see immediately that it was second. You wouldn't even have to look at the bars to compare, just the order of the labels. The length of the bars will then show how close the results were.

Surely the graphs want to show which results do better than others, and the new feature which turns the top result a red colour does this, but it only shows what is first. We then have to put the effort into working out what came second and third and so on. This would be made easier if the results are already ordered.

If your concern is that a user would want to see the performance of one particular system configuration in every graph, and that it would therefore be harder to find that configuration if the order is different every time - that is the whole point. Instead of looking at the bar on the right every time and comparing the length of the bar to the others, you find the configuration you are interested in in the labels, and you would immediately be able to see that it came second or third without looking at the bar. You would then look at the bars to see how close it was to the other results.

Perhaps giving each configuration a different colour would make it easier to find each label when the order keeps changing?

The red highlight is for the Core i7 970 to highlight that and not to do with being first place or anything.

Do any other publications or anywhere else order their graphs based upon placement order? I can easily implement such support, but I just don't want it to be confusing for people.