The Importance Of Benchmark Automation & Why I Hate Running Linux Games Manually

[phoronix]

Phoronix: The Importance Of Benchmark Automation & Why I Hate Running Linux Games Manually

Yet again with today's GeForce GTX 1080 Linux review there were multiple people asking "why XYZ Linux game wasn't tested", a recurring topic now over the past several years...

http://www.phoronix.com/scan.php?page=news_item&px=Why-Test-Automation

[davidbepo]

wow, i didnt think it was so important

keep up with good work

[peppercats]

Have you tried to contact Feral to see if they'd work with you on this? They're usually very responsive to the linux community.

[Michael]

Have you tried to contact Feral to see if they'd work with you on this? They're usually very responsive to the linux community.

I communicate with them regularly whenever there is a new release and they do provide review keys to Phoronix. Generally they have just said they don't port the CLI functionality to Linux. In the times they do such as with the F1-2015 release, it actually looks like they did it unintentionally and they just said using the -benchmark switch is "line launch option is unsupported and untested. So any results and issues you might be experiencing are due to the unsupported nature of what you're trying to achieve."

Thus why I explicitly mentioned them in there as if they have more from the Linux community requesting said functionality be ported from Windows, hopefully it will happen...

[eydee]

I communicate with them regularly whenever there is a new release and they do provide review keys to Phoronix. Generally they have just said they don't port the CLI functionality to Linux. In the times they do such as with the F1-2015 release, it actually looks like they did it unintentionally and they just said using the -benchmark switch is "line launch option is unsupported and untested. So any results and issues you might be experiencing are due to the unsupported nature of what you're trying to achieve."

Thus why I explicitly mentioned them in there as if they have more from the Linux community requesting said functionality be ported from Windows, hopefully it will happen...

Consider politely reminding them that a very large amount of the marketing work for these linux ports is (unintentionally) done by Phoronix through the continuous benchmarks. They should consider porting these features paid, and the investment to return through the additional sales publicity causes.

And really, is there any frequent Phoronix reader who couldn't list all the usual test programs by heart? I doubt it.

[Michael]

Consider politely reminding them that a very large amount of the marketing work for these linux ports is (unintentionally) done by Phoronix through the continuous benchmarks. They should consider porting these features paid, and the investment to return through the additional sales publicity causes.

And really, is there any frequent Phoronix reader who couldn't list all the usual test programs by heart? I doubt it.

I've mentioned it to them many times... but without actual Linux gamers mentioning it, probably won't be too worthwhile for them to act.

[SystemCrasher]

Automation is the victory. That's why we love *nix way, its all about ability to automate your systems. Say, windows is really poor at it, so it is pain for advenced users, admins, devs and so on and good luck to do benchmarks and create some reports for the web without shitload of manual labor. When someone works 100h a week, his time clearly matters and if someone is refuses to respect this, IMHO it is perfectly fair NOT to benchmark things like this. So let 'em get some lack of PR. Phoronix got quite popular I guess, so it probably costs these companies some money

[sheepdestroyer]

Your reliance on automation is totally understandable ; however in all those bullet-points you listed, you forgot maybe the most important one :
- Relevance. Without testing state of the art modern game engines, actually used by commercial products, your benchmarks do no in fact provide any real understanding of how the hardware will actually perform in real life. Isn't that the goal of any benchmark? Your GFX benchs usually fall short of that sadly.

[gutigen]

I've mentioned it to them many times... but without actual Linux gamers mentioning it, probably won't be too worthwhile for them to act.

Considering the quality of their ports no wonder they don't want to port said functionality

[kaprikawn]

Your reliance on automation is totally understandable ; however in all those bullet-points you listed, you forgot maybe the most important one :
- Relevance. Without testing state of the art modern game engines, actually used by commercial products, your benchmarks do no in fact provide any real understanding of how the hardware will actually perform in real life. Isn't that the goal of any benchmark? Your GFX benchs usually fall short of that sadly.

Maybe I don't want to know how many FPS I'll get in Tomb Raider. Maybe I want to know the relative performance of the Nvidia blob vs Nouveau. Or Nouveau vs AMDGPU. Or Ubuntu vs Fedora. Or the last kernel/mesa vs git kernel/mesa. Where else on the web do you get a more comprehensive view than on Phoronix?

And I get an idea of real life performance by looking at the relative performance of the stuff Michael tests against what I've seen before. I don't need to know that my card will get an average of 56FPS on Tomb Raider, I can look at the performance on something Xonotic and I know roughly the relative performance of that test against more modern games. It doesn't have to be exact.