( Continuing from the Phoronix Test Suite 0.9.9 message. )
Once Phoronix Test Suite 1.0.0 is released, new profiles and test suites will be accepted, etc. While open to discussion and changes, my plans are to let Phoronix Test Suite 1.0.x be the main focus for the next couple of months with point releases every few weeks or whenever the changes warrant a new release. There are a few major changes I can see for the next major release (likely to be called PTS 1.2 -- still under the Trondheim codename until 2.0), which will be detailed in the future. I am thinking that version may be ready for Q4'2007 depending upon what comes about (in regards to feedback, etc) after 1.0 is released.
In regards to what patches will be accepted for inclusion into Phoronix Test Suite point releases versus the ones that will be held off until the next major release, below is the criteria I am thinking at this point.
- New Suites / Test Profiles: They may be accepted into whatever new release comes next... New suites and test profiles shouldn't cause any regressions within pts-core, so as long as the XML profile doesn't rely upon a feature not available until the next major release, I am content with accepting it into the current branch. A new test profile may be delayed from a point release depending upon the timing and if it presents a new external dependency that requires the distro-xml files to be updated.
- Updated Test Profiles: Once a test has been stabilized, I think it's in everyone's best interest if there aren't major changes to it between point releases. Trivial changes such as changing the string description, capitalization of characters, cleaning up the script files, etc all should be fine. I am also fine with a test profile adding/extending new options in the test profile, as long as it remains backwards compatible and doesn't impact the comparison of existing results (i.e. listing new resolutions to run at would be acceptable).
In between point releases, however, I don't want a test profile that is a major update to the test software itself, removes options, or otherwise contains drastic changes that make the results incompatible. It should be pretty clear what changes may make such an impact.
Note to test maintainers: If you haven't looked at the pts-core merging code, results from two different versions of a test will merge if the test version is the same or only the minor version has changed (i.e. going from 1.0.2 to 1.0.3). If you jump from 1.0 to 1.1 with your version, the test results will not be compared/merged. Keep this in mind.
- Updated Test Suites: Changes to the suite information/description are fine between point releases. Depending upon the situation, I may allow new tests be added or removed from a suite between point releases, but if you have major changes planned, just align them with the next major release.
- XML Syntax/Specification: Changes are only allowed in __major__ releases, though that's not to say some tags coming up in a future release may appear early in an unsupported state (permitting the tag is optional).
- distro-scripts / distro-xml: I'll allow them to be updated whenever a change is required.
- pts-core: Patches for bug fixes, code clean-ups, etc always welcome. Patches that add new features to pts-core can be accepted between point releases permitting the risk of introducing new bugs are low, it doesn't require significant rework to other areas of pts-core, and doesn't remove any functionality. It's also at the discretion of me or a major contributor of whether to postpone a patch until the next major release.