- From: Michael Kay <mike@saxonica.com>
- Date: Thu, 10 Mar 2016 14:51:34 +0000
- To: Abel Braaksma <abel.braaksma@xs4all.nl>
- Cc: Public XSLWG <public-xsl-wg@w3.org>
Suggested addition: ### CHALLENGING AND CORRECTING TESTS ### If you believe that there is an error in a test, raise a bug report at https://www.w3.org/Bugs/Public/ - use product = "XPath / XQuery / XSLT", component = "XSLT 3.0 Test Suite". If you have write access to the repository, please obey the following protocol for changing existing tests: (a) Changes that can be made without formality * If it's a simple error in a test that you yourself committed very recently, you can change it without any record other than the commit message * You can also make non-substantive changes without formality, for example expanding the description of the purpose of a test or improving the layout, or fixing the catalog to be schema-valid. (b) Changes that should be marked by adding <modified by="your name" on="date" change="reason for change"/> below the <created/> element. * If you are confident that a test needs to change because the specification has changed, you don't need to raise a bug report: fix the test, but always include a record of your change * If someone else has recently committed a new test and it contains obvious errors, you can correct the test with a simple record of the change you have made * If you are confident that the dependencies of the test have been incorrectly documented, for example if the test requires a streamable processor, then again, you can fix the test with a simple record of the change that you made. * Similarly, if you find that the test makes assumptions about the result that are unwarranted (for example if the result assertions assume the order of items in the result), and these assumptions are not central to the purpose of the test, you can replace them with assertions that improve the interoperability of the test. (c) Changes that require consensus * Any change that goes beyond those listed above requires consensus. This doesn't mean it necessarily needs to be debated by the Working Group. It may be enough just to check with the author of the test that you haven't overlooked anything. You should ask yourself "is there any possibility that someone will disagree with this change?", and if the answer is yes, you need to raise a bug report. * If the test raises questions about the correct interpretation of the spec, it's probably better to raise the bug against the spec rather than against the test. But cite the test case in your bug report. * Once a bug report is raised, the test suite coordinator will decide whether a WG decision is needed, or whether it is possible to reach informal consensus through discussion on the bugzilla forum. (d) Housekeeping * For all changes, include a pertinent commit message when committing. For all non-trivial changes, also include a record of the change in the form of a <modified by=.../> element. * Always check that your changes to test catalogs are schema-valid. Remember that the order of elements in a test-case is significant. Michael Kay Saxonica
Received on Thursday, 10 March 2016 14:52:13 UTC