- From: C. M. Sperberg-McQueen <cmsmcq@blackmesatech.com>
- Date: Mon, 12 Jun 2023 20:34:20 -0600
- To: ixml <public-ixml@w3.org>
Test suite metadata
I've been carrying around several outstanding actions relating to the
schema for test suites and test results. This mail is intended to
discharge them. The actions are:
ACTION 2023-01-10-d: Michael to work out the metadata for identifying
the Unicode version(s) associated with a test.
ACTION 2023-02-07-b: Michael to think about adding to the test suite
schema to cover browser (etc) version
ACTION 2023-04-11-b: Michael to explore the use of named environments
in the test suite
The discussion will make occasional reference to pull request 173:
https://github.com/invisibleXML/ixml/pull/173
* Identifying the Unicode version(s) associated with a test
** Example
As an example, consider the simple grammar
S = [So].
This matches a single-character input if and only if the character is
a member of class 'Symbols - other'.
As input, let us use the alchemical symbol U+1F774 (🝴) 'LOT OF
FORTUNE' or 'pars fortunae'. This character was added to Unicode in
version 15.0.0, so any processor using an earlier version of Unicode
will treat this input as not a sentence, and any processor using
Unicode 15.0.0 (or, in due course, a later version) will produce the
result
<S>🝴</S>
** PR 173 proposal
The proposal in PR 173 allows a `dependencies` element to label a test
case; the test case applies (and the expected results are as
specified) if the dependencies are satisfied, and otherwise not.
So a test set (like the one attached to this mail) will have the grammar:
<ixml-grammar>S = [So].</ixml-grammar>
and then two test cases, mostly identical except for the
'dependencies' element (and name and description):
<test-case name="unknown-character">
<created by="CMSMcQ" on="2023-06-12"/>
<description>
<p>Expected result is to fail if the processor is using any
version of Unicode before 15.0.0.</p>
</description>
<dependencies Unicode-version="14.0.0"/>
<dependencies Unicode-version="13.0.0"/>
<dependencies Unicode-version="12.1.0"/>
<dependencies Unicode-version="12.0.0"/>
<test-string>🝴</test-string>
<result>
<assert-not-a-sentence/>
</result>
</test-case>
<test-case name="known-character">
<created by="CMSMcQ" on="2023-06-12"/>
<description>
<p>Expected result is to succeed if the processor is using
Unicode 15.0.0 or a later version.</p>
</description>
<dependencies Unicode-version="14.0.0"/>
<test-string>🝴</test-string>
<result>
<assert-xml><S xmlns="">🝴</S></assert-xml>
</result>
</test-case>
** Alternative
An alternative plan would be to allow `dependencies` to occur within
the `result` element, so that we would need only one test case, with
multiple results:
<test-case name="new-character">
<created by="CMSMcQ" on="2023-06-12"/>
<description>
<p>Expected result is to fail for Unicode before 15.0.0,
succeed for Unicode 15.0.0 or later.</p>
</description>
<test-string>🝴</test-string>
<result>
<dependencies Unicode-version="14.0.0"/>
<dependencies Unicode-version="13.0.0"/>
<dependencies Unicode-version="12.1.0"/>
<dependencies Unicode-version="12.0.0"/>
<assert-xml><S xmlns="">🝴</S></assert-xml>
</result>
<result>
<dependencies Unicode-version="15.0.0"/>
<assert-xml><S xmlns="">🝴</S></assert-xml>
</result>
</test-case>
** Discussion
The group should decide whether to adopt the plan in PR 173 or the
alternative plan. I could be persuaded either way, I think, but for
now I recommend PR 173, for these reasons:
- Even if we adopt the alternative proposal, we probably want to
continue allowing `dependencies` to occur as a child of
`test-case`, for situations when a test really makes no sense
unless certain dependencies are met. (No examples for ixml, which
may mean something.)
If so, then both forms shown above would continue to be possible.
Allowing just one is simpler for makers of test drivers than
allowing both.
- The savings in redundancy are minimal: if the test string is more
than a few characters long it can be moved into an external file
and referred to from each test case.
- It's possible for the expected result to be disputed for a
particular set of dependencies. If so, it's cleaner to have a
distinct revision history for each set of dependencies, which
means `dependencies` should be a child of `test-case`, not
`result`.
- Repetition of the `result` element will make sense only if there
are multiple sets of results for different sets of dependencies.
It is possible to write the schema so that multiple `result`
elements are allowed only if each contains a `dependencies` child.
But the schema is simpler if we don't do that.
I think PR 173 together with the discussion above discharges action
2023-01-10-d.
* Browser version and other environmental factors
Environmental factors are recorded using the `environment` element.
** Expected results which depend on environment
If a different result is expected when running a test in a particular
environment, then the alternate test result can be specified inside an
`app-info` element within the `test-case`, just as is done for
user-settable options which affect the expected result:
<test-case name="known-character">
<created by="CMSMcQ" on="2023-06-12"/>
<description>
<p>Expected result is to succeed if the processor is using
Unicode 15.0.0 or a later version.</p>
</description>
<dependencies Unicode-version="14.0.0"/>
<test-string>🝴</test-string>
<result>
<assert-xml><S xmlns="">🝴</S></assert-xml>
</result>
<app-info>
<assert-not-a-grammar jwl:browser="Joes-browser" jwl:version="2.3"
xmlns:jwl="https://johnlumley.github.io/jwiXML.xhtml"/>
</app-info>
</test-case>
** Reported results obtained in a specific environment
This element may be most commonly of use in reporting results, for
processors which can run in different environments. To show, for
example, that test "known-character" passed in a particular
environment, a test-set-results document could include a description
of that environment, using namespace-qualified attribute-value pairs.
<test-result name="known-character" result="pass">
<dependencies Unicode-version="14.0.0"/>
<test-string>🝴</test-string>
<result>
<assert-xml><S xmlns="">🝴</S></assert-xml>
<reported-xml><S>🝴</S></reported-xml>
</result>
<app-info>
<environment jwl:browser="Firefox" jwl:version="113.0.1"/>
</app-info>
</test-result>
But of course in practice a `test-set-results` document will have
results for all the tests in a test set run in a given environment.
So the overall structure will be:
<test-set-results xmlns="https://github.com/invisibleXML/ixml/test-catalog"
name="demo"
>
<app-info>
<environment browser="Firefox" version="113.0.1"/>
</app-info>
<test-result name="unknown-character" result="pass">
...
</test-result>
<test-result name="known-character" result="pass">
...
</test-result>
...
</test-set-results>
I believe these examples show that the changes in PR 173 suffice for
recording environment factors which affect expected or observed
results, and thus discharge action 2023-02-07-b.
* Named environments
** The problem
If there are a great number of ways in which the environment can vary,
it will be tedious to enumerate them all, if that enumeration must be
frequently repeated.
** A possible solution
An `environment-declaration` element could be added to the Metadata
class in the schema, with a required `name` attribute. Its scope
would be the set of its following siblings (and any external documents
referred to by them or their descendants).
The `environment` element could then be defined as taking either a set
of namespace-qualified attribute value specifications or a `name`
attribute, which would be interpreted as a reference to an environment
of that name (assuming the reference is within the scope of the
declaration). If the reference is in the scope of two declarations,
the one with narrower scope is preferred.
** Discussion
I expect that `environment` elements will be of most use in reporting
results, not in restricting expected results. That is, I think tney
will occur mostly in `test-set-results` documents, not in
`test-catalog` documents. In a set of results, the environment in
which the results were gathered needs to be specified only once, so
the use of named environments would not save much space or spare
anyone much tedium.
So I suggest that for now we do without named environments. If we
find ourselves needing to specify environmental factors on many test
cases and their expected results, and if more than one or two
environmental factors need to be specified, then we can revisit this
topic.
This may count as discharging action 2023-04-11-b.
* Conclusion
I propose that PR 173 be accepted without further changes, other than
deleting an accidentally duplicated comment in the schema.
--
C. M. Sperberg-McQueen
Black Mesa Technologies LLC
http://blackmesatech.com
Received on Tuesday, 13 June 2023 02:35:46 UTC