Re: [css-flexbox-1] Untestable assertions

(13/04/16 10:13), Arron Eicholz wrote:
> On Monday, April 15, 2013 5:18 PM Sylvain Galineau wrote:
>> Who is "we"?  If you're trying to imply that the CSSWG has to test
>> authoring  conformance criteria in order to exit CR, you're wrong.  No
>> WG has ever been  blocked from advancing for failing to test authoring
>> criteria.
>
> "We" is test writers and the testers that have to verify the test cases
> that are written. The working group has very little to do with either
> of these. The WG sets of the criteria to write tests and right now it
> is loosely defined to encompass all normative statements require tests.
> Thus we then run into my issue that I called out.

Where is this criteria written down then? Perhaps that should be
fixed/clarified instead of css-flexbox-1?

(13/04/16 10:13), Arron Eicholz wrote:
>  I have to write tests for them and figure out how we (the community
> and the WG) can run and verify the tests. I think testing author
> requirements is stupid and why I called out the issue to begin with.

I am not sure about what's the rule of writing a test report, but if you
look at these normative statements for authors in a broader (and perhaps
as you said, stupid) way, you can write a test report like this:


[[
Test input : Use 'order'


Test output 1: Example 1 on http://dev.w3.org/csswg/css-flexbox/ by Tab
Atkins

Comment: It does not violate normative statements for authors in Section
5, Section 5.4.1, Section 7.1


Test output 2: (find a random blog post about 'order' somewhere written
by AUTHOR X)

Comment: It does not violate normative statements for authors in Section
5, Section 5.4.1, Section 7.1


Conclusion: We have two conforming agents (Tab Atkins, AUTHOR X) passing
the normative statements for authors in Section 5, Section 5.4.1,
Section 7.1.
]]

(13/04/16 3:05), Simon Pieters wrote:
> On Mon, 15 Apr 2013 20:45:31 +0200, Arron Eicholz
> <Arron.Eicholz@microsoft.com> wrote:
>> "We" is test writers and the testers that have to verify the test
>> cases that are written. The working group has very little to do with
>> either of these. The WG sets of the criteria to write tests and right
>> now it is loosely defined to encompass all normative statements
>> require tests. Thus we then run into my issue that I called out.
>
> I think this is the problematic part. Some conformance classes (e.g.
> authors/validators) should probably be excluded from the "needs tests"
> requirement.

A potential use of the above "stupid" report is to make sure that

  1) The normative statements for authors are not self-contradictory
    (i.e. they are simultaneously satisfiable)
  2) An author can still/is willing to generate an non-empty output
     under the restrictions.

So perhaps these sorts of reports are not *completely* useless. They
just sound somewhat silly, indeed. On the other hand, some of the tests
for user agents are silly IMHO too, so this seems subjective.

(13/04/16 3:05), Simon Pieters wrote:
> Authoring conformance requirements that can be checked by an automated
> validator can in theory be tested by writing tests for the validator.
>
http://simon.html5.org/test/validator/attribute-values/img-usemap/just-a-hash.html
is an example of such a test.

I think this is independent. We might run into a situation when a
validator can check for a particular authoring requirement but yet no
author is willing to obey such requirement. For example, something like

   To use 'flex-shrink', authors must write down at lease 1000 digits
   for the <number>.

In such cases, this would not be implementable (by authors) and we can't
easily write a test report like the above.


We can of course define the passing criteria to ask for x percent of
passing rate for all authors, but that's going to be very controversial
in the HTML community of course.



Cheers,
Kenny
-- 
Web Specialist, Opera Sphinx Game Force, Oupeng Browser, Beijing
Try Oupeng: http://www.oupeng.com/

Received on Tuesday, 16 April 2013 09:27:22 UTC