Re: [css-flexbox-1] Untestable assertions

On 4/15/13 7:13 PM, "Arron Eicholz" <Arron.Eicholz@microsoft.com> wrote:

>On Monday, April 15, 2013 5:18 PM Sylvain Galineau wrote:
>> On 4/15/13 11:45 AM, "Arron Eicholz" <Arron.Eicholz@microsoft.com>
>>wrote:
>> 
>> >On Monday, April 15, 2013 11:18 AM Tab Atkins Jr. wrote:
>> >> On Mon, Apr 15, 2013 at 10:31 AM, Arron Eicholz
>> >> <Arron.Eicholz@microsoft.com> wrote:
>> >> > On Monday, April 15, 2013 9:52 AM Tab Atkins Jr. wrote:
>> >> >> On Mon, Apr 15, 2013 at 8:49 AM, Arron Eicholz
>> >> >> <Arron.Eicholz@microsoft.com> wrote:
>> >> >> > On Saturday, April 13, 2013 12:10 PM Tab Atkins Jr. wrote:
>> >> >> >> The words are intended to be part of RFC2119.  The fact that
>> >> >> >> they can't be automatically tested is irrelevant, because we
>> >> >> >> won't be testing them anyway (they're authoring conformance,
>> >> >> >> not UA
>> >> >> conformance).
>> >> >> >
>> >> >> > I really don't care about the RFC2119 in this situation. I am
>> >> >> > pointing out the
>> >> >> fact that we can't test author's conformance. And we cannot
>> >> >> require authors to do anything. Authors have the freedom of choice
>> >> >> to do
>> >>things.
>> >> >>
>> >> >> And *I'm* pointing out that we don't test authoring conformance
>> >>criteria.
>> >> >> That's for validators and evangelists to do.
>> >> >
>> >> > Ahh but we have to verify normative statements in the spec that
>> >> > have
>> >> testable assertions. And in these cases they are testable assertions.
>> >>
>> >> Who is "we"?  If you're trying to imply that the CSSWG has to test
>> >>authoring  conformance criteria in order to exit CR, you're wrong.  No
>> >>WG has ever been  blocked from advancing for failing to test authoring
>> >>criteria.
>> >
>> >"We" is test writers and the testers that have to verify the test cases
>> >that are written. The working group has very little to do with either
>> >of these. The WG sets of the criteria to write tests and right now it
>> >is loosely defined to encompass all normative statements require tests.
>> >Thus we then run into my issue that I called out.
>> >
>> >A WG would be blocked if such tests were written but there have been
>> >from what I can tell no cases written in CSS 2.1 (which has the most
>> >complete suite). In fact CSS 2.1 explicitly identified author
>> >information as non-normative. The Flexbox spec does not have similar
>> >text in the conformance section. This is another issue that should be
>> >corrected and actually would fix this problem all together.
>> >
>> >CSS 2.1 states
>> >#   At times, this specification recommends good practice for authors
>>and
>> >#   user agents. These recommendations are not normative and
>> conformance
>> >#   with this specification does not depend on their realization. These
>> >#   recommendations contain the expression "We recommend ...", "This
>> >#   specification recommends ...", or some similar wording.
>> >
>> >> >> >> As Henrik says, this style of authoring conformance is used
>> >> >> >> elsewhere, such as HTML.
>> >> >> >
>> >> >> > While I agree that there may be other places that follow this
>> >> >> > same
>> >> pattern.
>> >> >> It does not justify the fact that it is incorrect to state the
>> >>sentence this
>> >> way.
>> >> >> Also I have seen very few normative, if any, statements that use
>> >> >> this particular grammar and fall into this situation. We could
>> >> >> also make these notes and that would also solve the problem.
>> >> >>
>> >> >> No, this is literally used everywhere.  Every single time a
>> >> >> validator raises an error, it's because of an authoring
>>conformance
>> criteria.
>> >> >> These are sometimes implicit rather than explicit, but that's
>> >> >> largely because we simple don't care as much about being precise
>> >> >> with authoring conformance.
>> >> >
>> >> > Please point me to more situations like this. I will raise issues
>> >>because they
>> >> require tests to test authors' compliance with something. All the
>> >>other cases I  have seen similar to this right now are in notes, which
>> >>aren't normative.
>> >>
>> >> See: any part of CSS that would result in a validator error, any part
>> >>of HTML  that would result in a validator error, etc.
>> >
>> >I am not reviewing HTML, nor am I reviewing any other specs as the
>> >moment. I have found issues in the Flexbox spec and this is what I have
>> >called out for now. When I review the next CSS spec I will call out the
>> >same issues if I find them there as well.
>> >
>> >> >> > Wouldn't it just be easier to fix the issue since and issue was
>> >> >> > raised, than to
>> >> >> continue to argue this point over email?
>> >> >>
>> >> >> ...really?
>> >> >
>> >> > Yes really. It's easy to fix these simple grammar mistakes I have
>> >>pointed
>> >> out. I have edited specs and I know it takes only 5-10 minutes to
>> >>make simple  edits like this.
>> >>
>> >> I wasn't questioning the ease - of course it's easy.  I was
>> >>questioning the  tone, which implies that I should be taking the
>> >>easiest way out rather than  the correct way.  But this is an
>> >>irrelevant tangent, and I'm not going to pursue  it.
>> >
>> >Then you incorrectly took my tone from my email. I am not trying to be
>> >difficult or argumentative I am trying to be helpful and constructive.
>> >If you take offense to me correcting your grammar in your spec, I am
>> >sorry, but this grammar needs to change only slightly to be ok. Am I a
>> >little frustrated that you won't fix a simple grammar error, yes, but I
>> >am not trying to be rude to you about fixing this issue in any way.
>> >
>> >--
>> I'm not sure I quite follow the argument here; as I understand it, job
>>#1 for
>> the test suite is to verify normative requirements of the spec can be
>> *implemented*.
>
>Yes, that statements is correct.
>
>> Requirements that relate to how the feature should be
>> *used* - authoring requirements - are out of scope for the test suite.
>>I'm not
>> aware of any rule requiring the latter to not use RFC2119 language, nor
>>do I
>> grasp why they should not use such language.
>
>Actually no this isn't true in any of our modules. We forgot to include
>the text that explicitly states that author requirements are out of scope
>for verification and testing. This is my exact issue that I am pointing
>out. We currently override the conformance section of CSS 2.1 with the
>conformance section of the modules. Since we do that and we no longer
>have explicit text stating that author requirements are non-normative. I
>have to write tests for them and figure out how we (the community and the
>WG) can run and verify the tests. I think testing author requirements is
>stupid and why I called out the issue to begin with.
>
>> Figuring out which is which seems pretty easy to me (and I very much
>>doubt
>> you, Arron, have doubts); should we instead mark authoring requirements
>> with a specific class so they can be styled differently? HTML did this,
>>I think.
>> 
>
>It might be to you but (and I may be incorrect here) you are assuming
>that there is text saying that author requirements are non-normative.
>Well, unfortunately in the new CSS modules we removed the text making
>that statement. Thus it makes the requirements for authors normative and
>testable assertions. Thank you for agreeing with me that we need to mark
>authoring requirements with some identifying context. I totally agree. We
>need either the conformance section to change to include the proper text,
>the explicit rules themselves to change to be non-normative requirements
>or we need to make the text notes. Those are the only ideas I have right
>now where I won't have to test these dumb requirements we have
>inadvertently placed upon ourselves.
>
>We seem to all be in agreement with this change I have proposed. It's
>only the specs and the module template that disagree with us at the
>moment.

Not quite. I think everyone will agree with you that testing these
authoring requirements is out of scope for the test suite. The solution
you proposed was to reword them on the assumption that all RFC2119 clauses
need a test case regardless of context. I do not agree with that
assumption (never mind that you suggest 'should' as a substitute, which,
afaik, also requires a testcase). But I have no objection to stating in
the preamble that these statements are informative. That seems the
simplest, assuming the WG thinks this is a real issue (I don't).

Received on Tuesday, 16 April 2013 15:04:00 UTC