RE: [css-flexbox-1] Untestable assertions

On Tuesday, April 16, 2013 8:02 AM Sylvain Galineau wrote:
> On 4/15/13 7:13 PM, "Arron Eicholz" <Arron.Eicholz@microsoft.com> wrote:
> 
> >On Monday, April 15, 2013 5:18 PM Sylvain Galineau wrote:
> >> On 4/15/13 11:45 AM, "Arron Eicholz" <Arron.Eicholz@microsoft.com>
> >>wrote:
> >>
> >> >On Monday, April 15, 2013 11:18 AM Tab Atkins Jr. wrote:
> >> >> On Mon, Apr 15, 2013 at 10:31 AM, Arron Eicholz
> >> >> <Arron.Eicholz@microsoft.com> wrote:
> >> >> > On Monday, April 15, 2013 9:52 AM Tab Atkins Jr. wrote:
> >> >> >> On Mon, Apr 15, 2013 at 8:49 AM, Arron Eicholz
> >> >> >> <Arron.Eicholz@microsoft.com> wrote:
> >> >> >> > On Saturday, April 13, 2013 12:10 PM Tab Atkins Jr. wrote:
> >> >> >> >> The words are intended to be part of RFC2119.  The fact that
> >> >> >> >> they can't be automatically tested is irrelevant, because we
> >> >> >> >> won't be testing them anyway (they're authoring conformance,
> >> >> >> >> not UA
> >> >> >> conformance).
> >> >> >> >
> >> >> >> > I really don't care about the RFC2119 in this situation. I am
> >> >> >> > pointing out the
> >> >> >> fact that we can't test author's conformance. And we cannot
> >> >> >> require authors to do anything. Authors have the freedom of
> >> >> >> choice to do
> >> >>things.
> >> >> >>
> >> >> >> And *I'm* pointing out that we don't test authoring conformance
> >> >>criteria.
> >> >> >> That's for validators and evangelists to do.
> >> >> >
> >> >> > Ahh but we have to verify normative statements in the spec that
> >> >> > have
> >> >> testable assertions. And in these cases they are testable assertions.
> >> >>
> >> >> Who is "we"?  If you're trying to imply that the CSSWG has to test
> >> >>authoring  conformance criteria in order to exit CR, you're wrong.
> >> >>No WG has ever been  blocked from advancing for failing to test
> >> >>authoring criteria.
> >> >
> >> >"We" is test writers and the testers that have to verify the test
> >> >cases that are written. The working group has very little to do with
> >> >either of these. The WG sets of the criteria to write tests and
> >> >right now it is loosely defined to encompass all normative statements
> require tests.
> >> >Thus we then run into my issue that I called out.
> >> >
> >> >A WG would be blocked if such tests were written but there have been
> >> >from what I can tell no cases written in CSS 2.1 (which has the most
> >> >complete suite). In fact CSS 2.1 explicitly identified author
> >> >information as non-normative. The Flexbox spec does not have similar
> >> >text in the conformance section. This is another issue that should
> >> >be corrected and actually would fix this problem all together.
> >> >
> >> >CSS 2.1 states
> >> >#   At times, this specification recommends good practice for authors
> >>and
> >> >#   user agents. These recommendations are not normative and
> >> conformance
> >> >#   with this specification does not depend on their realization. These
> >> >#   recommendations contain the expression "We recommend ...", "This
> >> >#   specification recommends ...", or some similar wording.
> >> >
> >> >> >> >> As Henrik says, this style of authoring conformance is used
> >> >> >> >> elsewhere, such as HTML.
> >> >> >> >
> >> >> >> > While I agree that there may be other places that follow this
> >> >> >> > same
> >> >> pattern.
> >> >> >> It does not justify the fact that it is incorrect to state the
> >> >>sentence this
> >> >> way.
> >> >> >> Also I have seen very few normative, if any, statements that
> >> >> >> use this particular grammar and fall into this situation. We
> >> >> >> could also make these notes and that would also solve the problem.
> >> >> >>
> >> >> >> No, this is literally used everywhere.  Every single time a
> >> >> >> validator raises an error, it's because of an authoring
> >>conformance
> >> criteria.
> >> >> >> These are sometimes implicit rather than explicit, but that's
> >> >> >> largely because we simple don't care as much about being
> >> >> >> precise with authoring conformance.
> >> >> >
> >> >> > Please point me to more situations like this. I will raise
> >> >> > issues
> >> >>because they
> >> >> require tests to test authors' compliance with something. All the
> >> >>other cases I  have seen similar to this right now are in notes,
> >> >>which aren't normative.
> >> >>
> >> >> See: any part of CSS that would result in a validator error, any
> >> >>part of HTML  that would result in a validator error, etc.
> >> >
> >> >I am not reviewing HTML, nor am I reviewing any other specs as the
> >> >moment. I have found issues in the Flexbox spec and this is what I
> >> >have called out for now. When I review the next CSS spec I will call
> >> >out the same issues if I find them there as well.
> >> >
> >> >> >> > Wouldn't it just be easier to fix the issue since and issue
> >> >> >> > was raised, than to
> >> >> >> continue to argue this point over email?
> >> >> >>
> >> >> >> ...really?
> >> >> >
> >> >> > Yes really. It's easy to fix these simple grammar mistakes I
> >> >> > have
> >> >>pointed
> >> >> out. I have edited specs and I know it takes only 5-10 minutes to
> >> >>make simple  edits like this.
> >> >>
> >> >> I wasn't questioning the ease - of course it's easy.  I was
> >> >>questioning the  tone, which implies that I should be taking the
> >> >>easiest way out rather than  the correct way.  But this is an
> >> >>irrelevant tangent, and I'm not going to pursue  it.
> >> >
> >> >Then you incorrectly took my tone from my email. I am not trying to
> >> >be difficult or argumentative I am trying to be helpful and constructive.
> >> >If you take offense to me correcting your grammar in your spec, I am
> >> >sorry, but this grammar needs to change only slightly to be ok. Am I
> >> >a little frustrated that you won't fix a simple grammar error, yes,
> >> >but I am not trying to be rude to you about fixing this issue in any way.
> >> >
> >> >--
> >> I'm not sure I quite follow the argument here; as I understand it,
> >>job
> >>#1 for
> >> the test suite is to verify normative requirements of the spec can be
> >>*implemented*.
> >
> >Yes, that statements is correct.
> >
> >> Requirements that relate to how the feature should be
> >> *used* - authoring requirements - are out of scope for the test suite.
> >>I'm not
> >> aware of any rule requiring the latter to not use RFC2119 language,
> >>nor do I  grasp why they should not use such language.
> >
> >Actually no this isn't true in any of our modules. We forgot to include
> >the text that explicitly states that author requirements are out of
> >scope for verification and testing. This is my exact issue that I am
> >pointing out. We currently override the conformance section of CSS 2.1
> >with the conformance section of the modules. Since we do that and we no
> >longer have explicit text stating that author requirements are
> >non-normative. I have to write tests for them and figure out how we
> >(the community and the
> >WG) can run and verify the tests. I think testing author requirements
> >is stupid and why I called out the issue to begin with.
> >
> >> Figuring out which is which seems pretty easy to me (and I very much
> >>doubt  you, Arron, have doubts); should we instead mark authoring
> >>requirements  with a specific class so they can be styled differently?
> >>HTML did this, I think.
> >>
> >
> >It might be to you but (and I may be incorrect here) you are assuming
> >that there is text saying that author requirements are non-normative.
> >Well, unfortunately in the new CSS modules we removed the text making
> >that statement. Thus it makes the requirements for authors normative
> >and testable assertions. Thank you for agreeing with me that we need to
> >mark authoring requirements with some identifying context. I totally
> >agree. We need either the conformance section to change to include the
> >proper text, the explicit rules themselves to change to be
> >non-normative requirements or we need to make the text notes. Those are
> >the only ideas I have right now where I won't have to test these dumb
> >requirements we have inadvertently placed upon ourselves.
> >
> >We seem to all be in agreement with this change I have proposed. It's
> >only the specs and the module template that disagree with us at the
> >moment.
> 
> Not quite. I think everyone will agree with you that testing these authoring
> requirements is out of scope for the test suite. 

Great! This is one step forward to fixing this issue.

>The solution you proposed
> was to reword them on the assumption that all RFC2119 clauses need a test
> case regardless of context. 

As I pointed out this was one possible solution, not the only solution. Also without a clause stating that author requirements are not to be included in testing I MUST test them if they contain text pertaining to RFC 2119. There is no question about that from a testing perspective, and it’s the rule we have always followed. This is the whole reason I brought the issue up because testing author requirements makes no sense. Unfortunately, the way we have the specs written right now there is no clause excluding author requirements from testing. 

> I do not agree with that assumption (never mind
> that you suggest 'should' as a substitute, which, afaik, also requires a
> testcase). But I have no objection to stating in the preamble that these
> statements are informative. That seems the simplest, assuming the WG
> thinks this is a real issue (I don't).
> 

Who cares what I wanted to reword sentences too. I was trying to be helpful and constructive. Which is what all of us should do in this case. However, we seem to be attacking me for raising a valid issue that we missed a key piece of text in our module template, and not trying to understand the real issue of why I thought I needed to bring the issue up in the first place.

This is an important issue for the working group to address. I have to test these assertions unless there is text explicitly telling me not to. Can this all be handled on the mailing list? At this point I don't think so since I still don't see anyone updating their specs. I believe we will probably need some sort of obvious resolution to actually update the module template.

--
Thanks,
Arron Eicholz

Received on Tuesday, 16 April 2013 16:04:00 UTC