[CSSWG] Minutes Paris F2F 2017-08-03 Part II: Automating Test Coverage, CSS UI 3, F2F Scheduling [css-ui-3]

  These are the official CSSWG minutes.
  Unless you're correcting the minutes,
 Please respond by starting a new thread
   with an appropriate subject line.

Automating Test Coverage

  - gregwhitworth proposed finding a way to automatically link tests
      to a section to make it easier to review all tests. The problem
      is that specs are not static. fantasai listed various approaches
      that have been tried: there is no ideal method, so the plan is
      for each spec's QA owner to experiment.
  - This raised a larger question how to better test specs. There
      was interest in having an assigned QA resource per spec but
      ultimately that would require a commitment from WG member
      companies to send QA resources to the group.
  - RESOLVED: Changes list for a CR links to updated or additional
              test cases for each change.


  - RESOLVED: Close 1598 as no change.
  - RESOLVED: Clarify in level 3 that the UA should dispatch events
              as if 'text-ellipsis' were 'none'. (Issue #1637)
  - RESOLVED: Add informative note that links to the part of HTML
              that specifies how cursor works on image maps. (Issue
  - Florian will investigate if any SHOULD sections ought to become
      MAY sections since they're failing in most/all major browsers.
  - RESOLVED: Move directional navigation properties, nav-up/down/
              left/right to next level

F2F Scheduling

  - Still no decision on the location for the Summer 2018 F2F.
  - The group may put together an online consensus poll.


Agenda: https://wiki.csswg.org/planning/paris-2017#topics

Scribe: eae

Automating Test Coverage

  gregwhitworth: I was planning to have this discussion with
                 Geoffrey and Tab at some point.
  gregwhitworth: When we were working through table tests I was
                 going through all anchor tags to get an idea of
                 test coverage.
  gregwhitworth: It was pointed out to me that I can't depend on the
                 anchor tags as they change for various reasons.
  gregwhitworth: Basically from what I could see there is a
                 fundamental problem, we're waiting until CR until
                 doing a review of all tests.
  gregwhitworth: We need to keep these things up to date as we go.
  gregwhitworth: We could add an automatic solution, maybe in
  gregwhitworth: We could either anchor any line or sub anchors.
  gregwhitworth: Especially for things like MUST and SHOULD
  gregwhitworth: We need to resolve normative changes that have
                 corresponding tests.
  gregwhitworth: Would slow down the process.
  gregwhitworth: When you make a change include a test collateral.
                 Two action items: 1) For normative changes to
                 coincide with test collateral. 2) some kind of work
                 to bikeshed to aid in this.
  gregwhitworth: To prune as we go.
  gregwhitworth: Additionally worth considering that all specs would
                 have a test QA editor.
  gregwhitworth: We've seen the benefit of this at Microsoft.

  astearns: When you went through the flexbox tests manually did you
            find that the anchor tags where not sufficient.
  gregwhitworth: Too broad. Requires reverse engineering to find the
                 right section.

  astearns: For prior art, in terms of more specific anchor. You
            might want to take a look at the WOFF spec. Has anchors
            for all.
  TabAtkins: Very easy to do in bikeshed. Wrap everything in an
             assert tag.
  fantasai: Specs do change and the individual sentences that are
            associated with a particular test isn't stable.
            Automated association will result in broken links.
  TabAtkins: Manually associating isn't practical either.
  astearns: If automated we can show that this anchor change,
            anything associated with it must change. Automation can
            give us a checklist that is a subset rather than a full
  dbaron: The actual assertion often brings in other definitions.
  gregwhitworth: I agree, you cannot fully depend on this to solve
                 our testing needs. We need to resolve that when you
                 make normative changes there is also a test change.
  gregwhitworth: I want to be able to run the test suite to know
                 what changed.

  [discussion of requiring a test for every spec change]
  astearns: You claimed this would slow down the process. I don't
            think this is true. You had to manually do this for
            flexbox, if we can automate it it might be faster. Would
            speed up feedback for changes if it came with tests.
  Florian: Not sure this is true, it takes at the very least many
           months to get anyone to take review.
  Florian: Maybe for high-priority specs like grid or flex, you get
           the review quickly, but
  Florian: for mutlicol for instance it might take years for the
  Rossen: The problem is from what I have observed, a spec goes
          through an idea phase, someone is excited. A lot of people
          are talking about it then discussion starts in the domain.
          Very few people are participating in topic. Then we get to
          bikeshedding names, everyone is involved. :) This is the
          best part. Then we're in a no-mans land where we want to
          start implementing but there are no tests.
  Rossen: This is the part where nobody wants to participate.
  Rossen: Then we have someone who starts implementation.
  Rossen: If there are no tests that are there to verify the
          assumptions. Then we start making up our own assumptions.
          Then when the next implementor comes along there are a lot
          of interop issues.
  Rossen: Then we find ourselves here a few years later wondering
          about min-width for table cells, for instance. This
          happens when the test was written as a part of the
  Florian: I want tests. I like tests. I write a lot of them. And I
           harass people to review them.
  Florian: People *should* write tests with spec changes, yes. I
           agree. Requiring tests for all changes, maybe not so
           much. Would slow down process *a lot*.
  Florian: I wrote a bunch of UI tests, fantasai finally reviewed
           them a year later--after I ambushed her after dinner last
  dbaron: A bunch of specs at the whatwg, about 3-6 mo ago tried to
          do more test driven. Feedback from that is that they're
          moving faster than before.

  gregwhitworth: Florian, I think that is a valid point. Important
                 that when we sign up for a spec we're not only
                 signing up for the spec, you are also signing up
                 for tests as well.
  gregwhitworth: If no-one wants to sign up for QA then that is a
  gregwhitworth: It's fun to work on spec. Not as fun to review test
                 changes or start writing tests.
  gregwhitworth: Important that the QA people are here as well.
  fantasai: One advantage of QA not being here is that if the
            spec is not clear then the QA person cannot infer from
            discussion. Validates that the spec is clear.
  <fantasai> (That said, I'm all for having more QA involvement in
             this process. We've historically had that back when
             Opera and Mozilla had dedicated QA persons, and we've
             lost that along the way.)
  Florian: We should assign people to be in charge of the test
           suite. Hard to find volunteers.
  Florian: If we're explicit about ownership we'll see spec that
           lack a QA owner.

  Rossen: We started three months ago to move a bunch of specs that
          we believe as a working group to be close to rec. Fonts,
          for example, was one awesome example of a spec that got a
          lot of attention. A bunch of tests added to it. Reviewed.
          This spec is now moving very quickly compared to other.
  Rossen: On the other hand, we have background and borders, needs
          tests from Mozilla.. Cascade-3 needs tests from Mozilla.
  Rossen: Two more specs where we are waiting for tests. Specs that
          have been in the works for years. Specs that are already
          implemented and have been for years. Holding us back.
          We're missing tests. We don't even have the data to start
          the discussion around whether we're interoperable. Holds
          up the process.
  Rossen: Those specs, that could have been recs, arguably years
          ago. Will not be rec for years to come due to lack of

  fantasai: We need more dedicated QA. We've had that in the distant
  fantasai: We don't really have that as much as we need. We have
            Greg, working on some stuff, Florian is working on some
            tests. Japan funding it for writing mode. Only a little
            bit here and there.
  fantasai: But wrt having editors be responsible for tests: having
            Tab and I own QA for everything is not scalable nor
            reasonable, we're overloaded just on specs. Need more
            people involved for QA.
  Rossen: This is correct.
  Rossen: You (Tab + fantasi) are not the only ones who should be
          doing this.

  glazou: This is a really old problem. We're all implementors here.
          Some people do not consider it as rewarding. We've always
          had this problem where we've relied on the work of ???.
  fantasai: We've not always had this problem. Years ago we had
            people that were interested in this.
  TabAtkins: We should make glazou the QA owner for all specs :)
  glazou: Any decision made here relies on the commitment from
          corporate members of this group to commit resources for
          testing to this group, not just send people to work on
          specs and implementations.
  Rossen: I agree this is an old problem. Not the easiest to solve.
          I think most implementors companies have solved this
          problem years ago.
  glazou: Internally.
  Rossen: Internally. The spec writer knows how this is supposed to
          work. Commit your spec to an repo without review.
  fantasai: That is what editors draft is for.
  <fantasai> (ED = repo without review)

  Florian: If we have a resolution. We have a spec change, where am
           I allowed to commit that.
  Rossen: As soon as someone starts implementing your working draft
          they will stare review your tests.
  dbaron: They will look in web platform tests, not at unreviewed
          PRs. Do not like the review process.
  astearns: People will take the tests that are in web platform
            tests and run them. Will only review if there are
            errors. I think that is fine.

  * tantek liked when specs had demonstrations (basic tests) of
           feature right there inline in examples.

  fremy: We mentioned that spec implementors don't have time to
         write tests. Might be fair
  fremy: At the very least we know what tests needs to be written.
         That needs to be part of it. "We need tests for this". At
         the very least that way two years later we'll know.
  fremy: That should be the responsibility of the spec writer.
  <glazou> +1 with fremy but that's theory ; I'm afraid practice
           will differ a bit
  fremy: If the assertion changes, that should be part of the spec
         process. Flagging tests that needs to be changed.
  fremy: Otherwise impossible to know which tests needs to be
         written for a change. Should be part of responsibility for
         spec editor.
  fremy: Nobody is in a better position.
  <astearns> +1 to the idea of tracking what tests are needed, but
             issues in the test repo isn't the way to do it (we just
             threw out a bunch of issues that tracked this when we
             did the wpt merge)
  fantasai: The person making the change might think of 20% of the
            needed tests, but noting that is better than none, yes.

  gregwhitworth: I feel like for normative changes I don't think it
                 is too much to ask that a test case should already
                 be written. Maybe you don't write 700 of them to
                 cover all edge cases.
  gregwhitworth: We have a lot of people that want to be involved.
  gregwhitworth: For additive css for instance, who is going to
                 write all the tests?
  gregwhitworth: Have a QA owner, so when spec owner makes a change
                 says "hey, can you write some tests for this thing
                 / update the tests for this thing"
  gregwhitworth: The goal is interop. Webdevs wants it to just work.
                 That is why tests are important.
  gregwhitworth: Some type of tests commitment for normative changes
                 is important.

  glazou: Just one point about normative changes and tests. If we
          have a test for ??. In my opinion we need to write tests
          from the very beginning.
  fantasai: Disagree. Some of the specs Tab and I have worked on
            have changed so much that writing tests from day one
            would have been a huge waste of time.
  glazou: Hard to catch up if we don't have tests from the beginning.
  <glazou> so many FPWDs already implemented and shipped w/o tests
  <glazou> and used in the wild
  <dauwhe> stuff has been implemented before FPWD
  Florian: If we're saying that there is an expectation to have
           tests for normative changes I'm all for that. Adding
           explicit responsibility I'm also in favor. On the other
           hand I'm against a policy that would disallow changes
           without tests.
  gregwhitworth: When we get around to CR, here are four hundred
                 PRs. Test cases exists.
  gregwhitworth: There needs to be automation so that we're not
                 trying to puzzle it together years after the fact.
  fantasai: Resolutions that aren't followed more than 80% of the
            time aren't great.
  fantasai: We already have quite a few edits that don't make it
            into the spec until a year later. Wouldn't want to add
            more delay to that.
  Florian: We're already being slowed down for this. I've already
           considered hiring people to review my tests.

  <melanierichards> my comment earlier didn't get minuted so: if,
                    with a process change, not getting tests
                    reviewed in a timely fashion slows down progress
                    in a way that's noticeable/painful to other
                    people who want that spec to move forward (not
                    just the test writer), that could be a forcing
                    function to get tests reviewed faster / more
                    people committed to reviewing tests

  Rossen: Let's see if we can get to something a little bit more actionable.
  Rossen: 1) way forward, automation between tests and linking tests
          from specs.
  Rossen: What would be changed if a normative thing changed with a
  fantasai: Would require an assert tag around each part they change
            in a spec for bikeshed tooling support.
  fantasai: We could make it a habit to wrap all things in assert
            tag. Would be a bit annoying. Could help but we would
            end up with a lot of broken links. OK with someone
            trying this on one spec.
  <astearns> more-specific assertion tagging might be something to
             add once a spec is in CR (or late CR)
  fantasai: We should experiment with a couple of different
  Florian: Our system could detect when we make a change to a
           section where there are no tests then somebody needs to
  Rossen: Do we have a candidate spec for this?
  fantasai: You said we should have a QA person for a spec. Let's
            take three specs and assign a QA person and have them
            pick different approaches. Some might use <assert>,
            another might use a wiki index page and see how it works,
  fantasai: I don't think we have a good solution. If specs were
            static bikeshould would be obvious solution. That is not
            the case until they reach rec. We need to experiment and
            see what the right level of automation and linkage is.
            Individual people need to try something and see how that
            approach works.
  fantasai: I've tried many approaches and none are ideal.
  fantasai: We need somebody to own the testing practice for a spec.
  fantasai: Someone needs to own it: "this is my spec, I need it to
            be tested".
  fantasai: Then the tooling will fall out from that.
  <fantasai> there's various examples for how we've tried to
             coordinate spec and test and test assertion. Comments
             or quotes from the spec in tests are one way.
             Annotations in the spec source are another way.
             Bikeshed auto-ID cross-linking is another way. An
             intermediary wiki that connects spec sentences with
             test assertions with links to tests is another way...
  <fantasai> we don't have an obviously best way atm

  Rossen: We have a ton of specs. choosing two or three is not a
          problem. Finding the QA people would be more problematic
          but a good idea.
  Rossen: Won't volunteer people here but as a working group we
          should resolve to do this over the next few weeks.
  Rossen: We need people to step up and take test ownership for one
  Florian: If they determine the best approach is to add tooling
           that they aren't themselves capable of producing.
  Florian: If I'm the owner for a test, can I task people with that?
  Rossen: You can file a feature request for plinss.

  Rossen: Greg, was there anything else?
  gregwhitworth: I think we need to figure out the tests for
                 normative changes requirement
  fantasai: At CR it is a reasonable requirement. Prior to that, not
            so much.
  <tantek> agreed with fantasai
  gregwhitworth: My problem is that that is the situation we're in
                 with flex. We have a ton of tests, I don't know who
                 wrote them. I don't know what tests are for each
  fantasai: For CRs we have fairly close tracking of changes. We
            have a changes list and for a spec like flexbox they're
            listed in the changes list with a summary, a link to the
            issue, and diffs; it wouldn't be too hard to also include
            a link to the relevant tests.
  Rossen: Are there any formal requirements for normative changes to
          include tests?
  fantasai: Yes
  astearns: In part of you (rossen) and me to enforce.
  <fantasai> https://www.w3.org/TR/css-flexbox-1/#changes-20160301
  fantasai: In link posted, each change has an anchor, a summary of
            the change, link to issue generating the change, a
            difference for the change. We could add another
            component that is Test: <bunch of tests>
  Rossen: Yes.
  gregwhitworth: Yes but I don't want 500 tests there and having
                 them all reviewed at the end.
  Rossen: If we already have a resolution for requiring tests for a
          CR that is great. This week alone we already have a ton of
          CR changes coming. We'll watch very closely how that goes.
  fantasai: When we prepare for a CR we could look at the changes
            list and see the tests.
  Florian: I think I'm okay with this. Do we want it in the change
           list or the DOC?
  fantasai: Sometimes you have multiple comments for leading to a
            single change.
  <fantasai> Proposal is the Changes List for a CR links to updated
             or additional test cases for each change
  <Rossen> Normative changes to CR or later stages of the a spec
           need to link to an existing or a new test case(s).

  PROPOSED RESOLUTION: Normative changes to CR or later stages of
      the a spec need to link to an existing or a new test case(s).
  Rossen: Normative changes to a CR need to include test case.
  fantasai: The changes *list*, not the individual changes.
  fantasai: We require a list of changes for CR updates. Reason is
            that it is helpful for implementors. Changes are less
            expansive and easier to track than in WD.

  RESOLVED: Changes List for a CR links to updated or additional
            test cases for each change


  <Florian> https://test.csswg.org/harness/results/css-ui-3_dev/grouped/filter/
  <Florian> https://test.csswg.org/harness/results/css-ui-3_dev/grouped/filter/17/
  Florian: Link to test suite results ^.
  Florian: Normative requirements that do not have two or more
  Florian: Think coverage is sufficient. One test per normative
           assertion. Only 6 are missing for MUST. More are missing
           for SHOULDs.
  Florian: We're getting there. Since it is six and not zero I'm not
           pushing for CR today. Please fix them so that we can go
           to PR.
  Florian: One of the things I have not verified coverage for is
           directional navigation properties. Implemented in presto
           and TD.
  Florian: There are some informal issues raised. Marked at risk.

  <tantek> re: https://test.csswg.org/harness/results/css-ui-3_dev/grouped/filter/
           I would be strongly concerned with any test that fails in
           blink, edge, gecko, *and* webkit

Auto cursor should behaves as default cursor except for text?
  Github: https://github.com/w3c/csswg-drafts/issues/1598

  Florian: Specifically, one thing we have already resolved on that
           has been question is whether we should stick to what we
           have resolved on with regards to cursor.
  Florian: All the changes to cursor auto should be UA stylesheet.
           Have filed bugs to blink about this, they filed bug back
           with "are you sure?".
  fremy: In edge, there are some cases where we have things in the
         UA stylesheet. Also things that are magic. Much better to
         use UA stylesheet, files bugs.
  fremy: If you want to change, it should be in the UA stylesheet.
  Florian: If you find cases where it cannot be in the UA stylehseet
           and it isn't a known one then please file a bug back.
  Florian: Can we close this bug (1598) as no change?
  Rossen: Any objections?

  RESOLVED: Close 1598 as no change.

Event dispatching of pointer events on the ellipsis
  GitHub: https://github.com/w3c/csswg-drafts/issues/1637

  Florian: We have a statement in the spec, text-overflow ellipsis
           should not affect dispatching of pointer events.
  Florian: When you click the ellipsis the block that hosts it gets
           the event. Not clear to me that the statement normatively
           requires how the event is dispatches. Should it be
           dispatched directly to the element hosting it or the one
  Florian: Leave level 3 somewhat ambiguous about how events are
           being fired (but not that they *are* being fired) and be
           more specific in level-4.
  tantek: It is better than before.

  Florian: Block with nested block, event to block vs in-between
           spans. Not interop.
  tantek: Are there any strong preferences about how it should work?
  Florian: Firefox seems better.
  fremy: Agree, edge should match Firefox.
  eae: Chrome thinks it is reasonable to.

  tantek: Can we add a SHOULD here to capture the consensus?
  Florian: Haven't worked out the implications.
  tantek: It is already in issue description.
  tantek: Resolve on that same phrase, without being overly precise.
  tantek: Sounds like we have rough consensus.
  tantek: "dispatches the event to the elided inline element"
  tantek: That is all it takes, add it as a should. Capture
          consensus and keep moving.
  tantek: We have one impl and verbal agreement from other vendors.
  Florian: Normative change.
  tantek: Not all normative changes need to go to CR.
  Florian: If we can do this without process changes that's fine.
  tantek: We need to document ever change in the CR to PR process.
  Rossen: No problem with that.
  PROPOSED RESOLUTION: Clarify in level 3 that the UA should
      dispatch the event to the elided element.

  RESOLVED: Clarify in level 3 that the UA should dispatch the event
            to the elided element.

  <fantasai> So the resolution was that the event is dispatched as
             if text-ellipsis was none?

Cursor and image maps
  github: https://github.com/w3c/csswg-drafts/issues/1618

  Florian: As a note, this is not in our spec, HTML is weird when it
           comes to image maps.
  Florian: The cursor that you are supposed to use over an area in
           an image map depends on the area element and not the
           image you are hovering.
  Florian: The only property that is affected by area is the cursor.
  tantek: I agree
  Rossen: Other opinions?
  <tantek> yes, add the informative note
  PROPOSED RESOLUTION: Add informative note that links to the part
      of HTML that specifies how cursor works on image maps.
  Rossen: Any objections?

  RESOLVED: Add informative note that links to the part of HTML that
            specifies how cursor works on image maps.

Test results

  tantek: The tests results are really good, as pointed out by
          Florian, there are a some we do not have tests for.
  Florian: Directional navigation yes, I don't know.
  <Florian> https://test.csswg.org/harness/results/css-ui-3_dev/grouped/filter/17/
  tantek: In addition to the less than two problem, there are
          numerous tests, ...
  tantek: Looking at the list there are several tests that have
          failures across all of the edge/blink/webkit
  tantek: Indication that the normative text causing that needs
          looking into before we go to CR.
  tantek: If we're seeing four of the major engines not do something
          that is a strong indication it should be downgraded to MAY
  tantek: If you scroll down to outline tests 14 & 15. Have three of
          the four engines failing. No one else supporting./ 8 - 14
          all but one of them have all four engines failing it.
  tantek: Text overflow 18-19. Just talked about that.
  tantek: Cursor image, 13-15. Visually obvious.
  tantek: Everyone of those where three engines are failing, if
          these are coming from MUSTS they should be dropped.
  tantek: We shouldn't set the expectation that the SHOULD normative
          sections of the spec applies. Should be dropped or changed
          to a MAY if we still think it is a good idea. Should as in
  Florian: Some of them are already MAY. Can't tell which are MAY vs

  ACTION florian to see which ones are from SHOULD and consider
         whether to change to MAY.
  <trackbot> Created ACTION-856

  fremy: MAY is very open. Almost seems like a bug we're expecting.
  fremy: We want to progress, may doesn't do that.
  Rossen: Let's not discuss this now.
  Florian: If we downgrade to MAY that would require a new CR.
  tantek: Loosening conformance requirements does not necessarily
          require a new PR.
  dbaron: Opposite of SHOULD is SHOULD NOT
  dbaron: Maybe it should be a SHOULD.
  tantek: Leaving SHOULDs in there that four engines do not
          implement is a good way to block CR.
  Rossen: Let's talk about this once we have more information.
  tantek: By downgrading we could go to CR faster.

At-Risk Section

  Florian: Should I move the at-risk section now? nav-left,
           nav-down, .... No one has worked on them for years.
  Florian: Would rather push to next level
  Rossen: Do it. Move it.
  PROPOSED RESOLUTION: Move at-risk section to next level.
  Florian: Google has had an intent to implement which had problems.
           Concluded they wouldn't.
  Florian: Presto and old TVs do.
  Florian: Objection is "for tvs, why in browsers?"
  Florian: Would like to move to next level.
  iank: We rejected the intent for this.
  tantek: I see light green passes for webkit here too.
  Rossen: tantek, are you trying to keep this in spec?
  tantek: Want evidence based decision.
  Rossen: Absolutely no browser implement. No interest.
  tantek: Test results make it look more implemented then it is.

  RESOLVED: Move directional navigation properties, nav-up/down/left/
            right to next level

F2F Scheduling
  Scribe: fantasai
  Wiki: https://wiki.csswg.org/planning

  Rossen: Discussed 2018 summer meeting
  Rossen: Meeting April in Berlin, coordnated with TypeO.
  Rossen: We speculatively were thinking of doing in Hawaii (
          seriously, because easier for everyone to get to).
  TabAtkins: Still on the table, but not finalized.
  astearns: I'm concerned about holding a meeting in the US.
  glazou: Me too.
  astearns: Prefer somewhere not in the US.

  [discussion of options]

  <dbaron> if you're looking for the closest non-US place to Hawaii,
           you want to look at how hard it is to travel to airport
           code NHV
  <eae> dbaron: Do you have an office in NHV? :)
  <dbaron> eae, no
  <eae> Sad panda.
  <tantek> why not keep any survey inclusive of all options just to
           see how the US options rank relatively to others?
  <tantek> what are all the options for the summer meeting?

  [current ideas include Toronto and various other Google offices in
      Canada, Sydney, Taipei]
  dino: Costs of travel to Sydney?
  various: A lot.
  TabAtkins: That's why we were seriously considering Hawaii, would
             actually cost us less than Sydney
  <glazou> Iceland ?
  <glazou> Stockholm in july is gorgeous btw

  tantek: Why not vote on locations?
  <eae> Agree with tantek, why don't we vote on all options?
  astearns: My concern with US is minority that doesn't want to come
            / can't come
  TabAtkins: Consensus poll, not actual vote.
  astearns: that might work.
  <tantek> yes, consensus poll please among all options, with anon

  Rossen: ok, let's break for lunch.
  <br type=lunch>

Received on Tuesday, 29 August 2017 21:55:23 UTC