- From: Dael Jackson <daelcss@gmail.com>
- Date: Tue, 29 Aug 2017 17:54:28 -0400
- To: www-style@w3.org
=========================================
These are the official CSSWG minutes.
Unless you're correcting the minutes,
Please respond by starting a new thread
with an appropriate subject line.
=========================================
Automating Test Coverage
------------------------
- gregwhitworth proposed finding a way to automatically link tests
to a section to make it easier to review all tests. The problem
is that specs are not static. fantasai listed various approaches
that have been tried: there is no ideal method, so the plan is
for each spec's QA owner to experiment.
- This raised a larger question how to better test specs. There
was interest in having an assigned QA resource per spec but
ultimately that would require a commitment from WG member
companies to send QA resources to the group.
- RESOLVED: Changes list for a CR links to updated or additional
test cases for each change.
CSS UI 3
--------
- RESOLVED: Close 1598 as no change.
- RESOLVED: Clarify in level 3 that the UA should dispatch events
as if 'text-ellipsis' were 'none'. (Issue #1637)
- RESOLVED: Add informative note that links to the part of HTML
that specifies how cursor works on image maps. (Issue
#1618)
- Florian will investigate if any SHOULD sections ought to become
MAY sections since they're failing in most/all major browsers.
- RESOLVED: Move directional navigation properties, nav-up/down/
left/right to next level
F2F Scheduling
--------------
- Still no decision on the location for the Summer 2018 F2F.
- The group may put together an online consensus poll.
===== FULL MINUTES BELOW ======
Agenda: https://wiki.csswg.org/planning/paris-2017#topics
Scribe: eae
Automating Test Coverage
========================
gregwhitworth: I was planning to have this discussion with
Geoffrey and Tab at some point.
gregwhitworth: When we were working through table tests I was
going through all anchor tags to get an idea of
test coverage.
gregwhitworth: It was pointed out to me that I can't depend on the
anchor tags as they change for various reasons.
gregwhitworth: Basically from what I could see there is a
fundamental problem, we're waiting until CR until
doing a review of all tests.
gregwhitworth: We need to keep these things up to date as we go.
gregwhitworth: We could add an automatic solution, maybe in
bikeshed.
gregwhitworth: We could either anchor any line or sub anchors.
gregwhitworth: Especially for things like MUST and SHOULD
gregwhitworth: We need to resolve normative changes that have
corresponding tests.
gregwhitworth: Would slow down the process.
gregwhitworth: When you make a change include a test collateral.
Two action items: 1) For normative changes to
coincide with test collateral. 2) some kind of work
to bikeshed to aid in this.
gregwhitworth: To prune as we go.
gregwhitworth: Additionally worth considering that all specs would
have a test QA editor.
gregwhitworth: We've seen the benefit of this at Microsoft.
astearns: When you went through the flexbox tests manually did you
find that the anchor tags where not sufficient.
gregwhitworth: Too broad. Requires reverse engineering to find the
right section.
astearns: For prior art, in terms of more specific anchor. You
might want to take a look at the WOFF spec. Has anchors
for all.
TabAtkins: Very easy to do in bikeshed. Wrap everything in an
assert tag.
fantasai: Specs do change and the individual sentences that are
associated with a particular test isn't stable.
Automated association will result in broken links.
TabAtkins: Manually associating isn't practical either.
astearns: If automated we can show that this anchor change,
anything associated with it must change. Automation can
give us a checklist that is a subset rather than a full
set.
dbaron: The actual assertion often brings in other definitions.
gregwhitworth: I agree, you cannot fully depend on this to solve
our testing needs. We need to resolve that when you
make normative changes there is also a test change.
gregwhitworth: I want to be able to run the test suite to know
what changed.
[discussion of requiring a test for every spec change]
astearns: You claimed this would slow down the process. I don't
think this is true. You had to manually do this for
flexbox, if we can automate it it might be faster. Would
speed up feedback for changes if it came with tests.
Florian: Not sure this is true, it takes at the very least many
months to get anyone to take review.
Florian: Maybe for high-priority specs like grid or flex, you get
the review quickly, but
Florian: for mutlicol for instance it might take years for the
review.
Rossen: The problem is from what I have observed, a spec goes
through an idea phase, someone is excited. A lot of people
are talking about it then discussion starts in the domain.
Very few people are participating in topic. Then we get to
bikeshedding names, everyone is involved. :) This is the
best part. Then we're in a no-mans land where we want to
start implementing but there are no tests.
Rossen: This is the part where nobody wants to participate.
Rossen: Then we have someone who starts implementation.
Rossen: If there are no tests that are there to verify the
assumptions. Then we start making up our own assumptions.
Then when the next implementor comes along there are a lot
of interop issues.
Rossen: Then we find ourselves here a few years later wondering
about min-width for table cells, for instance. This
happens when the test was written as a part of the
implementation.
Florian: I want tests. I like tests. I write a lot of them. And I
harass people to review them.
Florian: People *should* write tests with spec changes, yes. I
agree. Requiring tests for all changes, maybe not so
much. Would slow down process *a lot*.
Florian: I wrote a bunch of UI tests, fantasai finally reviewed
them a year later--after I ambushed her after dinner last
night.
dbaron: A bunch of specs at the whatwg, about 3-6 mo ago tried to
do more test driven. Feedback from that is that they're
moving faster than before.
gregwhitworth: Florian, I think that is a valid point. Important
that when we sign up for a spec we're not only
signing up for the spec, you are also signing up
for tests as well.
gregwhitworth: If no-one wants to sign up for QA then that is a
problem.
gregwhitworth: It's fun to work on spec. Not as fun to review test
changes or start writing tests.
gregwhitworth: Important that the QA people are here as well.
fantasai: One advantage of QA not being here is that if the
spec is not clear then the QA person cannot infer from
discussion. Validates that the spec is clear.
<fantasai> (That said, I'm all for having more QA involvement in
this process. We've historically had that back when
Opera and Mozilla had dedicated QA persons, and we've
lost that along the way.)
Florian: We should assign people to be in charge of the test
suite. Hard to find volunteers.
Florian: If we're explicit about ownership we'll see spec that
lack a QA owner.
Rossen: We started three months ago to move a bunch of specs that
we believe as a working group to be close to rec. Fonts,
for example, was one awesome example of a spec that got a
lot of attention. A bunch of tests added to it. Reviewed.
This spec is now moving very quickly compared to other.
Rossen: On the other hand, we have background and borders, needs
tests from Mozilla.. Cascade-3 needs tests from Mozilla.
Rossen: Two more specs where we are waiting for tests. Specs that
have been in the works for years. Specs that are already
implemented and have been for years. Holding us back.
We're missing tests. We don't even have the data to start
the discussion around whether we're interoperable. Holds
up the process.
Rossen: Those specs, that could have been recs, arguably years
ago. Will not be rec for years to come due to lack of
tests.
fantasai: We need more dedicated QA. We've had that in the distant
past.
fantasai: We don't really have that as much as we need. We have
Greg, working on some stuff, Florian is working on some
tests. Japan funding it for writing mode. Only a little
bit here and there.
fantasai: But wrt having editors be responsible for tests: having
Tab and I own QA for everything is not scalable nor
reasonable, we're overloaded just on specs. Need more
people involved for QA.
Rossen: This is correct.
Rossen: You (Tab + fantasi) are not the only ones who should be
doing this.
glazou: This is a really old problem. We're all implementors here.
Some people do not consider it as rewarding. We've always
had this problem where we've relied on the work of ???.
fantasai: We've not always had this problem. Years ago we had
people that were interested in this.
TabAtkins: We should make glazou the QA owner for all specs :)
glazou: Any decision made here relies on the commitment from
corporate members of this group to commit resources for
testing to this group, not just send people to work on
specs and implementations.
Rossen: I agree this is an old problem. Not the easiest to solve.
I think most implementors companies have solved this
problem years ago.
glazou: Internally.
Rossen: Internally. The spec writer knows how this is supposed to
work. Commit your spec to an repo without review.
fantasai: That is what editors draft is for.
<fantasai> (ED = repo without review)
Florian: If we have a resolution. We have a spec change, where am
I allowed to commit that.
Rossen: As soon as someone starts implementing your working draft
they will stare review your tests.
dbaron: They will look in web platform tests, not at unreviewed
PRs. Do not like the review process.
astearns: People will take the tests that are in web platform
tests and run them. Will only review if there are
errors. I think that is fine.
* tantek liked when specs had demonstrations (basic tests) of
feature right there inline in examples.
fremy: We mentioned that spec implementors don't have time to
write tests. Might be fair
fremy: At the very least we know what tests needs to be written.
That needs to be part of it. "We need tests for this". At
the very least that way two years later we'll know.
fremy: That should be the responsibility of the spec writer.
<glazou> +1 with fremy but that's theory ; I'm afraid practice
will differ a bit
fremy: If the assertion changes, that should be part of the spec
process. Flagging tests that needs to be changed.
fremy: Otherwise impossible to know which tests needs to be
written for a change. Should be part of responsibility for
spec editor.
fremy: Nobody is in a better position.
<astearns> +1 to the idea of tracking what tests are needed, but
issues in the test repo isn't the way to do it (we just
threw out a bunch of issues that tracked this when we
did the wpt merge)
fantasai: The person making the change might think of 20% of the
needed tests, but noting that is better than none, yes.
gregwhitworth: I feel like for normative changes I don't think it
is too much to ask that a test case should already
be written. Maybe you don't write 700 of them to
cover all edge cases.
gregwhitworth: We have a lot of people that want to be involved.
gregwhitworth: For additive css for instance, who is going to
write all the tests?
gregwhitworth: Have a QA owner, so when spec owner makes a change
says "hey, can you write some tests for this thing
/ update the tests for this thing"
gregwhitworth: The goal is interop. Webdevs wants it to just work.
That is why tests are important.
gregwhitworth: Some type of tests commitment for normative changes
is important.
glazou: Just one point about normative changes and tests. If we
have a test for ??. In my opinion we need to write tests
from the very beginning.
fantasai: Disagree. Some of the specs Tab and I have worked on
have changed so much that writing tests from day one
would have been a huge waste of time.
glazou: Hard to catch up if we don't have tests from the beginning.
<glazou> so many FPWDs already implemented and shipped w/o tests
<glazou> and used in the wild
<dauwhe> stuff has been implemented before FPWD
Florian: If we're saying that there is an expectation to have
tests for normative changes I'm all for that. Adding
explicit responsibility I'm also in favor. On the other
hand I'm against a policy that would disallow changes
without tests.
gregwhitworth: When we get around to CR, here are four hundred
PRs. Test cases exists.
gregwhitworth: There needs to be automation so that we're not
trying to puzzle it together years after the fact.
fantasai: Resolutions that aren't followed more than 80% of the
time aren't great.
fantasai: We already have quite a few edits that don't make it
into the spec until a year later. Wouldn't want to add
more delay to that.
Florian: We're already being slowed down for this. I've already
considered hiring people to review my tests.
<melanierichards> my comment earlier didn't get minuted so: if,
with a process change, not getting tests
reviewed in a timely fashion slows down progress
in a way that's noticeable/painful to other
people who want that spec to move forward (not
just the test writer), that could be a forcing
function to get tests reviewed faster / more
people committed to reviewing tests
Rossen: Let's see if we can get to something a little bit more actionable.
Rossen: 1) way forward, automation between tests and linking tests
from specs.
Rossen: What would be changed if a normative thing changed with a
PR.
fantasai: Would require an assert tag around each part they change
in a spec for bikeshed tooling support.
fantasai: We could make it a habit to wrap all things in assert
tag. Would be a bit annoying. Could help but we would
end up with a lot of broken links. OK with someone
trying this on one spec.
<astearns> more-specific assertion tagging might be something to
add once a spec is in CR (or late CR)
fantasai: We should experiment with a couple of different
approaches.
Florian: Our system could detect when we make a change to a
section where there are no tests then somebody needs to
know.
Rossen: Do we have a candidate spec for this?
fantasai: You said we should have a QA person for a spec. Let's
take three specs and assign a QA person and have them
pick different approaches. Some might use <assert>,
another might use a wiki index page and see how it works,
etc..
fantasai: I don't think we have a good solution. If specs were
static bikeshould would be obvious solution. That is not
the case until they reach rec. We need to experiment and
see what the right level of automation and linkage is.
Individual people need to try something and see how that
approach works.
fantasai: I've tried many approaches and none are ideal.
fantasai: We need somebody to own the testing practice for a spec.
fantasai: Someone needs to own it: "this is my spec, I need it to
be tested".
fantasai: Then the tooling will fall out from that.
<fantasai> there's various examples for how we've tried to
coordinate spec and test and test assertion. Comments
or quotes from the spec in tests are one way.
Annotations in the spec source are another way.
Bikeshed auto-ID cross-linking is another way. An
intermediary wiki that connects spec sentences with
test assertions with links to tests is another way...
<fantasai> we don't have an obviously best way atm
Rossen: We have a ton of specs. choosing two or three is not a
problem. Finding the QA people would be more problematic
but a good idea.
Rossen: Won't volunteer people here but as a working group we
should resolve to do this over the next few weeks.
Rossen: We need people to step up and take test ownership for one
spec.
Florian: If they determine the best approach is to add tooling
that they aren't themselves capable of producing.
Florian: If I'm the owner for a test, can I task people with that?
Rossen: You can file a feature request for plinss.
Rossen: Greg, was there anything else?
gregwhitworth: I think we need to figure out the tests for
normative changes requirement
fantasai: At CR it is a reasonable requirement. Prior to that, not
so much.
<tantek> agreed with fantasai
gregwhitworth: My problem is that that is the situation we're in
with flex. We have a ton of tests, I don't know who
wrote them. I don't know what tests are for each
change.
fantasai: For CRs we have fairly close tracking of changes. We
have a changes list and for a spec like flexbox they're
listed in the changes list with a summary, a link to the
issue, and diffs; it wouldn't be too hard to also include
a link to the relevant tests.
Rossen: Are there any formal requirements for normative changes to
include tests?
fantasai: Yes
astearns: In part of you (rossen) and me to enforce.
<fantasai> https://www.w3.org/TR/css-flexbox-1/#changes-20160301
fantasai: In link posted, each change has an anchor, a summary of
the change, link to issue generating the change, a
difference for the change. We could add another
component that is Test: <bunch of tests>
Rossen: Yes.
gregwhitworth: Yes but I don't want 500 tests there and having
them all reviewed at the end.
Rossen: If we already have a resolution for requiring tests for a
CR that is great. This week alone we already have a ton of
CR changes coming. We'll watch very closely how that goes.
fantasai: When we prepare for a CR we could look at the changes
list and see the tests.
Florian: I think I'm okay with this. Do we want it in the change
list or the DOC?
fantasai: Sometimes you have multiple comments for leading to a
single change.
<fantasai> Proposal is the Changes List for a CR links to updated
or additional test cases for each change
<Rossen> Normative changes to CR or later stages of the a spec
need to link to an existing or a new test case(s).
PROPOSED RESOLUTION: Normative changes to CR or later stages of
the a spec need to link to an existing or a new test case(s).
Rossen: Normative changes to a CR need to include test case.
fantasai: The changes *list*, not the individual changes.
fantasai: We require a list of changes for CR updates. Reason is
that it is helpful for implementors. Changes are less
expansive and easier to track than in WD.
RESOLVED: Changes List for a CR links to updated or additional
test cases for each change
CSS-UI-3
========
<Florian> https://test.csswg.org/harness/results/css-ui-3_dev/grouped/filter/
<Florian> https://test.csswg.org/harness/results/css-ui-3_dev/grouped/filter/17/
Florian: Link to test suite results ^.
Florian: Normative requirements that do not have two or more
implementations.
Florian: Think coverage is sufficient. One test per normative
assertion. Only 6 are missing for MUST. More are missing
for SHOULDs.
Florian: We're getting there. Since it is six and not zero I'm not
pushing for CR today. Please fix them so that we can go
to PR.
Florian: One of the things I have not verified coverage for is
directional navigation properties. Implemented in presto
and TD.
Florian: There are some informal issues raised. Marked at risk.
<tantek> re: https://test.csswg.org/harness/results/css-ui-3_dev/grouped/filter/
I would be strongly concerned with any test that fails in
blink, edge, gecko, *and* webkit
Auto cursor should behaves as default cursor except for text?
-------------------------------------------------------------
Github: https://github.com/w3c/csswg-drafts/issues/1598
Florian: Specifically, one thing we have already resolved on that
has been question is whether we should stick to what we
have resolved on with regards to cursor.
Florian: All the changes to cursor auto should be UA stylesheet.
Have filed bugs to blink about this, they filed bug back
with "are you sure?".
fremy: In edge, there are some cases where we have things in the
UA stylesheet. Also things that are magic. Much better to
use UA stylesheet, files bugs.
fremy: If you want to change, it should be in the UA stylesheet.
Florian: If you find cases where it cannot be in the UA stylehseet
and it isn't a known one then please file a bug back.
Florian: Can we close this bug (1598) as no change?
Rossen: Any objections?
RESOLVED: Close 1598 as no change.
Event dispatching of pointer events on the ellipsis
---------------------------------------------------
GitHub: https://github.com/w3c/csswg-drafts/issues/1637
Florian: We have a statement in the spec, text-overflow ellipsis
should not affect dispatching of pointer events.
Florian: When you click the ellipsis the block that hosts it gets
the event. Not clear to me that the statement normatively
requires how the event is dispatches. Should it be
dispatched directly to the element hosting it or the one
elided.
Florian: Leave level 3 somewhat ambiguous about how events are
being fired (but not that they *are* being fired) and be
more specific in level-4.
tantek: It is better than before.
Florian: Block with nested block, event to block vs in-between
spans. Not interop.
tantek: Are there any strong preferences about how it should work?
Florian: Firefox seems better.
fremy: Agree, edge should match Firefox.
eae: Chrome thinks it is reasonable to.
tantek: Can we add a SHOULD here to capture the consensus?
Florian: Haven't worked out the implications.
tantek: It is already in issue description.
tantek: Resolve on that same phrase, without being overly precise.
tantek: Sounds like we have rough consensus.
tantek: "dispatches the event to the elided inline element"
tantek: That is all it takes, add it as a should. Capture
consensus and keep moving.
tantek: We have one impl and verbal agreement from other vendors.
Florian: Normative change.
tantek: Not all normative changes need to go to CR.
Florian: If we can do this without process changes that's fine.
tantek: We need to document ever change in the CR to PR process.
Rossen: No problem with that.
PROPOSED RESOLUTION: Clarify in level 3 that the UA should
dispatch the event to the elided element.
RESOLVED: Clarify in level 3 that the UA should dispatch the event
to the elided element.
<fantasai> So the resolution was that the event is dispatched as
if text-ellipsis was none?
Cursor and image maps
---------------------
github: https://github.com/w3c/csswg-drafts/issues/1618
Florian: As a note, this is not in our spec, HTML is weird when it
comes to image maps.
Florian: The cursor that you are supposed to use over an area in
an image map depends on the area element and not the
image you are hovering.
Florian: The only property that is affected by area is the cursor.
tantek: I agree
Rossen: Other opinions?
<tantek> yes, add the informative note
PROPOSED RESOLUTION: Add informative note that links to the part
of HTML that specifies how cursor works on image maps.
Rossen: Any objections?
RESOLVED: Add informative note that links to the part of HTML that
specifies how cursor works on image maps.
Test results
------------
tantek: The tests results are really good, as pointed out by
Florian, there are a some we do not have tests for.
Florian: Directional navigation yes, I don't know.
<Florian> https://test.csswg.org/harness/results/css-ui-3_dev/grouped/filter/17/
tantek: In addition to the less than two problem, there are
numerous tests, ...
tantek: Looking at the list there are several tests that have
failures across all of the edge/blink/webkit
tantek: Indication that the normative text causing that needs
looking into before we go to CR.
tantek: If we're seeing four of the major engines not do something
that is a strong indication it should be downgraded to MAY
tantek: If you scroll down to outline tests 14 & 15. Have three of
the four engines failing. No one else supporting./ 8 - 14
all but one of them have all four engines failing it.
tantek: Text overflow 18-19. Just talked about that.
tantek: Cursor image, 13-15. Visually obvious.
tantek: Everyone of those where three engines are failing, if
these are coming from MUSTS they should be dropped.
tantek: We shouldn't set the expectation that the SHOULD normative
sections of the spec applies. Should be dropped or changed
to a MAY if we still think it is a good idea. Should as in
English.
Florian: Some of them are already MAY. Can't tell which are MAY vs
SHOULD.
ACTION florian to see which ones are from SHOULD and consider
whether to change to MAY.
<trackbot> Created ACTION-856
fremy: MAY is very open. Almost seems like a bug we're expecting.
fremy: We want to progress, may doesn't do that.
Rossen: Let's not discuss this now.
Florian: If we downgrade to MAY that would require a new CR.
tantek: Loosening conformance requirements does not necessarily
require a new PR.
dbaron: Opposite of SHOULD is SHOULD NOT
dbaron: Maybe it should be a SHOULD.
tantek: Leaving SHOULDs in there that four engines do not
implement is a good way to block CR.
Rossen: Let's talk about this once we have more information.
tantek: By downgrading we could go to CR faster.
At-Risk Section
---------------
Florian: Should I move the at-risk section now? nav-left,
nav-down, .... No one has worked on them for years.
Florian: Would rather push to next level
Rossen: Do it. Move it.
PROPOSED RESOLUTION: Move at-risk section to next level.
Florian: Google has had an intent to implement which had problems.
Concluded they wouldn't.
Florian: Presto and old TVs do.
Florian: Objection is "for tvs, why in browsers?"
Florian: Would like to move to next level.
iank: We rejected the intent for this.
tantek: I see light green passes for webkit here too.
Rossen: tantek, are you trying to keep this in spec?
tantek: Want evidence based decision.
Rossen: Absolutely no browser implement. No interest.
tantek: Test results make it look more implemented then it is.
RESOLVED: Move directional navigation properties, nav-up/down/left/
right to next level
F2F Scheduling
==============
Scribe: fantasai
Wiki: https://wiki.csswg.org/planning
Rossen: Discussed 2018 summer meeting
Rossen: Meeting April in Berlin, coordnated with TypeO.
Rossen: We speculatively were thinking of doing in Hawaii (
seriously, because easier for everyone to get to).
TabAtkins: Still on the table, but not finalized.
astearns: I'm concerned about holding a meeting in the US.
glazou: Me too.
astearns: Prefer somewhere not in the US.
[discussion of options]
<dbaron> if you're looking for the closest non-US place to Hawaii,
you want to look at how hard it is to travel to airport
code NHV
<eae> dbaron: Do you have an office in NHV? :)
<dbaron> eae, no
<eae> Sad panda.
<tantek> why not keep any survey inclusive of all options just to
see how the US options rank relatively to others?
<tantek> what are all the options for the summer meeting?
[current ideas include Toronto and various other Google offices in
Canada, Sydney, Taipei]
dino: Costs of travel to Sydney?
various: A lot.
TabAtkins: That's why we were seriously considering Hawaii, would
actually cost us less than Sydney
<glazou> Iceland ?
<glazou> Stockholm in july is gorgeous btw
tantek: Why not vote on locations?
<eae> Agree with tantek, why don't we vote on all options?
astearns: My concern with US is minority that doesn't want to come
/ can't come
TabAtkins: Consensus poll, not actual vote.
astearns: that might work.
<tantek> yes, consensus poll please among all options, with anon
option.
Rossen: ok, let's break for lunch.
<br type=lunch>
Received on Tuesday, 29 August 2017 21:55:23 UTC