[CSSWG] Minutes Fukuoka F2F 2019-09-17 Part I: Joint Meeting with WPT

=========================================
  These are the official CSSWG minutes.
  Unless you're correcting the minutes,
 Please respond by starting a new thread
   with an appropriate subject line.
=========================================


Joint Meeting with WPT
----------------------

  - Everyone was appreciative for all the work done to improve
      documentation on WPT.
  - There is still a gap for manual testing. One follow up is to see
      if there's a way to convert tests run elsewhere to show on WPT
      as a temporary solution.
  - Work is being done to associate labels with tests which will
      hopefully solve for the open issues of MAY/SHOULD vs MUST tests
      and being able to pull tests by spec instead of by directory.

===== FULL MINUTES BELOW =======

Agenda: https://wiki.csswg.org/planning/tpac-2019#agenda

Present:
  Rachel Andrew, Fronteers
  Rossen Atanassov, Microsoft
  L. David Baron, Mozilla
  Amelia Bellamy-Royds, Invited Expert
  Christian Biesinger, Google
  Brian Birtles, Invited Expert
  Oriol Brufau, Igalia
  Tantek Çelik, Mozilla
  Emilio Cobos, Mozilla
  Emil A Eklund, Google
  Elika Etemad, Invited Expert
  Jihye Hong, LGE
  Koji Ishii, Google
  Sanket Joshi, Microsoft
  Brian Kardell, Igalia and Open JS Foundation
  Eugene Kashida, Salesforce, Observer
  Rune Lillesveen, Google
  Chris Lilley, W3C
  Peter Linss, Invited Expert
  Stanton Marcum, Amazon
  Myles C. Maxfield, Apple
  Cameron McCormack, Mozilla
  Nat McCully, Adobe
  Chase Phillips, Google
  Florian Rivoal, Invited Expert
  Dominik Röttsches, Google
  Devin Rousso, Apple
  Peter Rushforth, Natural Resources Canada, Invited Expert, Observer
  Hiroshi Sakakibara, Beyond Perspective Solutions(BPS)
  Simon Sapin, Mozilla
  Dirk Schulze, Adobe
  Markus Stange, Mozilla
  Alan Stearns, Adobe
  Aleks Totic, Google
  Lea Verou, Invited Expert
  Matt Woodrow, Mozilla
  Fuqiao Xue, W3C

Scribe: birtles

Joint Meeting with WPT
======================

  <astearns> foolip presenting
             https://docs.google.com/presentation/d/1F1II3lx0krPBIqAzgU9YFIENC-UBEnZtN796hye_bWw/edit?usp=sharing
  archived at https://lists.w3.org/Archives/Public/www-archive/2019Oct/att-0001/wpt-review-2019.pdf

  <astearns> +1 to wpt documentation improvements. It's much much
             better!

  fantasai: Thank you for updating the docs
  fantasai: It's great to have everything up front
  fantasai: It would be nice to have a navigation system like a tab
            bar or footer
  fantasai: to connect them all
  fantasai: It would be easier to get from wpt.fyi to the docs and back

  fantasai: Second comment: do we have any plans for testing printing
            or paginated modes?
  jgraham: We have talked about it a bit

  fantasai: Do we have any way of helping people find and run and
            report results for manual tests
  fantasai: There are some things that are not automated and are
            unlikely to be automated in the near future
  foolip: I know Florian does a lot of manual tests
  foolip: The answer is no
  foolip: and it doesn't report results in the right format
  foolip: There are not so many building blocks missing
  fantasai: Support for manual testing can close the gaps for the types
            of the tests for which we don't have infrastructure
  fantasai: e.g. printing and media queries

  fantasai: I don't know how we'll test media queries
  fantasai: a person has to run those
  fantasai: e.g. plugging in a monitor
  fantasai: We don't have to run these constantly, but need to run at
            least once
  fantasai: and maybe a little more frequently, e.g. once a year

  foolip: Would it be acceptable if you have to use wpt to run the
          test?
  foolip: locally wpt run
  foolip: ./wpt run --manual would be less work than a website approach
  fantasai: It would be nice if you can load the test on a website
  fantasai: but the important thing would be to find the manual tests
            and then compile a result
  fantasai: If you can accept data from Peter's system into wpt
            reporting system
  fantasai: that might be a way
  foolip: Just converting them would be straightforward I guess
  foolip: but do you want to keep maintaining that?
  plinss: I would be happy for it to be replaced
  plinss: once its functions are covered
  plinss: It tracks what git revision of each test is used
  jgraham: Perhaps we can follow up later
  fantasai: Minimum requirement is to get a list of manual tests then
            provide a report of results
  foolip: Would it be fine to get all the manual tests in a specific
          directory?
  fantasai: Yes

  florian: If the system is designed so that you have to run all of
           them, that wouldn't be so convenient
  florian: There are thousands of tests in writing-modes for example
  foolip: If there's no subdirectories then we can maybe use chunking
  jgraham: The requirements to run this probably need to be considered
           in more depth
  jgraham: There are data model considerations with regards to wpt.fyi
  jgraham: There are lots of technical details
  florian: I think it helps breaking the problem down because
           eventually we want to replace and retire Peter's system
  florian: even working out how to display manual tests is an
           interesting problem
  florian: there might be conflicting reports
  florian: Using an existing data source and then working out how to
           display them later
  astearns: We do need to work out how to fix this

  astearns: I want to second the joy at the improved documentation
  astearns: I went through the docs to submit a reftest recently and
            everything was there
  astearns: when I submitted a PR, however, something broken and I
            didn't know what to do
  astearns: I could see the test results, I submitted another patch
            and it passed
  astearns: but I don't know how to fix it the first time
  astearns: The first two checks were red and I didn't know what to do
            about it
  astearns: I didn't want to bother anyone
  foolip: Please bother us
  foolip: We need to fix this
  jgraham: Some of this have failed for reasons out of our control
  jgraham: e.g. GitHub actions being beta
  jgraham: sometimes it takes quite a bit of experience to work out
           what went wrong
  jgraham: There are infrastructure problems but there are also things
           we can do better
  dbaron: Is pinging you on #testing the way to reach you?
  jgraham: Yes
  foolip: Or on the PR
  jgraham: IRC is generally more likely to get attention
  jgraham: Can get lots on GitHub email
  zcorpan: It depends a bit on what it is
  zcorpan: If it's something you see repeatedly
  zcorpan: it might be a system bug we need to fix
  zcorpan: otherwise ping one of us on the PR / IRC

  florian: In addition to the manual testing, we need 2 more things to
           retire Peter's tool
  florian: One is MUST tests vs SHOULD/MAY
  florian: important for exiting CR
  florian: The other is that wpt.fyi groups report by directory
  florian: but using the metadata pulling results by spec link, not
           just directory would be helpful
  florian: then we can retire Peter's tool
  jgraham: We are working on associating labels with tests
  jgraham: I think that would cover these cases
  jgraham: If you want that information to be in the test files rather
           than in metadata
  jgraham: then you could have a bot to check they are in sync
  florian: Currently that data is in the test, the flags meta
  florian: so we need to sync that with labels
  jgraham: That labelling system is quite limited but it is being
           worked on
  jgraham: We could have a script to check they are synced
  foolip: If you can search for not labeled tests is that fine?
  florian: If there is a button to click that's better, but as long as
           it's do-able

  florian: One more question about fuzzy tests
  florian: For non-fuzzy reftests we often have problems where one
           side is done with Ahem and one is not
  florian: but the box is quite large so many pixels are affected
  florian: is that covered by fuzzy?
  jgraham: There are two parameters in fuzzy reftests -- one is the
           total number of pixels you allow to differ
  jgraham: One is the amount of you allow in difference per color
           channel
  jgraham: so in this case you would allow a very small range in
           difference per color channel
  jgraham: If you specify one number it means exactly this number must
           be difference, but you can specify a range, e.g. from 0
  jgraham: The other thing is that in Gecko we can have meta files
           that layer over the wpt tests so we can apply different
           fuzzy parameter for Gecko only
  jgraham: e.g. we know our form controls differ
  florian: For the color channel parameter, it sounds like what I want
           to do
  florian: but getting the right numbers might be difficult
  florian: If we can find a way to make that easier that would be good
  jgraham: The screenshot tool could do that
  dbaron: The gecko reftest failure tells you that -- the failure range
  dbaron: makes it easy to know what numbers to use
  AmeliaBR: For SVG I have been planning to work around this for SVG
            reftests
  AmeliaBR: by adding a stroke overline to cover the anti-aliased part
  AmeliaBR: Not the ideal solution
  AmeliaBR: but you can block off the regions where you don't care
            about differences

  emilio: Thank you for all your work
  emilio: The tests that keep going to Gecko's internal test suite are
          crashtests
  emilio: Is there any way we can convert them to wpt in the future
  jgraham: I think for crashtests the thinking is that it's normally a
           very browser-specific thing
  jgraham: There might be cases where that's not true
  jgraham: if we think it's worth doing it's quite do-able
  emilio: I can try to see how many of the Gecko crashtests affect
          other browsers

  smfr: Thanks for all your hard work
  smfr: getting Safari's stuff working right is great
  smfr: I want to confirm that if a run fails in the middle there's a
        clear differentiation between failed tests and not run
  foolip: If it fails in the middle we don't get results...
  smfr: I'm interested in cases where a test doesn't not run... it
        should show up as a fail
  foolip: A missing test is better than a fail

  smfr: Just following up on AmeliaBR's point
  smfr: Sometimes there are differences we do care about in a test
        with fuzzy parts
  smfr: A number is not sufficient, sometimes you want to obscure the
        irrelevant part
  jgraham: The fuzzy thing is a useful thing, it works in Gecko
  jgraham: and when running it locally it will spit out the numbers
           you need to annotate them
  dbaron: The fuzzy number notation is very useful when they are exact
          numbers
  dbaron: since if you get any other kind of number you get an
          unexpected pass

  majidvp: Thank you for adding testdriver annotation
  majidvp: For writing scrolling tests, we often need to inject input
  majidvp: you support pointer actions, what about mouse
  foolip: If pointer actions supports it we can do it
  jgraham: If it's not in webdriver we can add a feature request for
           that

Received on Tuesday, 3 December 2019 23:13:37 UTC