W3C home > Mailing lists > Public > public-houdini@w3.org > October 2015

Minutes Paris F2F 2015-08-29 Part III: Scroll Customization, Rendering Pipeline, Passing Data Between Custom Phases

From: Dael Jackson <daelcss@gmail.com>
Date: Fri, 16 Oct 2015 19:44:54 -0400
Message-ID: <CADhPm3tC1CT=gBVqoq5TSfPXpnJoPPnxSrc7hEcYCr-X0Oi0Ow@mail.gmail.com>
To: public-houdini@w3.org
Scroll Customization
--------------------

  - The spec has moved to focus on just doing it on the composite
      worker.
  - There was interest in coming up with an alternate to composite
      workers for the spec which will be worked on outside the F2F.

Rendering Pipeline
------------------

  - Ojan brought his proposal for standardizing the rendering
      pipeline across browsers to allow developers to create a
      consistent mental model for how rendering is handled.
  - It was agreed to be interesting and something needed for
      Houdini, but not necessarily a part of Houdini. It was instead
      suggested this should replace scripted animations and have
      HTML refer to this or update the webapps spec to this model.
  - Canvas and webGL should probably be a part of this proposed spec.
  - The use cases of handling of an offscreen iframe that moves
      onscreen and throttling an offscreen iframe need to be
      addressed.
  - Ojan will work with webapps to further work on this spec.
  - RESOLVED: Add Render Processing Model as a new module (ED)

Passing Data Between Custom Phases
----------------------------------

  - There were use cases for having layout information at paint
      time, but it was agreed that this isn't far enough along to
      discuss with the group.

===== FULL MINUTES BELOW ======

  scribe: dael

Scroll Customization
--------------------

  Rossen: Let's get going
  rbyers: I can do a quick 2min update. Last time we talked we
          talked about out scroll customization we were doing it on
          the main thread and on the composite worker. The main
          thread is on the back burner. We think we can do a better
          job shifting the ecosystem before we ship.
  rbyers: The API is dependent on the composite worker so we're
          waiting for that. It keeps shifting at the moment. We're
          implementing snap points on top of scroll customization
          and we're learning a lot from that because scrolling
          interacts a lot on the web.
  rbyers: Once we work through that we can come back with a proposal.

  smfr: The thing I wanted to discuss was alternatives to composite
        workers. This comes back to the existential question, but at
        some point I'd like people to think if there are other ways
        to do this. There are things like synchronous media playback
        to scrolls that haven't been addressed. I don't have a
        concrete proposal.
  rbyers: Do you think you could? It would be great if we could have
          a shared set of use cases. Like the pull to refresh case,
          I'm not sure how to do that without composite workers, but
          I think you're smarter so I'd like to see what you come up
          with.
  rbyers: Pull to refresh, our focus was lets look at the common
          native apps. It's not used in 90% of apps, but it is very
          common. We're not happy unless we address that scenario.
          Those apps, like Twitter, in iOS have a very nice pull to
          refresh.
  dino: We can do our system pull to refresh. The tricky bit is when
        you commit to the refresh. In iOS 7 I think you repainted.

  rbyers: The other tricky thing is the composition. What if you're
          scrolling an iframe and than you hit the main document and
          you have a pull to refresh.
  smfr: We have a behavior called latching where a single user
        action never does more than one thing.
  esprehn: Can we do that?
  flackr: You'd still have the same behavior, but it's more well
          defined.
  TabAtkins: What do you all do will a scrollable text area larger
             than the screen? Oh, you need a second gesture. Got it.

Rendering Pipeline
------------------

  <rbyers> Rendering pipeline doc:
https://docs.google.com/document/d/1Mw6qNw8UAEfW96CXaXRVYPPZjqQS3YdK7v57wFttAhs/edit#heading=h.1qfloic3tvvr
  ojan: I'm not sure this is the appropriate WG for this, but let's
        talk through it and decide if it's something to do.
  ojan: We have different processing models today, like what order
        do things happen in. We'd like to standardize so we get
        consistency and dev can reason about the platform. This is a
        proposal. We're not attached to the spec but more let's get
        consistency in implementations.
  ojan: This is what blink does. When a vsync happens we update the
        animation time, we fire paint-linked events, and then we
        fire requestAnimationFrame. We do computed style update,
        layout, advance animations, and then paint.
  ojan: I realize different browsers use different terms.
  dbaron: If one event has handlers that ask for up to date style
          layout presumably you'll flush animation then and flush it
          again later?
  ojan: Yes.
  ojan: We want to propose you can't just paint or just
        requestAnimationFrame, you have to do the whole pipeline. We
        think this helps devs think.
  ojan: The other thing to point out is there is nothing else that
        can interject into the pipeline. It makes for a platform
        people can reason about.
  ojan: I think that's roughly all there is for high level. Assuming
        we agree on some subset I'd want to turn this into a proper
        spec.

  Proposed pipeline:
  __________________________________________________________________
 |                    | Update Animation Frame                      |
 |                    |---------------------------------------------|
 | Callbacks (step 1) | Per Frame Events (scroll, window size, etc.)|
 |                    |---------------------------------------------|
 |                    | requestAnimationFrame                       |
 |__________________________________________________________________|
 |                    | Update Style                                |
 |                    |---------------------------------------------|
 |   Layout (step 2)  | Update and Advance Animations               |
 |                    |---------------------------------------------|
 |                    | Update Layout                               |
 |__________________________________________________________________|
 |    Paint (step 3)  | Paint/Record                                |
 |__________________________________________________________________|

  <smfr> https://html.spec.whatwg.org/multipage/webappapis.html#event-loops
  smfr: Web apps has the event loop and specifies when request
        animation fires. It seems this needs to build on that.
  ojan: It could.
  dbaron: The animation chunk, is that for animations on the
          compositor?
  esprehn: That's for all of them.
  smfr: If you're running transitions it's the result of style.
  ojan: If you're on the compositor you're not in this.
  esprehn: So the time step sets the time for the entire animation.
  dbaron: So for use the start happens inside update style.
  Rossen: Same for us.
  dbaron: I think the idea is once we've updated the animation time
          if you flush style or layout you get the new stuff.
  ojan: Maybe the 'animate' phase doesn't make sense.

  esprehn: The only way this is observable is with iframes.
  esprehn: Each colored bar here processes the entire frame tree
           which gives you predictable observations of the same
           animate-able iframes.
  ojan: I don't think you can in the animation case.
  shane: Advancing animations can impact layout so it has to be
         before then.
  ojan: Ok.
  esprehn: Animations are always one frame behind. This requires
           that they're fired at requestAnimation time.
  smfr: If you get computed style do you see the first keyframe or
        the new keyframe?
  ojan: I'm looking to codify what browsers do. There are some cases
        I'm saying this is what I think we should do but only when
        we don't agree.

  dbaron: This is a good thing and I feel this is the right group
          because it has the right people.
  smfr: It's not Houdini, but it's the right stuff.
  ojan: There's some overlap, but it's a stretch, I admit.
  Florian: It's not a Houdini spec, but it's one Houdini needs.
  dino: It should be in with the micro tasks stuff in HTML.
  ojan: I don't think micro tasks have a relation.
  dino: They describe when a raf is fired.
  ojan: I think they're disjointed, but there is a spec that talks
        about raf.
  esprehn: It was defined by mutation observers and then hoisted.
  dino: HTML spec describe when raf events fire.
  <zcorpan. https://html.spec.whatwg.org/multipage/webappapis.html#processing-model-9
  <esprehn> http://w3c.github.io/animation-timing/
  <iank> https://html.spec.whatwg.org/multipage/webappapis.html#animation-frames
  ojan: Could we aim to replace that part of the spec or have it
        refer to this?
  heycam: The spec hasn't been touched for a long time.
  heycam: It would make sense to merge with the animation taskforce.

  ojan: I propose we write a spec and it replaces scripted
        animations or whatever it's called and we'll try and get
        html to refer to this spec.
  smfr: Or we update the webapps spec for this.
  ojan: I'm okay with that too, it's more overhead.
  ojan: Do the restrictions I suggested seem reasonable to the
        implementors? I listed the changes off the top of my head
        that implementors would have to do at the bottom.
  ojan: A concrete example, in this model you couldn't throttle just
        request animation, you'd have to do the whole thing.
  ojan: Main thread script.

  vollick: Can you clarify what you mean by frame?
  ojan: What's missing in this diagram is a separate pipeline for
        threaded scrolling that links to this one.
  dbaron: I think the interjection thing in Firefox means we do the
          painting separate.
  <dbaron> (I'm not sure)
  Rossen: Same for us.
  Rossen: Our paint is off main thread.
  esprehn: This paint approach is aggregate the information
           necessary to paint, not actually paint.
  <dbaron> Actually, it looks like we do do the painting
           synchronously within that process -- although maybe it
           can happen at other points as well...
  Rossen: Visually in our data the paint is behind by a few frames.
          You can have layout change a bit while you're rendering
          five frames before. You'll catch up.
  esprehn: The screen picture doesn't have to be current. The screen
           will never show a document that wasn't snapshotted at the
           end of this.

  ojan: You can't have a requestAnimationFrame and you do the layout
        part of the pipeline and you're doing your painting
        asynchronously and have a script run in the layout part that
        modifies it is the proposal.
  Rossen: I think this is a logical set up steps, but insisting that
          they always complete and run sync is a bit..
  ojan: A use case. You ought to be able to rely on that the
        requestAnimationFrame fires immediately before paint. I
        think what you're saying violates that.
  Rossen: No. you have a sentence somewhere...the entire tree should
          be processed.
  ojan: If I said that I don't mean it.
  Rossen: [reads first bullet]

  ojan: Imagine you have requestAnimationFrames for different frames.
        You can't do one frame first a requestAnimationFrame. We
        want to change this for across iframes because everyone is
        moving to throttle, but throttle is for the whole pipeline,
        not just the raf.
  rbyers: It might help to cast this in what are the guar.that
          authors can rely on
  ojan: There's the author assurances section.
  rbyers: So we reword this a bit. What we mean is timers can't
          insert themselves after requestAnimationFrame and still
          have an effect users can see.
  ojan: That's a good way, I think paint is the wrong word.
  <dbaron> yeah, I think the problem is that we have other entry
           points into our painting process that don't come from our
           refresh pipeline (from
           nsViewManager::UpdateWidgetGeometry and
           nsViewManager::WillPaintWindow)

  smfr: There's nothing about media playback or webGL and do we care?
  ojan: Probably. It's not there because I'm not familiar with it.
  esprehn: Media playback is more complicated. That might be a
           version 2 and it may be impossible if you're using system
           overlays.
  smfr: I think canvas drawing should be included. So drawing to a
        canvas that has a request animation should get done.
  ojan: It makes sense that canvas and webGL should do this.

  smfr: Are you saying a UA wouldn't be able to throttle the iframe?
  ojan: Not just the callback.
  ojan: So you have an offscreen iframe. When it gets put on screen,
        was the requestAnimationFrame able to interject before the
        user could see?
  esprehn: You will not get scroll events and animation events if
           the requestAnimationFrame is not running.
  smfr: We would want to be able to throttle iframes and
        requestAnimationFrames. I think it's quite likely you could
        scroll stale pixels.
  ojan: When the stale pixels got generated they were from after the
        requestAnimationFrame was run.
  ojan: So they will use requestAnimationFrame to take a web
        component thing into a thing the user sees and if the frame
        doesn't run they get junky fallback.
  smfr: I think we still did layout on the iframe.
  ojan: That's fine. You don't turn it into pixels.
  smfr: We would have at that point.
  ojan: That's something I proposing we won't do.
  esprehn: There's a lot of discussion to be had for how we handle
           that case. But showing blank might be better.
  smfr: I think showing pixels from when you scroll offscreen if
        better.
  esprehn: But if you lost the pixel backing that's worse.

  dbaron: I was poking around in Gecko's code. The blue stuff
          [callbacks] I haven't checked. With paint we have
          additional places that call into that that aren't from the
          refresh cycle but are various widget stuff. It may be
          things like window resizes.
  esprehn: We're proposing in all those cases you're required to run.
  dbaron: I think that's reasonable, but that may be 6 months work.
  ojan: I think it is. It's not that I expect people to shift
        tomorrow, I want us to agree on the behavior and try and
        work toward it. Similar to the new HTML parser. Everyone
        agreed to a spec and implemented it eventually.

  Rossen: Next steps?
  esprehn: We should merge this and the processing model, which I'm
           not sure matches any browser, and then build a
           conformance suite. And the Houdini specs can invoke
           portions of this.

  ojan: smfr, you seem not sold.
  smfr: I'd like to be able to throttle offscreen iframes without
        too much trouble
  ojan: Let's talk offline. I think that's an issue that needs to be
        solved.

  dbaron: I'd like to get feedback from others.
  rbyers: What if we write the perf test suite and we can understand
          where people violate it and debate over that.
  esprehn: I have a bunch of tests for this. Safari is sensitive to
           the tree. Some browsers timeout. These tests are hard to
           automate.
  ojan: My only hesitation is that there are some big things we
        should talk about it.
  rbyers: I didn't realize esprehn had lots of tests.
  ojan: At a higher level what I've heard from implementors is you
        want to run it by experts, but this in a wide view seems
        reasonable, though there are lots of bugs that need
        addressing.
  <brucel> author feedback on browser pipline spec: "Thrice yes! As
           Roy Castle used to sing "predictability is what we need"."
           https://twitter.com/deanleigh/status/637593952238051328

  dbaron: I don't know if syncXHR throws some fun into this.
  esprehn: I'm comfortable saying during syncXHR there's no
           guarantee.
  smfr: I think when an image loads layout can be invalidated.

  ojan: I'm not asking you to commit, I'm asking if high level you
        have concerns.
  Rossen: I think the feedback is this 1,2,3 is nice. Write it up
          and come back and we'll beat up on it more.

  zcorpan: There is an outstanding issue with the existing sync spec
           when Mozilla tried to update.
  rbyers: He was saying he caused performance problems.
  ojan: We haven't seen any, but that doesn't mean we know.
  smfr: Your scroll events appear to be sync.
  ojan: They're right before a requestAnimationFrame.
  rbyers: We attempt to delay posting a frame when there's a scroll
          event handler, but if you make that complex we'd get jenky
          too.
  ojan: I think the janky is just bugs. I think it's CPU load.
  esprehn: Our scheduler has a bunch of complicated cases that tries
           to give priority to animation events.
  ojan: It's how long it waits to decide if the main thread went to
        long.
  rbyers: That's about authors, though.
  ojan: It's the can I reason about the model.
  Rossen: We're off topic.

  Rossen: I think we have an agreement to work on this. You
          mentioned pulling this to webapps
  smfr: It builds on top of existing text on webapps.
  ojan: I think we create a pull request on webapps or have an open
        issue on webapps that says this doesn't match what webapps
        does.
  Rossen: Are you offering?

  ACTION ojan to work with webapps in furthering the work on this
         topic.

  Rossen: And we need to have this as an ED.
  Rossen: What would be the name?
  ojan: We called it the browser rendering pipeline.
  ojan: I don't feel strongly..
  Rossen: What do people feel comfortable with a name. Render model?

  RESOLVED: Add Render Processing Model as a new module (ED)

  zcorpan: Things that cause synchronous layout is not current spec
           in HTML or CSSOM and we need to figure that out. If in
           the middle of your script you change the style and ask
           for layout info. It's a bit handwave-y and we need to
           spec properly
  ojan: Should this be part of this spec?
  zcorpan: It's related.
  gregwhitworth: The OM could point to this.
  ojan: This would give a place for specs to point to.

Passing Data Between Custom Phases
----------------------------------

  shane: I think we've had conversations about this.
  Rossen: Offline?
  shane: Yeah.
  shane: Is there anything we need to talk about?
  Rossen: I don't know, I wasn't there.
  shane: Why don't we draw on the whiteboard and see.
  Rossen: Perhaps re-hash the conversation?
  shane: I wasn't there either.

  shane: It seems like we're building out this picture of phases
         starting from simple to complex property at the style
         resolution level. We've got this picture of using custom
         properties to pass data in and possibly some side channels.
  shane: We haven't specifically talked about being able to generate
         additional data and if you can make changes with custom
         layout.
  shane: Is this stuff we should talk about now or details to hash
         out now?

  smfr: Do we have use cases for passing custom data?
  shane: I think at the style level we might. The compositor might.
         But layout to paint there isn't any need yet.
  smfr: And there are use cases for having layout information at
        paint time?
  shane: Yep. That's true.
  smfr: It sounds to me that we're not far enough along. At certain
        times we've talked about if we should store data from one
        hook to another and maybe have hooks communicate somehow.
  shane: We had proposed a caching API to store data between
         iterations of the same hook. At lunch there was talk about
         forming trees to pass data. I think this is very much brain
         storming. There's nothing specific here. I think we should
         move on.
  Rossen: It's your topic. If you have nothing else we can move on.

  Rossen: Next topic is 'some tasks need information from outside
          the element.
  rbyers: It sounds related to the previous topic
  Vollick: We touched on it a bit with async.
  shane: We've got this prioritization...I was going to suggest we
         do the next topic and split to discussion groups.
  Rossen: If we go through prioritization we can adjourn and people
          will group.

  <franremy> @Rossen: if we've some time, can I just get 2min of
             time after current topic to add a new one (Main-Thread
             Style Mutation Observer, similar to
             CustomPaint/ApplyHook/... dependencies but executed on
             the main thread?) and explain why I think it may be
             useful
Received on Friday, 16 October 2015 23:45:52 UTC

This archive was generated by hypermail 2.3.1 : Friday, 16 October 2015 23:45:53 UTC