- From: Dael Jackson <daelcss@gmail.com>
- Date: Fri, 16 Oct 2015 19:44:54 -0400
- To: public-houdini@w3.org
Scroll Customization
--------------------
- The spec has moved to focus on just doing it on the composite
worker.
- There was interest in coming up with an alternate to composite
workers for the spec which will be worked on outside the F2F.
Rendering Pipeline
------------------
- Ojan brought his proposal for standardizing the rendering
pipeline across browsers to allow developers to create a
consistent mental model for how rendering is handled.
- It was agreed to be interesting and something needed for
Houdini, but not necessarily a part of Houdini. It was instead
suggested this should replace scripted animations and have
HTML refer to this or update the webapps spec to this model.
- Canvas and webGL should probably be a part of this proposed spec.
- The use cases of handling of an offscreen iframe that moves
onscreen and throttling an offscreen iframe need to be
addressed.
- Ojan will work with webapps to further work on this spec.
- RESOLVED: Add Render Processing Model as a new module (ED)
Passing Data Between Custom Phases
----------------------------------
- There were use cases for having layout information at paint
time, but it was agreed that this isn't far enough along to
discuss with the group.
===== FULL MINUTES BELOW ======
scribe: dael
Scroll Customization
--------------------
Rossen: Let's get going
rbyers: I can do a quick 2min update. Last time we talked we
talked about out scroll customization we were doing it on
the main thread and on the composite worker. The main
thread is on the back burner. We think we can do a better
job shifting the ecosystem before we ship.
rbyers: The API is dependent on the composite worker so we're
waiting for that. It keeps shifting at the moment. We're
implementing snap points on top of scroll customization
and we're learning a lot from that because scrolling
interacts a lot on the web.
rbyers: Once we work through that we can come back with a proposal.
smfr: The thing I wanted to discuss was alternatives to composite
workers. This comes back to the existential question, but at
some point I'd like people to think if there are other ways
to do this. There are things like synchronous media playback
to scrolls that haven't been addressed. I don't have a
concrete proposal.
rbyers: Do you think you could? It would be great if we could have
a shared set of use cases. Like the pull to refresh case,
I'm not sure how to do that without composite workers, but
I think you're smarter so I'd like to see what you come up
with.
rbyers: Pull to refresh, our focus was lets look at the common
native apps. It's not used in 90% of apps, but it is very
common. We're not happy unless we address that scenario.
Those apps, like Twitter, in iOS have a very nice pull to
refresh.
dino: We can do our system pull to refresh. The tricky bit is when
you commit to the refresh. In iOS 7 I think you repainted.
rbyers: The other tricky thing is the composition. What if you're
scrolling an iframe and than you hit the main document and
you have a pull to refresh.
smfr: We have a behavior called latching where a single user
action never does more than one thing.
esprehn: Can we do that?
flackr: You'd still have the same behavior, but it's more well
defined.
TabAtkins: What do you all do will a scrollable text area larger
than the screen? Oh, you need a second gesture. Got it.
Rendering Pipeline
------------------
<rbyers> Rendering pipeline doc:
https://docs.google.com/document/d/1Mw6qNw8UAEfW96CXaXRVYPPZjqQS3YdK7v57wFttAhs/edit#heading=h.1qfloic3tvvr
ojan: I'm not sure this is the appropriate WG for this, but let's
talk through it and decide if it's something to do.
ojan: We have different processing models today, like what order
do things happen in. We'd like to standardize so we get
consistency and dev can reason about the platform. This is a
proposal. We're not attached to the spec but more let's get
consistency in implementations.
ojan: This is what blink does. When a vsync happens we update the
animation time, we fire paint-linked events, and then we
fire requestAnimationFrame. We do computed style update,
layout, advance animations, and then paint.
ojan: I realize different browsers use different terms.
dbaron: If one event has handlers that ask for up to date style
layout presumably you'll flush animation then and flush it
again later?
ojan: Yes.
ojan: We want to propose you can't just paint or just
requestAnimationFrame, you have to do the whole pipeline. We
think this helps devs think.
ojan: The other thing to point out is there is nothing else that
can interject into the pipeline. It makes for a platform
people can reason about.
ojan: I think that's roughly all there is for high level. Assuming
we agree on some subset I'd want to turn this into a proper
spec.
Proposed pipeline:
__________________________________________________________________
| | Update Animation Frame |
| |---------------------------------------------|
| Callbacks (step 1) | Per Frame Events (scroll, window size, etc.)|
| |---------------------------------------------|
| | requestAnimationFrame |
|__________________________________________________________________|
| | Update Style |
| |---------------------------------------------|
| Layout (step 2) | Update and Advance Animations |
| |---------------------------------------------|
| | Update Layout |
|__________________________________________________________________|
| Paint (step 3) | Paint/Record |
|__________________________________________________________________|
<smfr> https://html.spec.whatwg.org/multipage/webappapis.html#event-loops
smfr: Web apps has the event loop and specifies when request
animation fires. It seems this needs to build on that.
ojan: It could.
dbaron: The animation chunk, is that for animations on the
compositor?
esprehn: That's for all of them.
smfr: If you're running transitions it's the result of style.
ojan: If you're on the compositor you're not in this.
esprehn: So the time step sets the time for the entire animation.
dbaron: So for use the start happens inside update style.
Rossen: Same for us.
dbaron: I think the idea is once we've updated the animation time
if you flush style or layout you get the new stuff.
ojan: Maybe the 'animate' phase doesn't make sense.
esprehn: The only way this is observable is with iframes.
esprehn: Each colored bar here processes the entire frame tree
which gives you predictable observations of the same
animate-able iframes.
ojan: I don't think you can in the animation case.
shane: Advancing animations can impact layout so it has to be
before then.
ojan: Ok.
esprehn: Animations are always one frame behind. This requires
that they're fired at requestAnimation time.
smfr: If you get computed style do you see the first keyframe or
the new keyframe?
ojan: I'm looking to codify what browsers do. There are some cases
I'm saying this is what I think we should do but only when
we don't agree.
dbaron: This is a good thing and I feel this is the right group
because it has the right people.
smfr: It's not Houdini, but it's the right stuff.
ojan: There's some overlap, but it's a stretch, I admit.
Florian: It's not a Houdini spec, but it's one Houdini needs.
dino: It should be in with the micro tasks stuff in HTML.
ojan: I don't think micro tasks have a relation.
dino: They describe when a raf is fired.
ojan: I think they're disjointed, but there is a spec that talks
about raf.
esprehn: It was defined by mutation observers and then hoisted.
dino: HTML spec describe when raf events fire.
<zcorpan. https://html.spec.whatwg.org/multipage/webappapis.html#processing-model-9
<esprehn> http://w3c.github.io/animation-timing/
<iank> https://html.spec.whatwg.org/multipage/webappapis.html#animation-frames
ojan: Could we aim to replace that part of the spec or have it
refer to this?
heycam: The spec hasn't been touched for a long time.
heycam: It would make sense to merge with the animation taskforce.
ojan: I propose we write a spec and it replaces scripted
animations or whatever it's called and we'll try and get
html to refer to this spec.
smfr: Or we update the webapps spec for this.
ojan: I'm okay with that too, it's more overhead.
ojan: Do the restrictions I suggested seem reasonable to the
implementors? I listed the changes off the top of my head
that implementors would have to do at the bottom.
ojan: A concrete example, in this model you couldn't throttle just
request animation, you'd have to do the whole thing.
ojan: Main thread script.
vollick: Can you clarify what you mean by frame?
ojan: What's missing in this diagram is a separate pipeline for
threaded scrolling that links to this one.
dbaron: I think the interjection thing in Firefox means we do the
painting separate.
<dbaron> (I'm not sure)
Rossen: Same for us.
Rossen: Our paint is off main thread.
esprehn: This paint approach is aggregate the information
necessary to paint, not actually paint.
<dbaron> Actually, it looks like we do do the painting
synchronously within that process -- although maybe it
can happen at other points as well...
Rossen: Visually in our data the paint is behind by a few frames.
You can have layout change a bit while you're rendering
five frames before. You'll catch up.
esprehn: The screen picture doesn't have to be current. The screen
will never show a document that wasn't snapshotted at the
end of this.
ojan: You can't have a requestAnimationFrame and you do the layout
part of the pipeline and you're doing your painting
asynchronously and have a script run in the layout part that
modifies it is the proposal.
Rossen: I think this is a logical set up steps, but insisting that
they always complete and run sync is a bit..
ojan: A use case. You ought to be able to rely on that the
requestAnimationFrame fires immediately before paint. I
think what you're saying violates that.
Rossen: No. you have a sentence somewhere...the entire tree should
be processed.
ojan: If I said that I don't mean it.
Rossen: [reads first bullet]
ojan: Imagine you have requestAnimationFrames for different frames.
You can't do one frame first a requestAnimationFrame. We
want to change this for across iframes because everyone is
moving to throttle, but throttle is for the whole pipeline,
not just the raf.
rbyers: It might help to cast this in what are the guar.that
authors can rely on
ojan: There's the author assurances section.
rbyers: So we reword this a bit. What we mean is timers can't
insert themselves after requestAnimationFrame and still
have an effect users can see.
ojan: That's a good way, I think paint is the wrong word.
<dbaron> yeah, I think the problem is that we have other entry
points into our painting process that don't come from our
refresh pipeline (from
nsViewManager::UpdateWidgetGeometry and
nsViewManager::WillPaintWindow)
smfr: There's nothing about media playback or webGL and do we care?
ojan: Probably. It's not there because I'm not familiar with it.
esprehn: Media playback is more complicated. That might be a
version 2 and it may be impossible if you're using system
overlays.
smfr: I think canvas drawing should be included. So drawing to a
canvas that has a request animation should get done.
ojan: It makes sense that canvas and webGL should do this.
smfr: Are you saying a UA wouldn't be able to throttle the iframe?
ojan: Not just the callback.
ojan: So you have an offscreen iframe. When it gets put on screen,
was the requestAnimationFrame able to interject before the
user could see?
esprehn: You will not get scroll events and animation events if
the requestAnimationFrame is not running.
smfr: We would want to be able to throttle iframes and
requestAnimationFrames. I think it's quite likely you could
scroll stale pixels.
ojan: When the stale pixels got generated they were from after the
requestAnimationFrame was run.
ojan: So they will use requestAnimationFrame to take a web
component thing into a thing the user sees and if the frame
doesn't run they get junky fallback.
smfr: I think we still did layout on the iframe.
ojan: That's fine. You don't turn it into pixels.
smfr: We would have at that point.
ojan: That's something I proposing we won't do.
esprehn: There's a lot of discussion to be had for how we handle
that case. But showing blank might be better.
smfr: I think showing pixels from when you scroll offscreen if
better.
esprehn: But if you lost the pixel backing that's worse.
dbaron: I was poking around in Gecko's code. The blue stuff
[callbacks] I haven't checked. With paint we have
additional places that call into that that aren't from the
refresh cycle but are various widget stuff. It may be
things like window resizes.
esprehn: We're proposing in all those cases you're required to run.
dbaron: I think that's reasonable, but that may be 6 months work.
ojan: I think it is. It's not that I expect people to shift
tomorrow, I want us to agree on the behavior and try and
work toward it. Similar to the new HTML parser. Everyone
agreed to a spec and implemented it eventually.
Rossen: Next steps?
esprehn: We should merge this and the processing model, which I'm
not sure matches any browser, and then build a
conformance suite. And the Houdini specs can invoke
portions of this.
ojan: smfr, you seem not sold.
smfr: I'd like to be able to throttle offscreen iframes without
too much trouble
ojan: Let's talk offline. I think that's an issue that needs to be
solved.
dbaron: I'd like to get feedback from others.
rbyers: What if we write the perf test suite and we can understand
where people violate it and debate over that.
esprehn: I have a bunch of tests for this. Safari is sensitive to
the tree. Some browsers timeout. These tests are hard to
automate.
ojan: My only hesitation is that there are some big things we
should talk about it.
rbyers: I didn't realize esprehn had lots of tests.
ojan: At a higher level what I've heard from implementors is you
want to run it by experts, but this in a wide view seems
reasonable, though there are lots of bugs that need
addressing.
<brucel> author feedback on browser pipline spec: "Thrice yes! As
Roy Castle used to sing "predictability is what we need"."
https://twitter.com/deanleigh/status/637593952238051328
dbaron: I don't know if syncXHR throws some fun into this.
esprehn: I'm comfortable saying during syncXHR there's no
guarantee.
smfr: I think when an image loads layout can be invalidated.
ojan: I'm not asking you to commit, I'm asking if high level you
have concerns.
Rossen: I think the feedback is this 1,2,3 is nice. Write it up
and come back and we'll beat up on it more.
zcorpan: There is an outstanding issue with the existing sync spec
when Mozilla tried to update.
rbyers: He was saying he caused performance problems.
ojan: We haven't seen any, but that doesn't mean we know.
smfr: Your scroll events appear to be sync.
ojan: They're right before a requestAnimationFrame.
rbyers: We attempt to delay posting a frame when there's a scroll
event handler, but if you make that complex we'd get jenky
too.
ojan: I think the janky is just bugs. I think it's CPU load.
esprehn: Our scheduler has a bunch of complicated cases that tries
to give priority to animation events.
ojan: It's how long it waits to decide if the main thread went to
long.
rbyers: That's about authors, though.
ojan: It's the can I reason about the model.
Rossen: We're off topic.
Rossen: I think we have an agreement to work on this. You
mentioned pulling this to webapps
smfr: It builds on top of existing text on webapps.
ojan: I think we create a pull request on webapps or have an open
issue on webapps that says this doesn't match what webapps
does.
Rossen: Are you offering?
ACTION ojan to work with webapps in furthering the work on this
topic.
Rossen: And we need to have this as an ED.
Rossen: What would be the name?
ojan: We called it the browser rendering pipeline.
ojan: I don't feel strongly..
Rossen: What do people feel comfortable with a name. Render model?
RESOLVED: Add Render Processing Model as a new module (ED)
zcorpan: Things that cause synchronous layout is not current spec
in HTML or CSSOM and we need to figure that out. If in
the middle of your script you change the style and ask
for layout info. It's a bit handwave-y and we need to
spec properly
ojan: Should this be part of this spec?
zcorpan: It's related.
gregwhitworth: The OM could point to this.
ojan: This would give a place for specs to point to.
Passing Data Between Custom Phases
----------------------------------
shane: I think we've had conversations about this.
Rossen: Offline?
shane: Yeah.
shane: Is there anything we need to talk about?
Rossen: I don't know, I wasn't there.
shane: Why don't we draw on the whiteboard and see.
Rossen: Perhaps re-hash the conversation?
shane: I wasn't there either.
shane: It seems like we're building out this picture of phases
starting from simple to complex property at the style
resolution level. We've got this picture of using custom
properties to pass data in and possibly some side channels.
shane: We haven't specifically talked about being able to generate
additional data and if you can make changes with custom
layout.
shane: Is this stuff we should talk about now or details to hash
out now?
smfr: Do we have use cases for passing custom data?
shane: I think at the style level we might. The compositor might.
But layout to paint there isn't any need yet.
smfr: And there are use cases for having layout information at
paint time?
shane: Yep. That's true.
smfr: It sounds to me that we're not far enough along. At certain
times we've talked about if we should store data from one
hook to another and maybe have hooks communicate somehow.
shane: We had proposed a caching API to store data between
iterations of the same hook. At lunch there was talk about
forming trees to pass data. I think this is very much brain
storming. There's nothing specific here. I think we should
move on.
Rossen: It's your topic. If you have nothing else we can move on.
Rossen: Next topic is 'some tasks need information from outside
the element.
rbyers: It sounds related to the previous topic
Vollick: We touched on it a bit with async.
shane: We've got this prioritization...I was going to suggest we
do the next topic and split to discussion groups.
Rossen: If we go through prioritization we can adjourn and people
will group.
<franremy> @Rossen: if we've some time, can I just get 2min of
time after current topic to add a new one (Main-Thread
Style Mutation Observer, similar to
CustomPaint/ApplyHook/... dependencies but executed on
the main thread?) and explain why I think it may be
useful
Received on Friday, 16 October 2015 23:45:52 UTC