W3C home > Mailing lists > Public > public-houdini@w3.org > October 2016

[Houdini] Minutes Lisbon F2F 2016-09-22 Part I: Layout API, Worklets

From: Dael Jackson <daelcss@gmail.com>
Date: Sun, 9 Oct 2016 20:39:07 -0400
Message-ID: <CADhPm3uXgSiMW48+UEDZj+OPzSNwdnUuLDQu3FWD2O=DfGUsgw@mail.gmail.com>
To: public-houdini@w3.org
Cc: www-style@w3.org
=================================================
   These are the official Houdini Task Force
     minutes. Unless you're correcting the
      minutes, please respond by starting
 a new thread with an appropriate subject line.
=================================================


LayoutAPI
---------

  - iank introduced the work he's done so far to write the spec and
      asked that the group define scope for the first version.
  - There was a concern that floats becoming unpositioned boxes that
      bubble up may have severe runtime consequences. A note will be
      added that this needs to be looked into further and perhaps
      moved to level 2.
  - There is still a desire to be able to walk the tree and access
      all related blocks using this API, but that work was moved to
      the WICG.
  - To create scope for v1, iank proposed using it to create a
      foundation and leaving harder items, like exclusions and
      scroll triggers out.
  - iank will create a draft of what he believes should be v1 and
      then make a prototype of the API, hopefully before the Seattle
      F2F.

Worklets
--------

  - The proposal to use frozen realms to get around the expensive
      "create and destroy at any time" language was feared to be too
      limiting.
  - SimonSapin asked for other approaches to the problem as it is
      important to avoid side effects.
  - RESOLVED: Paint/Layout specs will have normative section that
              implementations must have at least 2 globals with an
              opt-out for memory constraints.

===== FULL MINUTES BELOW ======

Agenda: https://github.com/w3c/css-houdini-drafts/wiki/TPAC-F2F-September-2016

Present:
  Jake Archibald
  Rossen Atanassov
  Tab Atkins
  David Baron
  Rik Cabanier
  Emil Eklund
  Elika Etemad
  Daniel Glazman (IRC only)
  Koji Ishii
  Dean Jackson
  Brian Kardell
  Ian Kilpatrick
  Peter Linss
  nainar
  François Remy
  Simon Sapin
  Shane Stevens
  Surma
  Aleks Totic
  Ojan Vafai
  Lea Verou
  Jet Villegas
  Greg Whitworth
  Boris Zbarsky

Scribe: eae

LayoutAPI
=========

  <iank> https://drafts.css-houdini.org/css-layout-api/
  iank: This is basically a very rough draft of the changes we
        discussed in San Francisco. Working on explainer doc but not
        quite ready yet. Can quickly talk about that but then want
        to define the scope. What's a good first set we can aim for.

  iank: This is the concept of the box, represents a css element,
        input into layout algorithm.
  iank: Text nodes, elements, etc are all boxes. Box might be a
        terrible name here.
  iank: In order to get glyphs rendering correctly we might need to
        collapse all inlines into one box which will then produce
        multiple fragments. If we didn't collapse all inlines then
        nice rendering (shaping across) wouldn't be possible.
  iank: Goal here is to be able to shape text across inline element
        boundaries.
  iank: If we have a float here *points to example with floating
        div* I think what we need to do here it needs to be returned
        as an un-positioned box that can then bubble up to the top.
  Rossen: In that case wouldn't you have quadratic runtime?
  Rossen: If for each float you would need to bubble all the way.
          Say if you had 5000 floating divs you don't know which one
          will pushed out of line.
  iank: Yeah, that is kind of nasty. Is there a better way to do
        this?
  iank: Perhaps we could push this to v2?
  Rossen: Another way to do this would be to... In this model it is
          kind of hard. We might be able to returned the positioned
          floats as a part of the constraint space.
  Rossen: Or, you could interrupt and let the platform take care of
          it and then continue from there.
  Rossen: As a v1 we probably do not care but we have to be mindful
          of this when we make those additions and when making
          conscious design decisions.
  dbaron: Why is the algorithm bad?
  dbaron: It looks to me that this is a representation of what you
          have before doing line breaking?
  Rossen: No, this is after.
  dbaron: This is the layout boxes part of the spec?
  dbaron: It says this is *before* line breaking.
  iank: Need to draw it out.
  [iank sketches a fragment placed within a window with text and a
        float]
  iank: You perform layout on a child which returns a fragment.
  iank: Float is positioned and then move on t next text node.
  Rossen: In this particular example float would go before text.
  iank: For some reason I thought it was after.
  iank: One way to do this would be to allow introspection into a
        box would allow floats to be exposed.
  dbaron: Floats are hard as position depends on line breaking.
  dbaron: If you want to do inline layout in a single pass you could
          do floats as part of line breaking but that would be hard.
  Rossen: We almost do it in a single pass.
  Rossen: We could record a note to revisit and then move on.
  iank: Wanted to exclude floats from v1 but have a basic idea of
        how to do it.

  iank: Continuing. For each box we'd have a single style map.
        Multiple styles for a single grouping of elements (text,
        other inlines). We might want a more compact style map. For
        a particular range have it include style, like color.
  plinss: I'd rather see it be separate boxes and have a different
          mapping.
  iank: Not quite sure yet.
  iank: One easy thing initially to introspection is to have
        individual CSS styles and a fragment could return a range
        within all of the non-atomic inlines.
  plinss: Would you expect the span to generate its own fragment?
  plinss: That precludes a lot of things like being able to walk the
          tree. Like say for tooltips.
  iank: What do you mean by tooltips?
  plinss: Like mouse over tooltips.
  iank: We might be able to expose all boxes associated with a
        fragment in v2, i.e. these are the boxes I was generated
        from.
  plinss: That would create another type of geometrical information
          that isn't really geometrical. We need clean mapping
          between boxes and fragments.
  iank: I think that should be underneath the box. Perhaps that is
        why box is a bad name.
  plinss: I think there is a lot of value in a one to one mapping
          between boxes and layout entities.
  iank: Inside the layout worklet you won't have access to the DOM.
  plinss: I don't want to create a concept of layout and boxes that
          is limited to the layout api.

  Rossen: This might be unrelated, how we make a relationship
          between element and boxes shouldn't it be a part of this
          spec.
  plinss: I want to be able to walk the layout tree and find all the
          boxes associated with it.
  plinss: We either need to completely different API (one for layout
          API and one for walking the tree) or we'd break the
          relationship.
  iank: I think it would be reasonable to expose the fragment tree
        on the main thread.
  iank: Exposing all boxes for a given element, like getClientRects
        does today.
  iank: In terms of the boxes; the box here is a box that has a one
        to many elements relation. Conceptually if we ever added a
        back-link from say a fragment could all point to the same
        element. A box cannot reference it's fragment.
  plinss: It's an abstract concept but it does define the API
          between DOM, Style and Layout.
  plinss: That whole view is a mismatch that makes it really
          complicated to do things.
  plinss: This smells like the wrong layer to do this.
  Rossen: Otherwise you have to bind everything to the main thread.
  Rossen: Exposing everything between DOM and Visual and having
          access at the same time implies...
  plinss: There is a logical mapping and a runtime mapping.
  Rossen: Regardless you'll need to synchronize across threads.
  Rossen: The associated might not necessarily be exposed in whole
          in each context.
  plinss: This is a model where we are building a layout mapping
          without direct access to either the dom or style tree. If
          we are going to expose a new view it is not going to match
          the view that the layout sees.
  plinss: Style information associated with dom and pseudo elements.
  plinss: Here we have boxes without a mapping to elements different
          from css2 boxes.

  <zcorpan> https://html.spec.whatwg.org/multipage/dom.html#the-innertext-idl-attribute
  zcorpan: This associated, this reminds me of inner text, which
           iterates over the DOM tree and gets each node's boxes to
           get the text. It's synchronous.
  zcorpan: Not interop between implementations, would it be useful
           to use this API to polyfill that feature? In that case we
           would need that associated between DOM and Boxes.
  Rossen: We're all circling around the same point. We need to go
          back to talk about the box tree.
  Rossen: We decided in Sydney to move that work to the WICG.
  dino: We need to kick it to the WICG and need to continue working
        on it.
  Rossen: Don't see anything contradictory.
  plinss: Box api might be very different for authors, with this
          model we cannot have a correlation between box tree api
          and fragments and boxes. Dom nodes do not have their own
          boxes and fragments yet have style.
  plinss: Not a one to one correlation.
  plinss: You are taking style nodes and giving them fragments but
          without style information.
  shane: There could be multiple runs of text, there cannot be a one
         to one mapping.
  plinss: Text nodes do not get their own style information.

  frremy: How about not doing inline for the moment? Most uses cases
          would be solved by a api limited to blocks.
  Rossen: Inlines are also blockified.
  dbaron: I'm not quite sure what you meant by that.
  dbaron: Not clear to me what the goals of this API are at this
          stage.
  dbaron: Block and inline layout are really quite complicated. I
          don't think we're going to have an API that allows you to
          correctly re-implement it in v1. Not sure that is what we
          should be aiming for right now.
  dbaron: One thing we could aim for is to re-implement layout for
          things that establish a block formatting context.
  dbaron: Next step, things that establish pieces of the BFC, like
          placing lines
  <SimonSapin> +1 dbaron
  iank: I imagine that in v1 we always establish a BFC.
  frremy: Then why can't we position floats before we get there?
  iank: So from that you could have a set of boxes underneath you
        that do not have any of the float behavior around pushing
        exclusions, limited to placing fragments. Enough to
        implement most complex line layout.
  iank: *shows example around complex inline layout*

  Rossen: What about for v1 we really dumb down the text layout
          part: You only do layout for continuous runs of text that
          are uniform.
  Rossen: Given a run of text, you get to decide where they go and
          something will happen back in the code path, perhaps a
          flow is added, a span margin/padding added, you get called
          back with the text. You the go figure out how to align the
          baselines.
  Rossen: Two APIS, 1) layout a bunch of text. 2) Here is a bunch of
          baselines, tell me how you want to align.
  Rossen: Takes floats, inlines, as far as text is concerned, out of
          their control.
  Rossen: In this case, we don't have to push the complexity of box
          and text layout to this API.
  iank: That sounds like something to aim for for v1.
  iank: My hope for v1 was that this section ("box generation for
        nonatomic inlines") would apply so that we can do glyph
        shaping across inline boundaries. Perhaps I should work on
        that section and come back to the group.
  Rossen: OK.
  iank: This gets into what we want for v1. I'd like to strip out
        the un-positioned boxes from the Fragment interface. Your
        layout would always establish a new BFC and engine would
        handle it.
  iank: In terms of baseline information, I think dominant baseline
        is enough for the moment
  iank: The main thing would be automatically handling abs pos and
        rel pos for v1.
  iank: Does that sounds reasonable?

  dbaron: I still don't think I fully understand where you are going
          with this?
  iank: Didn't have time to finish explainer that would explain
        this...
  dbaron: What sort of capabilities are you trying to provide? There
          are a bunch of general-ish things you could do. Like the
          un-positioned boxes could be a general thing, this is a
          layout that wants to pass boxes to its parent. Assumes
          that both child and parent are aware, both would be custom
          layout.
  dbaron: Given that you are not handling that - giving custom
          layout BFC isn't enough. Would need to give both the
          parent AND the child to establish a BFC.
  iank: Would like to allow inline layout but all block children
        should establish a block formatting context.
  dbaron: Feels like a very different thing to do inline layout from
          BFC.
  plinss: We should expose API that works like the css2 spec.
  gregwhitworth: We're trying to explain the platform, not the spec.
  plinss: The box tree api should match the spec description of the
          box tree.
  Rossen: Your top point was between text and blocks.
  dbaron: To place inlines you need baselines and font information
          and break points
  bz: and hyphen dictionaries.
  iank: Conceptually you have a fragment with a series of text runs
        and style information. In v1 you could place the line but
        not access the internal text runs within that line box/
        fragment.
  dbaron: Would this allow you to flow around a shape defined in JS?
  iank: Yes.
  dbaron: Sounds useful but sounds like an entirely different API
          from placing a BFC block.
  TabAtkins: If limited to lines it might not be that different,
             takes this much space and having a break token for the
             line breaking.
  florian: With this API you could do flow around a shape but you
           wouldn't be able to mutate the fragment to do custom
           hyphenation etc?
  iank: Correct, for v1.
  iank: We need to add the fundamental concepts now to allow for
        that to work in a v2 or later
  dbaron: Not completely convinced but after thinking about it it
          doesn't seem entirely unreasonable.
  dbaron: Inline layout in terms of architecture still seems very
          different.

  florian: Do you have an idea of what this should be able to
           achieve? If there is a list of use cases you could share.
  iank: Yes, on my to do list. Will share. Could do non-square
        exclusion; you pass in a shape and construct a constraint
        space that forces a specific alignment.
  iank: The hardest thing to implement would likely be to re-layout,
        back tracking, for inline layout.
  iank: Fragments now, very simple, have size, overflow size,
        breakToken. Much simpler.
  iank: Also wanted to simplify constraint spaces. In current spec
        as you can add a bunch of different exclusions. Would like
        input into how much to simplify them.
  iank: Limit to size, size for percentage resolution. We might be
        able to remove scroll triggers if we limit scroll overflow
        to the new non-auto type.
  TabAtkins: Having Houdini use the stable, non-auto, scroll
             overflow, would simplify things a lot.
  iank: Doesn't seem like something we need for v1.
  iank: Without exclusions it is much simpler. Same without scroll
        triggers. Last one is, if we allow it to fragment in block
        direction or not. Tempted to leave that in.

  florian: Could you get back to block fragmentation?
  [iank explains block direction fragmentation and break tokens and
      by drawing boxes on sketchpad. Break tokens indicate where the
      break happened to allow subsequent fragments to be generated
      and placed. Not a hard requirement for v1.]
  Rossen: Do you want to take it out?
  iank: Could go either way. We should be able to do it in chrome
        soon but might be harder for other engines.
  Rossen: Would be difficult to retrofit, if we don't have it in v1
          it'll be hard to add later.
  florian: Our use case would use fragmentation all the time.
  iank: We could revisit once we have implementations. Will leave in
        for now.
  iank: All of the exclusion stuff will also be left out from v1.
        Polyfill for exclusions possible by modifying constraint
        space. Would not allow piercing exclusions. Pretty
        reasonable v1.
  Rossen: Every level you would go back to your main thread where
          you could continue to adjust the constraint space so that
          should be sufficient.
  iank: Limiting the API as disused simplifies things quite a bit.
        Still have all the break token.
  iank: Will remove the childrenChane method for v1.
  Rossen: Sounds reasonable to me. Thank you Ian.

  iank: Final comments?
  koji: Looks great
  iank: We'll try to prototype in January. Might be able to have
        something by Seattle.

  [15 min break. Back by 10:35.]

Worklets
========

  SimonSapin: In the worklets draft there is a section on code
              idiosyncrasy. If we want to do custom layout on
              separate threads then we cannot depend on execution
              order. The risk being that if an implementation that
              doesn't have multiple threads then content might
              depend on there only being one thread.
  SimonSapin: I think the intention is good but it isn't strict
              enough. Would prefer must instead of "User agent *may*"
  shane: We were hoping that different user agents would choose
         different implementations.
  SimonSapin: "user agents may create and destroy context at any
              time". This may be expensive, perhaps we could use
              frozen realms instead. Would allow the global scope to
              have a frozen prototype. Would potentially be a lot
              cheaper.
  SimonSapin: Would be a way to ensure no share state between code
              without the overhead.
  iank: Could still be possible if you have code that looks like
        this:
        const arr = []'
        registerThing(''. class {
            method() {
                arr.push(1);
            }
        });
  shane: Frozen realms do not give a guarantee around Array
         prototype?
  esprehn: I don't think we want to use frozen realms, would prevent
           polyfilling. Would even prevent use of timers. Very big
           hammer mostly is a security feature.
  shane: We looked into other options but couldn't find any that
         worked.
  SimonSapin: The spec also says that the code is in multiple
              script, doesn't have a shared "this". Would it have
              access to the array prototype?
  shane: Another problem with frozen realms; you wouldn't be able to
         load libraries.

  SimonSapin: If frozen realms do not work, it is important to avoid
              side effects. Are there other solutions to prevent
              side effects.
  esprehn: I *think* we start and stop service workers when devtools
           is open.
  JakeA: We tried that but it caused debugging problems.
  esprehn: Our system is more restricted than service worker. We
           could document in devtools, every other call restarts the
           worklet unless you opt-out.
  shane: A summary here: user agents would, I hope, be very
         interested in building checks into their non-parallelized
         systems to ensure them to be parallel in the future.
  iank: We can talk to our devtools people, I think we could kill
        the worklet after each paint call or something.

  iank: Question around multiple global scopes. Would user agents
        killing the global scope be enough of a guarantee for you?
  SimonSapin: Why limited to devtools?
  iank: If we do it during normal runtime it would have a huge
        performance penalty.
  esprehn: Hopefully restarting in devtools and warning user would
           discourage behavior that depends on state
  shane: Would still be possible to depend on global scope if you
         never used devtools.
  SimonSapin: How about enforcing having at least two global scopes?
  esprehn: Memory overhead is a big deal, especially on mobile.
           Being forced to have two on desktop is likely fine,
           having two on a $15 phone is a different question.
  esprehn: Vague to allow low-end.
  shane: What if we make it SHOULD/MUST but with an out for low-end
         devices?
  iank: Only spec for ones where needed. Paint and layout. For audio
        one is likely fine.
  iank: For audio the global scope would be associated with the
        source.
  iank: Lifetime there more controlled than for paint/layout.
  dino: If you have one global scope per worklet on a $15 device
        then is two really that much worse?
  TabAtkins: Sharing is fine, want to avoid dependency on global
             state.
  esprehn: Hopefully being constraint and forcing the use of modules
           one would be less likely to get strange interactions.
  esprehn: To start, lets try having two and randomly pick one for
           each run.

  JakeA: Was this instead of devtools killing?
  shane: We might do both.
  ojan: Experimentation needed.
  iank: We would release to the wild and then starting toying with
        it to gather data and see what works.
  SimonSapin: Tab, you proposed having a MUST with an opt-out? How
              would you phrase it.
  TabAtkins: "You must do it unless you have memory constraints"
             perhaps?
  SimonSapin: Sounds much better than MAY.
  dbaron: There is also SHOULD.
  TabAtkins: That's just a strongly worded MAY.
  iank: I'll rework the spec, will still mostly be non-normative.
  shane: Should we have a resolution?
  Rossen: Any objections?
  * silence *

  RESOLVED: Paint/Layout specs will have normative section that
            implementations must have at least 2 globals with an
            opt-out for memory constraints.

  <SimonSapin> esprehn, was it you who said many libraries wouldn’t
               work in frozen realms? Could you expand on why, in
               https://github.com/w3c/css-houdini-drafts/issues/308 ?
  <esprehn> SimonSapin: sure
  <SimonSapin> thanks
Received on Monday, 10 October 2016 00:40:14 UTC

This archive was generated by hypermail 2.3.1 : Monday, 10 October 2016 00:40:16 UTC