W3C home > Mailing lists > Public > public-houdini@w3.org > February 2016

[Houdini] Minutes Sydney F2F 2016-01-31 Part III: Font Metrics, Composited Scrolling and Animation

From: Dael Jackson <daelcss@gmail.com>
Date: Sun, 21 Feb 2016 15:04:19 -0500
Message-ID: <CADhPm3ubWVWiDRwHPYjYASTGMFHHZTX1R5pgSB2FhP6KZbM8CQ@mail.gmail.com>
To: public-houdini@w3.org
Cc: www-style@w3.org
=======================================================
  These are the official Houdini Task Force minutes.

         Unless you're correcting the minutes,
        please respond by starting a new thread
           with an appropriate subject line.
=======================================================


Font Metrics
------------

  - Typographic layout may be added to the layout API and this spec
      will only contain metrics from the font and the style
      attributes. No matter the end decision, the spec will start at
      the smallest subset solving an important use case and grow
      from there.
  - The spec as written only provided the dominant baseline, but
      there were several different arguments that the first version
      of the spec should be able to provide more information,
      possibly the entire line grid. This would also require a
      method for the browser to say it doesn't know the information
      if a font hasn't provided that information.
  - There were four major topics that need to be addressed in this
      spec.
      1) What font is used (on what? first inflow glyph?)
      2) information that browsers use to align text on the page
          should be exposed, including the baselines.
      3) a low level API for raw (often wrong) font metrics.
      4) font geometry detection / font metrics synthesis.
      - SteveZ and astearns will work on the base line table
      - eae and dbaron will look into exposing why font is used
  - Inferred geometric data will be deferred.

Composited Scrolling and Animation
----------------------------------

  - There are five main use cases that have been mentioned for
      composited scrolling and animation:
      1) parallax
      2) scroll headers
      3) video sync
      4) linked scrollers
      5) drag and drop
      - The hope is that the group can decide together what are
          important and if this is the right place to solve them.
  - Google is currently working on implementing the approach listed
      in the spec. Most other browsers didn't respond, but Apple
      said they would likely follow a similar approach to Google's.
  - There was interest from Apple in solving as much of the use
      cases as possible using declarative script so that browsers
      can optimize.
      - This led to a wider conversation about how to ensure that
          items were ready for the fast path or at least telling
          authors if they had something that wouldn't go on the fast
          path.

===== FULL MINUTES BELOW ========

Agenda: https://github.com/w3c/css-houdini-drafts/wiki/Sydney-F2F-January-2016

NOTE: The group split into two concurrent sessions, one for font
       metrics, the other for composited scrolling and animation

Font Metrics
============
  Scribe: gregwhitworth

  astearns: When I originally brought up font metrics, I was
            thinking more text measurement.
  astearns: There are font metrics that we get from the fonts
            themselves and style attributes,
  astearns: then there are the typographic layout.
  astearns: Maybe typographic layout may just get added to the
            layout API.
  astearns: The font metrics may be desired by someone for custom
            position of the glyphs.
  florian: If you have a nested, text you may need stretching of the
           glyphs to wrap around the text.
  florian: For example math specific fonts have info for this.
  shane: That sounds like we need two different font APIs.
  florian: The author of mathJax really has a desire for this
           capability.
  astearns: There are two topics and they are very, very separate.
  astearns: Does anyone else have other topics?
  florian: Let's throw use cases as well.
  florian: For example, you would need the this info for first letter.
  astearns: It really depends. But if you try to do things only with
            text measurements, you end up doing layout to get the
            measurements and do layout again to achieve what you want.
  astearns: I think you always need the measurements, but you may
            not need the glyphs.

  dbaron: Do we want to expose different phases of the font process,
          such as shaping?
  astearns: We want to expose as little as possible and the smallest
            subset that solves an important use case.
  astearns: And keep adding to it, rather than exposing a large API.
  dbaron: You may want a measurement where you pass it a string of
          text.
  dbaron: You may also want a measurement of the text with certain
          styles.
  astearns: That's a little different than I was thinking of.
  astearns: There's also, just layout results.

  shane: Something to keep in mind is that the layout is expensive.
  eae: You could use measureText API to form canvas, you can get the
       baseline or baselines in some cases.
  astearns: You can, but it gets drawn into the canvas by different
            browsers.
  eae: Yes, it's very limited, but it's better than nothing,
  shane: It's like 50x faster.
  astearns: I want to avoid putting in divs just to find out layout
            information.
  astearns: I have two blocks that are laid out, and I want to align
            the second box with the first one's baseline, you're
            simply adjusting the info.
  shane: I think you should do specific layout into containment
         regions.
  eae: Right, you can give it an offscreen area and run this stuff.
  florian: I think it's cool and important to be able to trigger the
           API without triggering the rest of the pipeline if you
           don't care about it.

  astearns: I have done an API before, the current layout API shows
            just a baseline, there can be a glyph, a run, and each
            has a baseline and they can differ.
  astearns: But it has been carefully defined what you're talking
            about.
  astearns: The most popular one is the dominant baseline of the
            first character of the first run, and it was
            tremendously useful.
  astearns: Making it available at each step of the tree makes it so
            that you don't care about the glyph.
  astearns: Saying that a block has a dominant baseline glosses over
            a lot of complexity.
  florian: Yes. It's wrong, but it's useful.
  shane: Well it's not wrong.
  florian: Is it useful?
  stevez: Is that enough to do line-grid.
  astearns: No.
  stevez: That's what I thought.
  eae: You can also imagine segmenting on the different baselines.
  florian: You still need to know all of the baselines.
  florian: The line grid itself needs to have them all so that you
           can align to it.
  stevez: We could avoid needing to document to where we ask the
          interface for specific ones and if it doesn't know it will
          say it doesn't.
  astearns: Once you get the content rendered, the browser has
            determined the dominant baseline.
  stevez: One area where you need a baseline and top line are in
          international languages for top cap.
  shane: These lines are in the font?
  florian: No.
  florian: Some.
  stevez: You're lucky to get the alphabetic if anything.

  shane: I'm just curious if we'll ever be able to lift this up to
         JS, it may just need to be in the font engine.
  florian: That's my issue with this discussion, while I don't want
           to lift this to JS, but when I look at the use cases we
           need this information.
  shane: I think it's ok that if something isn't there in the font,
         it's ok for us to say we don't have the information.
  stevez: When fantasai, dauwhe, and I discussed the for CSS inline
          the best to generate values that are missing from the font
          information.
  florian: Tying it to your point *points at shane* it should be up
           to the UAs to be able to put their hands up.
  shane: I think you should be talking to the font engines.

  iank: My only perspective is that if this doesn't force layout
        into a corner I'm OK with it.
  shane: We should have an API that provides a null response if we
         don't know, but when you want a response we provide the
         dominant baseline
  astearns: There could be a fallback.
  Bert: Let's say you want to know the cap height, does it say
        something?
  florian: No, you're forcing the browser to lie.
  stevez: What if the browser is using the cap height?
  florian: Then the browser should tell us that.
  stevez: Some of the Adobe products create font information for
          cross platform interoperability.
  [gregwhitworth says something about font fallback]
  astearns: And that's a different need where we need to know about
            which fonts are used on a given range.

  stevez: Could we live with just getting a fixed number of things
          in a baseline table (alphabetic, ideographic, etc)?
  astearns: But those are things that the browser DOES synthesize.
  florian: Do you envision asking the element foo for it's baseline
           and it gives you if it knows it.
  stevez: Different. If you implement L1, you get the basic ones.
  shane: Your adobe tool, does it run in real time or does it cache
         it?
  stevez: I don't know really, I think it may be both,
  astearns: Unfortunately there are a lot of typographic engines in
            adobe products.
  <eae> In summary, my point was that if the browser needs to be
        able synthesize a baseline for other reasons that
        synthesized baseline should be provided as a part of the API.
  shane: We should possibly try and standardize the heuristics that
         Adobe uses in these tools if possible.
  shane: Maybe we provide devs with the opportunity to choose which
         path they want, cross platform or matching the system.
  eae: But you have to provide all of the paths to correctly match.
  stevez: This will be very hard on Apple as they don't always
          provide the correct info.

  astearns: I think we can simplify even further for V1, we just
            provide that first dominant baseline and provide the
            table in future versions
  stevez: I wanted to start off with the table so that people start
          thinking in that direction rather than coming in with a
          new interface.
  astearns: Yeah, ok.
  eae: *shakes head up and down* I think that's a really good idea.
  astearns: You would be able to determine which baseline is the
            dominant one?
  florian: Yes, I think you should.
  stevez: Ideographic characters are positioned in one of two ways,
          either at the bottom of the em box or at the center.
  stevez: Those are both used, so if you give the em box position in
          the table then that would be sufficient as well.
  florian: Then with math finding the center if you have the top and
           bottom it would be easy.
  koji: Vertical is always center, horizontal is either bottom or
        center.
  stevez: You're going to miss cap height on any font that isn't
          western.
  astearns: You'll have ascent and descent but they aren't helpful.
  florain: And they'll all be buggy I suppose.
  astearns: Yes.
  stevez: Cap height is not reliably available.
  <liam> [ cap height ascender/descender height and x-height where
         available are incredibly useful]

  astearns: Right now the text measurements are hanging off the
            fragments, I suggest rolling this up to the element and
            it will be the first inflow fragment.
  stevez: But the element may not have gone through layout.
  shane: Yes we have hit issues with conflating the left of the line
         and we should ensure that this doesn't happen.
  astearns: Yes we need to determine if we should allow custom
            layout stuff to potentially allow layout to be invoked
            since ergonomically it's what devs are used to.
  shane: There are strong opinions within the Chrome team against
         doing that.
  shane: I think going down that path is bad for the web
  stevez: So then we agree it's on the right of the line (of
          Rossen's process model design).
  stevez: When he says the first inflow run, the drop cap will be
          wrong since it's out of flow.
  stevez: If the baseline is taken from the first inflow glyph, then
          floats and drop caps are not inflow.
  shane: So it needs to be in text, not inflow.
  stevez: What's text?
  eae: It's special.
  stevez: We're changing the interface to have alphabetic, top,
          bottom and parameters for which is dominant.
  florian: Can it happen that the dominant baseline is not one of
           these? It should be yes.
  stevez: For practical purposes, is no.
  stevez: Good point,
  stevez: you could define ideographic center.
  eae: Or we could just provide the position of the dominant
       baseline rather than the name.
  stevez: That allows us to add additional baselines without
          changing the interface.
  heycam: Are we talking about glyph data?
  astearns: The input of the metrics or the result of layout?
  heycam: The actual font glyph info.

  Scribe: Florian

  shane: Do the type of raw metrics diverge between font types?
  all: Yes.
  dbaron: Font by font even.
  SteveZ: The tools people use sometimes plug in values without
          justification, and you may be better off when the tools
          remained silent.
  dbaron: All fonts form a foundry may have the same data,
          regardless of the font.
  Bert: Can I find out where the ink is?
  SteveZ, astearns: There is bounding box data, it sometimes is wrong.

  Florian: How comes anything works if all font info is wrong?
  dbaron: Hard coded font specific heuristics.
  astearns: Several libraries do manage to extra useful information
            and do things with them, so we are not wasting our time.
  astearns: We should expose the raw font data, knowing it has
            issues, and let JS deal with it,
  astearns: but we have to expose the actual font being used.
  <liam> [when the information gets used, over time, some of the
         errors get corrected, people fix their fonts]
  shane: There are 3 categories of things we can do
  shane: 1) What font is used (on what? first inflow glyph?)
  eae: What if you use multiple fonts for a single glyph?
  eae: We have APIs exposing the list of fonts used in a list.
  heycam: How do you expose this?
  astearns: Some object that you can use to get metrics from.
  dbaron: An object that represents a face.
  gregwhitworth: How do you untangle things when all fonts are used
                 together?
  shane: Don't dive into the details to quickly.
  shane: 2) information that browsers use to align text on the page
         should be exposed, including the baselines.
  shane: 3) a low level API for raw (often wrong) font metrics.
  shane: 4) font geometry detection / font metrics synthesis.
  shane: We should identify a leader per area.
  SteveZ: We need to get something written down at the level just
          described.
  [SteveZ + astearns will do base line table]
  [eae + dbaron will look into exposing why font is used]

  Florian: Do we need to work on raw font metrics in level 1?
  shane: While exposing which font is used, we should also include a
         way to get the raw font file, so that people who want to
         parse it themselves can get to the data easily.
  <liam> ["get the data easily" easier said than done e.g. for a
         graphite font!]
  astearns: There is a risk of fingerprinting, but the cat is out of
            the box already.
  dbaron: The only way to close this is for browsers to ship their
          own fonts and not use OS fonts.
  dbaron: We shouldn't expose raw font files for local fonts, only
          for webfonts.
  shane: For inferred geometric data, we should defer.
  all: Agree.
  eae: There's a 5th area: line break opportunities?
  florian: That's useful, but separate topic?
  dbaron, astearns: I agree, it's a separate topic.
  shane: And I'll write the high level explainer

Composited Scrolling and Animation
==================================
  Scribe: vollick

  rbyers: We're hoping to avoid API debates and focus on use cases
          and discussing other approaches.
  rbyers: We'd also like to know about constraints of other
          implementations.
  rbyers: As well as knowing what data would be helpful for us to
          gather.
  rbyers: Ian and I have been scrambling to get together a list of
          use cases in the repo. Would be great to collaborate on
          this list.
  rbyers: Anyone else have suggestions on how to make this time
          productive?

  rbyers: We've been thinking about this as two separate things. 1)
          compositing and asynchrony. 2) scrolling extensibility.
  rbyers: We'll start with how to plug into compositing and access
          to that thread.
  rbyers: surma was going to present a demo that he's built to start.
  surma: This is slightly distorted, but you'll get the gist.
  surma: This is a test build that has a version of CompositorWorker.
  surma: I've hooked into scroll position to transition a header.
  surma: This permits a very high frame rate effect.
  surma: The response on twitter has been enthusiastic.
  surma: This is probably the proposal that would get quickest
         adoption.
  surma: I will try to share the demo as soon as I can.
  rossen: Can you explain how this was built?
  surma: Currently I could only use pixels for transform presently,
         but this is bald html/css with JS that hooks to scroll
         position.
  jet: Is it the fact that you're just talking in pixels what makes
       it fast?
  surma: I don't think so, it's that it happens on the thread. Could
         probably handle other units in the future.
  <surma> Here’s a video of what I just demo’d:
https://www.youtube.com/watch?v=EUlIxr8mk7s

  Scribe: rbyers

  vollick: Another demo: it's a toy,
  vollick: but it's a physics simulation happening in the compositor
           worker.
  <rbyers> https://github.com/w3c/css-houdini-drafts/blob/master/composited-scrolling-and-animation/UseCases.md
  vollick: Jank that's happening on the main thread isn't affecting
           the physics sim / animation.
  vollick: We've been working on an implementation of compositor
           worker in the hopes that we can ship position:sticky and
           snap points in terms of it.
  vollick: Largely so we can learn about the performance
           implications. We'll share that once we've got it.
  vollick: The initial compositor worker code is almost entirely
           landed in chromium trunk now.

  vollick: Let's walk through example use cases for composited
           scrolling and animation.
  vollick: 1) parallax
  vollick: Maybe not very interesting since you can do it with 3d
           perspective.
  dino: Really? seems pretty interesting - may want to do different
        sorts of math
  smfr: I'd say compositor worker is the natural way to do parallax.
  dino: Seems like one of the most important use cases
  vollick: 2) scroll headers
  vollick: 3) video sync
  vollick: This is an example brought up by dino - video position
           coupled to scroll position
  vollick: 4) linked scrollers
  vollick: 5) drag and drop
  vollick: Want elements that stick to your finger, regardless of
           what's happening on the page.
  rbyers: To step back for a second, these use cases are not just
          for CW - we'd like to collectively figure out which ones
          are important and brainstorm other solutions.
  rbyers:  e.g. dino's "animation time bases" proposal solves some
           of these.
  dino: Actually, it doesn't "solve" them - just enables a couple
        specific scenarios.
  vollick: It covers other use cases - drawers, performant effects
           libraries, gesture recognition, element location tracking
  vollick: [scribe fails to keep up] other use cases
  vollick: any objections
  [rbyers and smfr agree that we can leave raw input to v2 of the
      API since there's complexity to explain hit testing and other
      things]

  Scribe: Rossen

  jet: Is the idea of compositor worker to be able to keep up with
       the fast path only? if we have getComputedStyle we'd fall out?
  vollick: This is certainly the hope
  vollick: [showing examples of scrollmagic.io]
  <vollick> https://github.com/w3c/css-houdini-drafts/blob/master/composited-scrolling-and-animation/Explainer.md
  vollick: Some of the examples should already work.
  rbyers: Should we move the input example to a different place?
  [discussion of how exactly the input example works]
  vollick: The inputs to the callback are traits and not properties.

  dino: If a compositor worker is killed, what happens?
  vollick: It won't work, assumption is that it will not.
  vollick: I'm hoping that we'll be able to use a worklet for this.
  rbyers: An important question/design goal is to be
          suspendable/resumable

  smfr: How could you do collision detection?
  vollick: It's tricky.
  smfr: From a given compositor worker can you access multiple
        proxies?
  vollick: Yes.
  dino: You can be getting more post messages?
  rbyers: Yes but it'll depend on what we use.
  vollick: Another idea - being able to notify the user they're
           behind.
  dino: raf and worker runs after the main thread?
  vollick: We do best effort in the same frame.
  dino: Changing a background color based on manipulation?
  rbyers: Yes, this is a good used case we should consider.
  rbyers: We don't want this to be the new feature that introduces
          even more jank.
  vollick: That's about it.
  dino: about what exactly?

  Scribe: esprehn

  Rossen: What do other vendors think?
  dino: Google is moving ahead, what do other browser vendors feel
        about this?
  dino: We might standardize something very close to Google because
        we don't have a prototype yet.
  rbyers: I'd like to get to the point of figuring out what's next,
          ideally next time we have it behind a flag so developers
          can use an API key with the experiment framework to try
          this out.
  rbyers: At that time I think we want to really make sure we get a
          solid spec that's not just impl details.

  philipwalton: Is there a declarative version of this?
  rbyers: dino's proposal is as close
  rbyers: Actually lots of CSS features are close to this like snap
          points, etc. could spend 10 years making all the various
          declarative things.

  philipwalton: What about cases like the width of the progress bar
                at the top that's based on the scroll position of
                something on the page?
  rbyers: No spec for that yet, there's animation time base which is
          in the same space.
  Rossen: You're asking if we can make the scroll offset into a
          value type in CSS.
  philipwalton: Yes, in CSS set a function of the scroll offset of
                some element; width: scroll(#id)
  dino: Yeah we have use cases, not actually the scroll top, but
        it's a special calc like scroll height - some other things,
        need to invent more new features for this.
  dino: For simple parallax there's probably a way to create a
        declarative syntax.
  smfr: Other way is like web animations where you setup a thing on
        the main thread and run in async.
  rbyers: Setup a graph data structure.
  smfr: Don't really need script to just do .8 * scrollTop.

  rbyers: I don't want us to be afraid of running a tiny bit of
          script every frame.
  smfr: Should be afraid of adding scripting hooks where authors can
        make mistakes.
  vollick: My hope is we can come up with a shape that alleviates
           some of these mistakes.
  smfr: No, difference is script vs declarative is the UA can
        optimize the declarative approach,
  smfr: script based approach you can't optimize (the algorithm).
  smfr: Only mitigation is put it back on the main thread when it's
        too slow.
  smfr: I'm not totally against this, I would just prefer a
        declarative approach.
  vollick: I know you haven't had time yet, but it'd be great if we
           could look at the use cases with the declarative approach.
  rbyers: I have a hard time imagining how this would work for all
          the use cases with the declarative graph language.
  rbyers: Like Google did where cards spread out as you scroll.
  dino: Yeah, Apple does that for messages.
  rbyers: If we can come up with how to redo this with some
          declarative language I'm okay, but it seems hard.
  rbyers: It seems developers may not have enough tools.
  smfr: But developers would probably tell us when things were
        missing.
  dino: It is limiting, but maybe not as much as you think if we get
        enough cases.

  vollick: It would be nice if there wasn't a foot gun to avoid
           developers fall into the slow path such as when animating
           the width.
  ojan: Yeah, it's bad to only have the declarative approach because
        you end up trying to solve all use cases.
  ojan: For example flex box has n^2 trying to solve all the cases.
  smfr: You don't need to solve all cases.
  dino: Yeah.
  ojan: That's really hard.
  ojan: I would have both in an ideal world, encourage declarative
        approach since we can optimize, but have fallback.
  rbyers: It's the same argument for custom layout.
  rbyers: Developers will prefer to use built in CSS things, and
          build things that are popular in custom layout libs back
          into the browser
  rbyers: but custom layout allows innovation.
  ojan: I'm okay doing declarative but I'm worried if we only do that.
  ojan: It's a mixed blessing.
  ojan: You can find cases where it's terrible and cases where it's
        great.
  rbyers: What if we said you can't position objects with JS, you
          must use CSS?
  rbyers: I don't think we would want to do that.

  ojan: I'm curious about dino's question about what is Microsoft
        and Mozilla sentiment.
  Rossen: What thing?
  rbyers: The concept of giving developers a thing that's typically
          synced with scrolling.
  Rossen: It's pretty crazy, on our side we're trying to do all
          kinds of compositor optimizations using the lower level
          parts of the system which are fairly restrictive, even for
          us, to know at any given time where things are.
  Rossen: And if at any given time we have to go look at script to
          find what to do that's going to throw away a lot of
          optimizations.
  Rossen: I'm not sure it's even a question of declarative vs
          imperative.
  rbyers: You mean both fight with the lower level optimizations?
  Rossen: Yes, so far we've found most ways to work around the
          declarative things since ahead of time you can categorize
          and figure out what to do later.
  Rossen: If you don't know until you get on the right side you
          can't optimize.
  Rossen: Even if all your script callback does is simple like * 2.
  smfr: You can limit what's allowed in the declarative approach to
        allow keep being efficient.
  vollick: It's the same as having a limited set of animatable
           properties in core animation.
  jet: Mozilla is working hard to make things that fall off the fast
       path fast, like animating left.
  jet: They would not want to be painted into a corner, saying
       things like you can't put left on the compositor.
  vollick: This design allows UAs to expose new properties.
  vollick: For example you can ask if a proxy supports a property.
  rbyers: We wouldn't support properties we can't make fast.
  rbyers: So the concern that servo still stands, since devs won't
          use a feature which is slow/doesn't work in all browsers
          that isn't servo.

  dbaron: I think I'm reasonably supportive of the general idea
          here, but it's sort of one of the things where we're
          freezing today's architecture into a standard more than
          usual.
  dbaron: Is this still useful in 10 years?
  ojan: I share that concern.
  dbaron: Is the compositor thread a thing in 10 years?
  ojan: I agree, that's why we switch to worklets.
  rbyers: The UA could also choose to do it on the main thread in
          this design (the one with postMessage + compositor proxy).
  vollick: We could run this on the same thread, in that world this
           is a win because it limits what the code can do.
  vollick: I think the primary issue is if something we put in this
           subset is slow later, which seems rare, do you think so?
  smfr: I agree.
  ojan: It depends what your subset is.
  dbaron: The case I can think of is inverted color drawing.

  rbyers: Do we move to scroll customization?
  dino: No, lets break now.

  [break]
Received on Sunday, 21 February 2016 20:05:20 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 19:53:26 UTC