Re: Reasoning behind srcN replacing srcset and <picture>

For those of you reading in plain text, the missing images (which help
understanding) are:

Graphic: How image sizes change as the number of columns changes in a
responsive grid
http://i.imgur.com/50qFT2r.png

Graph: Relationship between viewport width and image width
http://i.imgur.com/56O1lrJ.png


On Sun, Sep 29, 2013 at 12:42 AM, John Mellor <johnme@chromium.org> wrote:

> Hi folks,
>
> By now most of you will have seen and started discussing<http://lists.w3.org/Archives/Public/public-respimg/2013Sep/0064.html>the alternative
> markup proposal <http://tabatkins.github.io/specs/respimg/Overview.html> Tab
> and I came up with, intended to replace both srcset and <picture>.
>
> Tab's spec write-up gives great examples of how this solves the various
> use cases; but I think we owe you an explanation of why srcset, <picture>,
> or some other simpler approach can't solve the same use cases. Let me try
> to explain, and also preemptively answer some questions.
>
> You're all familiar with the 3 main use cases<http://usecases.responsiveimages.org/> for
> responsive images: devicePixelRatio based resolution switching, viewport
> size based resolution switching, and art direction.
>
> The key difference between viewport-based switching and art direction is
> that in the viewport case a bandwidth-constrained browser should be allowed
> to download a lower resolution version, whereas in the art direction case
> it's essential that the browser strictly obey the switch, as the images may
> have different aspect ratios, and downloading the wrong one could break the
> layout.
>
> The simple srcset syntax that WebKit recently implemented handles
> dpr-based switching, but neither of the other cases. The extended srcset
> syntax with 'w' and 'h' attempts to handle viewport-based switching, but
> makes no attempt to solve art direction - and in fact, it doesn't
> successfully solve viewport-based switching either, for rather subtle
> reasons. Consider a page with a simple 1-3 column responsive grid:
>
> .grid-column { width: 100%; }
> @media (min-width: 640px) {
>     .grid-column { width: 50%; }
> }
> @media (min-width: 960px) {
>     .grid-column { width: 33%; }
> }
>
>
> If this grid contains images that are 100% of the width of their column,
> then smaller viewport sizes sometimes lead to larger images than large
> viewport sizes. Paul Robert Lloyd has a great illustration of this in his 2012
> post <http://24ways.org/2012/responsive-images-what-we-thought-we-needed/>
> :
> [image: Inline image 2]
> I plotted the required image width at a range of viewport widths:
> [image: Inline image 3]
> Notice how the widest image is required at a 639px wide viewport, and on
> wider devices smaller images are used. This is why srcset isn't good enough
> for viewport-based switching. If a bandwidth-constrained browser chooses to
> download an image destined for narrower viewports, it might actually end up
> downloading a larger image file!
>
> So what about the <picture> element? <picture> handles dpr-based switching
> and art direction well, but isn't suitable for viewport-based switching, as
> the browser isn't allowed to download images destined for smaller screens,
> even if they are in fact just different resolutions of the same image,
> since it has no way to know this. You can sort of support viewport-based
> switching if you create a lot of artificial breakpoints at different
> viewport widths, and provide a 0.5x, 1x, 2x and 3x image (the 0.5x image
> being for bandwidth constrained devices) at each; but that generally means
> you have to repeat every image url 4+ times, which simply isn't practical.
> In addition, working out what breakpoints to use for viewport-based
> switching is very laborious and confusing. For the responsive grid case
> above, you'd have to encode it as something like:
>
> <picture>
>     <!-- This source element covers the range 320-400, so the midpoint is
> 360, and that's what I'm using to calculate the x values, e.g.
> 320/360=0.89; similarly below -->
>     <source media="(max-width: 400px)" srcset="160.jpg 0.44x, 320.jpg
> 0.89x, 480.jpg 1.33x, 640.jpg 1.78x, 960.jpg 2.67x">
>     <source media="(max-width: 520px)" srcset="160.jpg 0.35x, 320.jpg
> 0.7x, 480.jpg 1.04x, 640.jpg 1.39x, 960.jpg 2.09x, 1280.jpg 2.78x">
>     <source media="(max-width: 639px)" srcset="320.jpg 0.55x, 480.jpg
> 0.83x, 640.jpg 1.1x, 960.jpg 1.66x, 1280 2.2x, 1920 3.31x">
>     <!-- Note that in the next 2 source elements, images are only half the
> viewport width -->
>     <source media="(max-width: 800px)" srcset="160.jpg 0.44x, 320.jpg
> 0.89x, 480.jpg 1.33x, 640.jpg 1.78x, 960.jpg 2.67x">
>     <source media="(max-width: 959px)" srcset="160.jpg 0.36x, 320.jpg
> 0.73x, 480.jpg 1.09x, 640.jpg 1.45x, 960.jpg 2.18x, 1280.jpg 2.91x">
>     <!-- And in the next 4, images are only 1/3 of the viewport width -->
>     <source media="(max-width: 1200px)" srcset="160.jpg 0.44x, 320.jpg
> 0.89x, 480.jpg 1.33x, 640.jpg 1.78x, 960.jpg 2.67x">
>     <source media="(max-width: 1440px)" srcset="160.jpg 0.36x, 320.jpg
> 0.73x, 480.jpg 1.09x, 640.jpg 1.45x, 960.jpg 2.18x, 1280.jpg 2.91x">
>     <source media="(max-width: 1920px)" srcset="320.jpg 0.57x, 480.jpg
> 0.86x, 640.jpg 1.14x, 960.jpg 1.71x, 1280 2.29x, 1920 3.43x">
>     <!-- This covers the range 1920-2560, but I left off the upper bound
> so any truly huge monitors get sensible behaviour -->
>     <source srcset="320.jpg 0.43x, 480.jpg 0.64x, 640.jpg 0.86x, 960.jpg
> 1.29x, 1280 1.71x, 1920 2.57x">
> </picture>
>
>
> That's pretty hard to read, let alone write. Notice how each image url
> gets repeated up to 9 times here! And in order to reuse the same set of
> images at the various breakpoints, I had to get out a calculator to work
> out all the x multipliers :-( It could be made slightly simpler if you're
> willing to make larger jumps in viewport width, but that has the downside
> that users will download more unnecessary data. Ultimately, the problem is
> that the web developer needs to encode that graph I plotted above into the
> <picture>, but they are forced to approximate the lines by plotting a
> series of points, whereas it would be much easier if there was a "straight
> line" primitive.
>
> Instead with the srcN proposal, this becomes way simpler:
>
> <img src1="100% 640 50% 960 33%; xxs.jpg 160, xs.jpg 320, s.jpg 480, m.jpg
> 640, l.jpg 960, xl.jpg 1280, xxl.jpg 1920">
>
>
> That first part might look a little bit cryptic at first, but if you
> compare it to the breakpoints in the grid media queries above you'll see
> how it's actually a pretty intuitive way of expressing the column widths
> above. Remember that this is for the tricky case of a responsive grid; for
> simpler cases like a full width header image you just put "100%" (i.e. the
> graph is a single straight line), whereas with srcset or <picture> you'd
> still have been stuck giving 7+ points along the line.
>
> And then you simply list the pixel widths of the available images, which
> is much easier than having to do a bunch of maths to link each image to a
> variety of viewport widths and devicePixelRatios; crucially you never have
> to repeat each image!
>
> srcN also covers art direction by allowing you to select which set of
> images you want using <picture>-style media queries, but these should be
> used only for art direction, not for viewport switching (which is covered
> by the mechanism above).
>
> *FAQ*
>
> Q) This seems to focus on flexible-width images (e.g. width: 100%). What
> about fixed-width images?
>
> A) For fixed-width images, srcN lets you use the familiar "s.jpg 1x, l.jpg
> 2x" syntax that srcset had, as for fixed-width images that works fine. If
> you prefer, you can also use the new syntax with a fixed width instead of a
> percentage width, in the form "320; s.jpg 320, l.jpg 640", but that's more
> verbose, so it only really makes sense to use that in cases where an image
> is sometimes flexible and sometimes fixed width, for example if you have a
> max-width in px set on an image whose width is set in %.
>
> Q) Does this fully obsolete srcset and <picture>?
>
> A) Yes, this completely replaces all functionality of both of those.
> Ideally, browsers that have already implemented the basic srcset syntax
> would remove that as soon as possible before sites start to depend on it
> (but if not it would be possible for such browsers to integrate srcset<http://tabatkins.github.io/specs/respimg/Overview.html#syntax>,
> just like src acts as a fallback).
>
> Q) Why src1, src2, etc instead of child source elements like <picture> has?
>
> A) Some implementors complained that having child elements is one of the
> things they regret about <video>, because it bloats the DOM tree and adds
> more edge cases to parsing. But this is just a detail; the same syntax
> could be used in a <picture> with child elements if implementors change
> their mind about that.
>
> Q) But my image is neither fixed width nor a percentage of viewport
> width...
>
> A) Yep, this is one thing that's not quite perfect. Say you have a fixed
> 400px sidebar, then two more columns split the remaining viewport width
> between them. Those two columns will be calc(50vw - 200px) wide, but the
> <image-size> grammar only lets you specify a single integer or percentage.
> It would be possible to approximate this width with a series of
> breakpoints, e.g. "0 400px 10% 600px 23% 900px 33% 1400px 39% 2100px 41%"
> (at least only the first part of your srcN would be messy, and you wouldn't
> have to repeat any image urls). But perhaps we should extend srcN to handle
> this case better, by extending the image-size grammar as follows:
>
> <image-size> = <integer> | <percentage> | <percentage> + <integer> |
> <percentage> - <integer>
>
>
> Then you'd be able to express this case more cleanly as: "50% - 200".
>
> Q) What about CSS images?
>
> A) Yes, the same problems that apply to <picture> also apply to CSS image
> switching using media queries and image-set. So we should extend the CSS
> image-set function to support the equivalent "<size-viewport-list> ;
> <size-based-urls>" syntax, then CSS and HTML will be consistent.
>
> There's also the more radical option, that we could directly allow any CSS
> image value in srcN (you'd use image-set, enhanced as above, to do
> responsive images). So amongst other things you'd be able to do image
> slicing (which wasn't previously practical in HTML) using fragment
> identifiers <http://dev.w3.org/csswg/css-images/#image-fragment>. This
> could be pretty cool, but it would make the responsive images syntax
> slightly more awkward (e.g. src1="image-set(100%; 's.jpg' 320, 'm.jpg'
> 640)" instead of just src1="100%; s.jpg 320, m.jpg 640").
>
> Q) Why a markup solution? Wouldn't it be better to switch images
> server-side (or intercept them with a Service Worker)?
>
> A) Either is good. But the key observation, that the relation between
> viewport width and image width needs to be known, applies to server-side
> solutions too. Also, one benefit of markup-based solutions is that the
> browser (which has access to user preferences, and current network
> conditions) can decide whether and when to download lower quality versions
> of images; whereas a server can't easily second-guess the user's preference
> (which may depend on whether the user is roaming, etc).
>
> Once we have HTTP/2, I believe the long term solution to responsive images
> will be browser-driven parallel download of progressive images, where you
> just include a single high res image in your markup, and the browser stops
> downloading the image once it has enough detail. I owe you folks a blog
> post on that. But full rollout of HTTP/2 will take a very long time, and in
> the meantime this markup solution seems to best fit all the main use cases.
>
> Thanks for reading this far!
> -- John
>