RE: CfC: to publish "The picture element" specification as a First Public Working Draft (FPWD)

> From: mat@matmarquis.com
> Date: Sun, 3 Feb 2013 18:48:24 -0500
...

> Your objection was to the `picture` element, but you went on to describe `srcset`; I think it’s possible the two are being conflated here. `srcset` doesn’t make use of media queries at all.

My concern has always been with the support for resolution
choice by the UA which is in the srcset proposal.   I have no
objection to the 'picture' element in isolation and it seems to
handle the art-direction use case - the 'picture' proposal does
include the srcset attribute on the <source> element.  Other
significant players have objected to the verbosity of the 'picture'
proposal (WHATWG), but it's not my concern.

The srcset syntax does appear to be largely 'media query' based.
The width and height refer to the 'viewport' media feature.  The
pixel density is 'user-agent-defined' so does differs from media
queries but I presume it will typically be the 'resolution' media
feature.  The srcset algorithm does allow the UA to make a
choice which contrasts with media queries.  Have I missed
a distinction?

Contrast the proposed srcset syntax with:
srcset="1.jpg 1x, 2.jpg 2x" where '1x' and '2x'
where '1x' and '2x' are the image resolution multipliers and would
allow the UA to infer the resolution of 2.jpg given it knows the size
of 1.jpg.

Or another syntax example:
srcset="200x100.jpg 200x100, 400x200.jpg 400x200"
where '200x100' and 400x200' define the image sizes.

> In the simplest case: a single resolution “toggle” on  an `img` element, that syntax would look like this:
> `<img src="sd.jpg" srcset="fallback-hd.jpg 2x">`> 
> The `2x` is both the qualifying “hint,” and the “this is a high resolution image” flag for the UA—if the client’s browsing environment qualifies for the HD option at all, the UA then determines whether it should be requested based on factors like user preference, bandwidth, and so on. If the UA determines that the high resolution image shouldn’t be requested, the image’s `src` is requested as usual.

Unfortunately '2x' does not appear to define that the image is a high
resolution image or that it has twice the image pixel resolution.  The
viewport width and height parameters also have a weak relationship
to the actual image resolution.  For example, in the following example
the pixel density hints are the same for all images yet they have
different resolutions:
<img srcset="imgx1.jpg 1x 480w, imgx1.5.jpg 1x 800w, imgx2.jpg 1x 1920w">

This also raises the question of what is the intrinsic size of the image
for layout purposes?    Surely the intrinsic image size must be constant
and independent of the image chosen otherwise the layout could change
depending on the image choice which contradict the espoused goal of
srcset being only a resolution choice.

> I’m afraid I don’t fully understand your objection, which is likely my own fault.
> As authors, “hints” are the best we can provide for this sort of thing: “in an ideal
> world, request source X. If bandwidth is limited by connection or preference,
> don’t.” Leaving the bandwidth-throttling decision entirely in the hands of authors
> will lead to an inconsistent experience for users at best, and complete omission
> at worst. Again, these concerns are detailed in Yoav’s post better than I could
> hope to explain them: http://mobile.smashingmagazine.com/2013/01/09/bandwidth-media-queries-we-dont-need-em/ 
> If I’m misinterpreting you somehow, I’d be interested in seeing a (pseudocode)
> example demonstrating what you mean.
I agree with not trying to handle bandwidth choice through author hints etc.

My concern it how the UA can make a choice given the proposed srcset syntax?
I am concerned that the srcset syntax does not define the image resolutions
so the UA has little information to work with.  The density and viewport
parameters in the srcset proposal are only pre-layout hints.   I believe the image
sizes would be needed for the UA to choose any appropriate image to download
to meet technical goals such as a rendering a sharp image with smallest available
image. 

Perhaps there are two distinct cases:

1. Before the UA has computed the layout it can not be sure what the image box
size will be and the 'hint' seems to be the proposed solution.  Sorry, the full
implications of your proposal are not immediately clear to me but I am reviewing it.
If you have some example choice strategies in mind then sharing them would
be very helpful.

2. After the UA has computed the layout it knows the target box sizes of images
and could use knowledge of the resolution of each available image to choose
the smallest image to give sharp presentation.  It might also be useful to support
author hints to define alternative choice strategies - for example to suggest the
UA choose a lower resolution image but give the user a choice to sharpen etc.
This case also handles layout changes after the page has loaded, such as an
orientation change.


Here's a suggestion on how consensus might be more easily reached.  Split the
task into: 1. defining the required information and the reasons the
information is needed; 2 and then separately develop a syntax.  Here's a start:

Information that the UA will need:

* Optional pre-layout hints.

* The resolution of the images.  It would defeat the purpose if the UA has
to download each image to discover the sizes.

* Optional file size of the images (might be estimated from the resolutions).

* An author suggested strategy for selecting an image: sharp, thumbnail, etc.
This could have a defined common default.

* The intrinsic image size for the set for using in layout computation if the image
box size is not defined by styling or by a 'width' and 'height' attribute.


This information could be used as follows:

* If pre-layout hints are supplied then the UA may use these to select an image
to pre-load.  The user might instruct the UA to delay selection until after
layout computation so that a more accurate choice can be made - this would
generally introduce some latency but might reduce bandwidth usage.  Developer
tools and server/CDN optimizers should be capable of filling these statically or
dynamically.  It might be necessary to extend the parameters supported.

* Alternatively, the UA may in some cases be able to use the image
resolution/size/strategy to choose a unique image to pre-load.  For example
if the author suggests a 'thumbnail' strategy then the UA could pre-load the
smallest image.  If the user has requested the 'thumbnail' strategy be
forced to minimize bandwidth then the UA can also pre-load the smallest image.

* If the users has chosen to wait until after layout before making a selection
or pre-layout hints are not supplied then wait until after layout and then use the
image resolution/size/strategy to choose an image.  For example if the author
suggests the 'sharp' strategy and the UA concurs then it can select the smallest
image that still yields a sharp image.


The syntax:

* Consider avoiding redundancy in the information?

* Can only the first image resolution be defined, with the resolution of the others
images being defined by a scale factor?

* Should the pre-load hints be in a separate attribute?

* Can the resolution/size/strategy be used to support pre-load hints that might
be more intuitive and simpler for authors?


Perhaps everyone could find the patience to take one more look at this and
try and come up with a better outcome.  Can we get some buy-in from one
of the larger browser vendors.  Mozilla have a significant mobile strategy
now and you would think this issue would be on their radar?  Google have
CDN plans to optimize content based on the user agent, and could they
please consider helping this effort?


Then again, perhaps the srcset hints solve all the problems and I'm just too
dumb to understand how it will all work.

cheers
Fred