Re: [filter-effects] New syntax proposal for 'custom' filter function

On Mon, Dec 10, 2012 at 7:09 AM, Dirk Schulze <dschulze@adobe.com> wrote:
> On Dec 10, 2012, at 3:26 AM, David Sheets <kosmo.zb@gmail.com> wrote:
>> On Tue, Dec 4, 2012 at 10:09 PM, Dirk Schulze <dschulze@adobe.com> wrote:
>>> #1 custom-filter at-rule
>>>
>>> The syntax for the at-rule would look like in the following example:
>>>
>>>        @custom-filter curlGLSL {
>>>                format: "glsl-1"; /* we can find a better string for it */
>>>                fragment: url(fragmentShader.fs);
>>>                vertex: url(vertexShader.vs);
>>>                mesh: 4 4;
>>>                mix: source-in multiply;
>>>                parameters: identityMatrix mat3(1,0,0,0,1,0,0,0,1);
>>>        }
>>>
>>>        @custom-filter curlHLSL {
>>>                format: "hlsl";
>>>                fragment: url(fragmentShader.fs);
>>>                vertex: url(vertexShader.vs);
>>>                geometry: url(geometryShader.gs);
>>>                mesh: 4 4;
>>>                parameters: identityMatrix mat3(1,0,0,0,1,0,0,0,1);
>>>        }
>>>
>>>        filter: custom(curlHLSL, curlGLSL);
>>>
>>> The custom filter provides a list of fallback shaders. Every fallback would need another at-rule definition. This will blow-up the CSS code, but may could be solved with media queries to other possibilities. The problem are the descriptors. We can not define descriptors for all shaders today, nor can we predict descriptors for future languages. We would need to allow other shading languages to extend the CSS syntax for their needs.
>>
>> I refer to this proposal as an effective starting point as it provides
>> a complete representation for the present operational model in the
>> working draft.
>
> I thought you referred to #2, the reason for my confusion.
>
>>
>>> #2 a generic filter at rule
>>>
>>> The syntax could look like the following:
>>>
>>>        @filter curl {
>>>                src: url(shaderGLSL) format("glsl-1"), url(shaderHLSL) format("hlsl");
>>>                parameters: identityMatrix mat3(1,0,0,0,1,0,0,0,1);
>>>        }
>>>
>>>        filter: custom(curl);
>>
>> This proposal totally neglects the operational model in the working
>> draft. The parameters and semantics of the previous proposal are now
>> hidden inside of some new effect format.
>>
>>> I actually do not assume that every web developer will write the shader code their own. This is where your idea with the shader server comes into the game. The syntax is very clear, very short and can be adapted and integrated by web developers easily.
>>
>> Writing shader code is not required for customizing shader code or its
>> parameterization. This proposal is clear and short by hiding of
>> semantics inside of the effect formats. Where does the author specify
>> mesh dimensions?
>
> These can be passed by the 'parameters' descriptor as well.

What can't be overridden inside CSS? Put another way, what components
of the module parameterization must be expressed in your #2 effect
format? More than a vertex/fragment source pair? Will the mesh and
blending models be removed from Filter Effects 1.0?

>> Many developers will not want to use a new effect format. There are
>> similarly easy solutions that offer more flexibility to developers and
>> users and require less support by browsers.
>
> The question is: Do we need to make it unnecessary complex for people who don't understand shaders (but want to use shaders in their style sheets), if there is an easier alternative? #2 allows Filter Effects to be more flexible (see below) on the cost of introducing a packaging mechanism.

I don't understand what is unnecessarily complex about:

@import url("curl.css");

filter: custom(curl)

I don't understand how #2 is more flexible than a generic resource
model that matches the rest of the web. Can I use an xml-stylesheet
with #2 to render interactive documentation in a browser when a
developer loads the shader module URL in their address bar? I think
source-embedded in HTML is required for this (or XSLT generating
JavaScript strings from GLSL source... ewww).

If @import supported application/xml + xml:id (or text/html +
script[@type='text/css']/@id), several default parameterizations could
be bundled in arbitrary XML with shader source. This seems both
extremely flexible and simple to use.

>>> The fallback is integrated as well, and the CSS syntax does not need to be changed by the supported shading languages if there is a need to. The syntax is a lot more future proof and allows the extension by other formats.
>>
>> Fallback is easily integrated into the previous proposal. CSS syntax
>> only needs to be changed to allow a new @-rule and associated schema
>> for each individual operational model. As the operational model for
>> filter effects is part of filter effects and not necessarily tied to a
>> specific language, this coupling seems justified.
>>
>> See my proposal at the end of this mail for a future-proof and
>> extensible solution.
>
> The idea of the same name for the shader and it's fallback is nice. I really like it. It doesn't work for CSS Animations in the way we would need it for the at-rule on #1. I will check if this is even specified by CSS Animations.

One important distinction between my proposal at
<http://dsheets.github.com/custom-fx-modules/> and your proposal #1 is
the separation of different FX Module Parameterizations from FX
Function Resources. I believe falling back makes sense between FX
Function Resources which potentially use different (but operationally
compatible) FX Languages or between FX Function Resources which use
optional features (e.g. language extensions).

You could prioritize filters using different FX Operational Models by
constructing multiple filters each of which has only one alternative
in a single model. I find this unsatisfactory, though, in the case
where a user agent understands multiple operational models. In that
case, I would like the user agent to select the model it supports best
between multiple equivalent alternative expressions using different
models. I don't believe this UA discretion is available in a design
which treats different models the same as different function
resources.

>>> The syntax is very similar to @font-face and shares some concepts. @font-face does allow more then one format (OpenType, TrueType and WOFF as container).
>>
>> font-face is both more monolithic and less customizable than custom
>> filter effects functions.
>>
>>> The down side is that it just hides the parametrization. With the current syntax, just one URI would be possible. The parametrization and the shaders would need to be request able with just this one URI. There are different ideas how to solve this issue. XML is just one idea.
>>
>> I don't see anything about the parameterization that doesn't fit in
>> either the reference from CSS or the shader source itself. What does a
>> new place to encode parameterization win us?
>
> - A clean style sheet for users that don't want to deal with shaders.

Use @import.

> - No fragmentation of implementations for CSS

Most UAs will support only 1 operational model. Between the UAs, it
seems unlikely for more than 2 or 3 models to be simultaneously viable
for widescale use. Each model must be expressible with a
descriptor/type CSS @-rule schema. No new syntactic constructs should
be allowed in new models. Present UAs should already be able to easily
add new @-rule schemas.

> - Extensibility of Filter Effects with more effects (no spec change required)

Other specifications may at any time refer to Filter Effects'
operational model type extensibility feature (your "format") and
define their own @filter schema without modification to Filter
Effects. Filter Effects should describe how these unknown models are
to be ignored (as it already must).

> - Future proof Filter Effects, since the at-rule can be reused by features different from any shading language (see parametrization of SVG filters in the example from Lea[1])

>From SVG 1.1 Chapter 15 Filter Effects Section 7
<http://www.w3.org/TR/SVG/filters.html#FilterPrimitivesOverview>:
"Unless otherwise stated, all image filters operate on premultiplied
RGBA samples."

This would seem to indicate that SVG filters are a type of pixel
transformation FX Function Resource that would be referenceable in the
"pxfn" or "fragment" descriptor of my @filter rule or your
@custom-filter rule respectively. If a UA supports this type of pixel
transformation, any network request for a resource in this context
should include type "image/svg+xml" or "application/svg+xml" whichever
is appropriate.

The SVG Filter Effects model appears to be a subset of the Filter
Effects 1.0 model (vertex+fragment in GLSL parlance). By generating
network requests with appropriate media types, multiple shading
languages (SVG, GLSL, OtherSL...) that use the same operational model
may be transparently used with the same @filter rule.

> These are some benefits of #2.

I don't see anything decisive over #1 with an added complication of
the mandatory introduction of XML.

>>
>>> It looks like some of your arguments are based on proposal #1 (your reference of the geometry descriptor for example). In this case I would understand the proposals with fragments and <script> elements.
>>
>> Proposal #1 contains the model that proposal #2 hides. I referred to a
>> geometry descriptor in response to your assertion that such an
>> extension would require a new 'format'. I believe both #1 and #2 can
>> handle this issue but #2 does not require standards to use consistent
>> semantics (bad) as #1 does (good).
>>
>> Using fragids and associated machinery is one example of reusing an
>> existing system to achieve similar goals as bundled effects.
>
> Right, we don't loose functionality with both proposals.

With a mandatory XML bundle format we lose flexibility to reference
GLSL directly, easy inlining of module parameterization, and ability
to use an already existing or completely custom bundle format. We also
lose the ability to bundle other CSS rules with module
parameterizations and to serve a single resource that supports
multiple FX Operational Models or FX Languages.

Additionally, dynamic applications would be required to construct XML
to load custom filter effects and advanced CSS authors would have to
incorporate XML processing into their workflow.

>>> Or I don't understand how fragments can solve the problem of proposal #2.
>>
>> See <http://dsheets.github.com/custom-fx-modules/> which may answer
>> your question (@import + frauds).

Oops, s/frauds/fragids/.

> Ah I see, in this case you are still relying on some kind of format of the shader.

Which format? CSS @filter? An arbitrary format that has fragment
identifier semantics defined? No specific format is required (CSS is
already in use).

> Which would need to be described in Filter Effects and is not necessarily compatible with current behavior of browsers, even if it solved the multiple request issue.

Fragment identifiers resolve the multiple request issue. Fragment
identifiers are compatible with current behavior of browsers. A
generic URI reference would have to be described in Filter Effects but
no specific format would need to be (nor should be) described by
Filter Effects.

> That is what you mean with either an XML file or what I called "compiler flags" in the combined shader file.

The point is that there are many way to bundle secondary resources and
none of them should prevent using the others. The author should have
the freedom to choose the format that fits their use the best.

>>> I don't see your point that we would break existing web technologies with any of the above proposals, but am happy to adapt the proposal if it does.
>>
>> I don't believe I ever asserted that any of these proposals broke
>> existing web technologies. Proposal #0 does not satisfy extensibility
>> requirements. Proposal #1 could be refined into a powerful tool
>> <http://dsheets.github.com/custom-fx-modules/>. Proposal #2 ignores
>> features of HTTP, URI, HTML, XML, and CSS in favor of introducing a
>> specific new XML vocabulary which will not have the same flexibility
>> as easily as the rest of the platform already has.
>
> Your post actually points out the issue: "The server may respond to this request with a message indicating Content-Type of "application/xml" encapsulating "application/essl" source which the client will then interpret as detailed below.". This means the client can not rely on the header anyway. So why should we consider it?

The header is for server applications that understand how to negotiate
package formats and shading languages. If the client understands
application/xml and application/webglsl, it should tell the server
that it understands these ways to bundle and render graphics,
respectively. The server may then take this information into account
and respond with a raw shader or shader bundle that the client
understands.

Of course, many servers do not offer this functionality or may respond
with nonsensical content types. This is always the case and UAs
already have sophisticated content type sniffing algorithms to deal
with this eventuality. This absolutely should not prevent the (small
number) of browsers from making sensible, extensible network requests.

> Furthermore, "application/xml" would just be the case if you embed the shader files in an XML file.  Your main argument against proposal #2.

No. Please reread my proposal and my arguments carefully. My main
concern with proposal #2 is the *mandatory* use of a *specific* XML
vocabulary. For many applications and users, XML shader bundles make
sense. Many authors will want to use JSON bundles (and can with
appropriate fragment identifier semantics). We have a generic way to
enable this in the form of xml:id. A server *should not* respond with
application/xml if the result cannot be interpreted by a *generic* XML
processor. Proposal #2 would have a server respond with something like
application/filter-effect+xml and would have clients only understand
this format.

If a specific XML format is mandated, it will require a slew of
adapters to transform already in-use asset bundle formats into that
XML format. This seems unnecessarily restrictive and prescriptive when
nearly every other web technology has (already implemented, deployed,
and proven) extensibility features that Custom Filter Effects could
rely upon.

What can a 3rd syntax (specific XML vocabulary) do that CSS and *SL
can't already do? What are the benefits of mandating an entirely new
technology stack between CSS and *SL?

David

Received on Monday, 10 December 2012 19:47:20 UTC