Re: [filter-effects] New syntax proposal for 'custom' filter function

Hi David,

On Dec 3, 2012, at 5:39 PM, David Sheets <kosmo.zb@gmail.com> wrote:

> On Mon, Dec 3, 2012 at 3:46 PM, Dirk Schulze <dschulze@adobe.com> wrote:
>> On Dec 3, 2012, at 2:30 PM, David Sheets <kosmo.zb@gmail.com> wrote:
>>> Once a media subtype is chosen for the shading language, I believe
>>> this parameter should mirror that string (i.e. "application/webglsl"
>>> => "webglsl" or "text/essl" => "essl" or "application/glsl" => "glsl"
>>> (yikes)).
>>
>> Dependent on the file format, it might not be necessary to rely on the MIME type. MIME types can also be a burden if the server does not support certain MIME types. The string for the format would be to determined. Any format needs to make sure that the requirements of the used language are fulfilled.
>
> I don't understand your objection to alignment with a media type. HTML
> has a media type, "text/html". CSS has a media type, "text/css".
> Javascript has several media types. WebGLSL should have a media type
> to refer to shading language resources that use WebGLSL's syntax and
> semantics. If a package format is served, the serving host needs to
> know what shading language should be served in the package. Each
> language may differ in its CSS parameters.

It is not an objection per se. Requesting documents with MIME type from a server is always a risk. I have problems with servers that don't accept "image/svg+xml" for SVG documents today. The specification is more then 12 years old. For interpretation on the client side it may work, since the server can still send it as text.

>
> If an author controls both page and shader server, this doesn't matter
> very much. Hopefully, it is a goal of FXTF to consider third-party
> shader effect hosts for which knowing as much as possible
> declaratively about the interpretive capabilities of the client is
> critically important.

In a lot of cases the author is not under control of the server. Therefore it is hard to rely on this assumption. This actually does not block your idea of a shader server. Furthermore, the shader files are not fully compatible with GLSL or WebGL, since we have more security requirements and predefined variables.

>
> How can accurate media types be a burden? The server is free to ignore
> them. The browser should only send what it can understand in a given
> context.

This depends how you request the document and how flexible the client is on the MIME type send by the server. With our proposal the user does not need to worry about it, since he defines the vertex and fragment shader on his own. This is already the case with the syntax in the working draft of Filter Effects.

>
>>>
>>>> The 'src' descriptor is very generic and allows "packaging" of a feature. The feature is identified by the format passed as a string. This makes Filter Effects "extensible" by other specifications. We don't need to adapt Filter Effects to extend the support. Furthermore, we just have one HTTP request instead of multiple requests as before, which is a great benefit.
>>>
>>> A pipeline package/effect concept would definitely be useful in the
>>> future but I am concerned about this direction for a number of
>>> reasons:
>>>
>>> 1. A new mandatory, bespoke package format
>>> 2. A new XML vocabulary that deserves careful design to support more
>>> than just this use case
>>> 3. Mixing and matching vertex and fragment effects becomes harder
>>>
>>> Can I package only a vertex effect and use a default fragment shader?
>>> Custom fragment with default vertex?
>>>
>>> As an extension to custom CSS shaders, browser vendors may certainly
>>> wish to implement package format support which may be indicated by the
>>> appropriate Accept header.
>>
>> Ideally the package format should be determined by the string passed to the 'format()' function.
>
> Why is a package format required? Why would the proposed XML
> vocabulary be referenced by "glsl"? The platform already supports
> means to request equivalent representations and extract fragments
> thereof.

I speak about a package format in a general manner. An XML file does not need to be a package of course. But a package could still be an option. Future formats can define other ways to define the feature. For "glsl" it could be XML. Again, XML is just a suggestion, since browsers already support this format natively and no extra parsing instruction is required.

>
>>> As I understand it, you have two goals with this package format:
>>>
>>> 1. shader bundling for semantic grouping
>>> 2. shader bundling for network request reduction
>>>
>>> It occurs to me that previous shader effects proposals already had
>>> semantic grouping in the form of separate CSS properties for separate
>>> pipeline stages in a common at-rule. In this way, CSS files with
>>> at-rules are essentially shader composition descriptions that may be
>>> included, imported, or preprocessed into a single stylesheet or page.
>>> The present proposal is essentially an XML representation of the
>>> previous at-rule proposal. Is this accurate?
>>>
>>> In lieu of a mandatory new XML vocabulary, I propose an approach which
>>> takes advantage of the already-present platform features of cascading
>>> style sheet rules, fragment identifiers, media types, and/or shading
>>> language analysis.
>>>
>>> Some facts:
>>>
>>> 1. CSS is modular/packaged via the cascade
>>> 2. Multiple URLs with identical scheme, authority, path, and query
>>> components but differing fragment parts used in a substantially
>>> similar context (defined by browser session cache rules) result in a
>>> single request
>>> 3. HTML defines a subresource with @id that may be attached to <script>
>>> 4. XML defines a subresource with @xml:id that may be attached to any element
>>> 5. ESSL includes a lexical preprocessor with conditional blocks
>>> 6. Shader type is detectable from and required by already deployed analysis
>>>
>>> Why not accept a URL for each pipeline component (vertex, fragment for
>>> WebGLSL) and allow authors to refer to subresources? This gives
>>> authors the freedom to use different URLs or supply a single effect
>>> stage and does not require a specific XML package format unless
>>> desired. This also gives each separate programmable pipeline stage a
>>> unique name as a first-class resource.
>>
>> If we chose an XML file as manifest file, the proposed <fragment> and <vertex> elements can use xlink:href to reference external shader files:
>>
>> <fragment xlink:href="fragmentShader.fs"/>
>>
>> Is it that what you mean?
>
> No. While this is possible and perhaps desirable for specification of
> a packaging format (though requires multiple round-trips for
> questionable gain), I am advocating that such a format remain optional
> and is specified separately after the minimum and most flexible,
> natural, and extensible support is specified and implemented.
>
> Here are some examples to illustrate my proposed generic approach to
> resource reference:
>
> <html>
> <head>
> <script type="application/webglsl" id="my-vs">
> // some VS code
> </script>
> <script type="application/webglsl" id="my-fs">
> // some FS code
> </script>
> <script type="application/webglsl" id="other-fs">
> // FS code
> </script>
> <style>
> // @filter referencing url("#my-vs") and url("#my-fs") in vertex: and
> fragment: rules for type("webglsl")
> </style>
> </head>
> ...
> </html>
>

I am not objecting to this ideas either and happy to discuss them as well. But I am skeptical because it does not cleanly separate between style and content in the document. Even if a lot of WebGL content already uses this way, it is not cleanly defined to it this way.

> OR
>
> serve a resource called "shader-pkg":
>
> <my-app-specific-package-vocab>
> <vertex xml:id="vs1">
> // vs
> </vertex>
> <vertex xml:id="vs2">
> // vs
> </vertex>
> <fragment xml:id="fs1">
> // fs
> </fragment>
> <fragment xml:id="fs2">
> // fs
> </fragment>
> <other-tag />
> </my-app-specific-package-vocab>
>
> with CSS referencing "shader-pkg#vs1" or "shader-pkg#fs2" to select
> which shader.

Would that expose a separate call for fragment and vertex shader in the CSS again? That is something that we tried to avoid with the last syntax, so that Filter Effects and different "extensions" (like GLSL shaders) can be updated independent of each other. This would make Filter Effects more future proof for changes in GLSL as well, since a new format would be used to differ between fundamental changes in GLSL. Who know if GLSL gets a geometry shader in the future as well? If so, the format can be updated and new string for the format() function would be introduced by the new format.

>
> OR
>
> serve a resource called "combined-shader":
>
> #ifdef VS
> // some VS code
> #elif defined(FS)
> // some FS code
> #endif
>
> with references to "combined-shader#?D=VS" and "combined-shader#?D=FS"
> for vertex shader and fragment shader respectively at the author's
> discretion for their particular needs. More implicit approaches are
> also possible, of course ("combined-shader#VS" might mean the macro
> "VS" should be defined if no name "VS" exists in the source), but
> begin to suffer from lack of flexibility and namespace collisions.

I am not sure if we want to expose "compiler flags" here. This seems not to be very intuitive. Do you see it as an alternative for serving an XML file (or other web based format)?

>
> Support for this fragment identifier macro definition syntax is more
> involved than the HTML/XML fragment support but has a number of useful
> properties for shader customization (exactly the purpose of the
> preprocessor) without sending nearly identical shaders or using
> inefficient run-time conditionals in shaders.
>
> Each of these options is specifiable and implementable separate from
> the others and still allows loading each shader from a different URL
> by just supplying that URL. No intermediary XML manifest format is
> required (though, one may be easily supported with the xml:id method).
>
> This approach additionally supports things like serving COLLADA files
> as shader package resources with embedded shaders identified by
> @xml:id.

The idea is to make the @filter rule generic enough, so that it could be extended with techniques like COLLADA (even if I did not thought about COLLADA).

>
>> An XML file seems to be easier to support by browsers, since browsers already have a way to parse XML documents. This would avoid defining a parser for the description document. But we would indeed meet to define a schema for the XML file.
>
> I am proposing to reuse the browser's CSS parsing for shader
> composition, the browser's session cache for request deduplication,
> the browser's HTML parser for embedding WebGLSL source, and the
> browser's generic XML parser for selecting XML document fragments that
> correspond to shader source. Implementation of a shader source macro
> fragment query syntax would reuse the browser's shader analysis system
> to insert name-value pairs into the shader's macro namespace. This
> would allow functional specialization of generic shaders at load time,
> something not presently possible with only uniform/attribute
> parameters. This also requires coordination with Khronos in their
> media type registration work to mandate macro fragment query syntax
> support in resource resolution.

I would like to limit exposing features as much as possible. Obviously shaders in CSS have needs that need to be addressed differently then for WebGL as example. In general, our positions seem not to be to far away, just the how differs.

>
> A new, mandatory package format appears to reduce author freedom,
> increase vocabulary implementation count in browsers, and require
> redefinition of concepts already present in the platform.

No, we do not redefine any concept already used by browsers and reuse all existing concepts. We are using established technologies and formatting, like @font-face, shaders and shader parsing from WebGL and possibly XML. There is nothing that is not supported by at least an alpha version of three different browsers.

>
>> At this point we didn't thought about the details of the file format, since we would like to agree on the CSS syntax first.
>
> My point is that the details of the file format do not matter if you
> totally decouple it from the CSS syntax through the use of URLs with
> fragment identifiers.

I was not talking about fragments in URLs so far. You introduced fragments in your concepts. Fragments are valid as well, but don't need to matter here.

>
> The format of the manifest file can be directly represented in the CSS
> syntax as with previous proposals. Instead of two places you can
> specify parameters to override (XML and CSS), you only have one (CSS
> at-rule). The proposed XML format will only be used by CSS unless you
> are wanting to specify it for wider use among non-CSS/WebGL shader
> programmers. If so, the specification task is much larger and has a
> number of new constituencies. That work appears to be out of scope of
> this TF and I don't see what necessary use cases it would address than
> reusing the existing fragid system doesn't.

The idea was to decouple the CSS parsing from the current version of GLSL or WebGL. We don't know how these technologies are evolving in the future, while CSS should be downwards compatible. Introducing keywords that might be outdated in a couple of years for instance is suboptimal.

>
> Instead of making a new vocabulary for version 1, why not reuse the
> existing languages we already have: HTML, generic XML, CSS, WebGLSL?

That is what our proposal did.

> If you'd like to transport metadata for shaders, why not propose new
> #pragma directives to declare default parameters, copyright licenses,
> etc so the entire WebGLSL and ESSL ecosystem can benefit regardless of
> their XML manifest buy-in.

I am not stopping the evolution of WegGL or GLSL, and the ecosystem of these specifications is not in danger either.

>
> Using CSS for the effect composition format gets you:
>
> 1. built-in modularity, cascading, grouping just like any other CSS stylesheet

This is the case with all proposals.

> 2. familiar syntax for designers without requiring XML (that also
> works with their CSS preprocessing tools)

If we can decouple it from the current state of WebGL and GLSL, then I am fine with it.

> 3. a single name to refer to an effect (the IDENT for the @-rule)
> instead of both the @-rule IDENT as well as the shader package URL

You need to have the possibility of fallback formats, I don't see this with just providing an IDENT, without a URL.

> 4. trivial generation from XML with XSLT if you desire an XML source format

Indeed.

> 5. no new format semantics questions or hastily designed vocabularies
> that only solve 50% of the problem and have slow uptake

This won't happen if we have a simple and clean definition of the format.

> 6. fewer files, less duplication

Ideally yes. This is already a benefit to the new concept in comparison to the old one.

>
>>> Here are some simple uses:
>>>
>>> 1. Author writes script/@type="application/webglsl" with @id="curl" or
>>> @id="my-effect-vs" in HTML and refers to it in later stylesheet or
>>> from external stylesheet with fragment-identified page URL. Browser
>>> sends "application/webglsl, text/html" for shader resource request (or
>>> simply reads the loaded DOM if URL matches).
>>> 2. Author embeds shaders in any XML dialect that they want with
>>> @xml:id. Browser sends "application/webglsl, application/xml" for
>>> shader resource request.
>>> 3. FXTF proposes a WebGLSL fragment identifier syntax for a
>>> to-be-registered WebGLSL media type that allows macros to be supplied.
>>> For example: "pipeline-url#?D=VS&D=ITERATIONS=6" and
>>> "pipeline-url#?D=FS". (-D for Define like command-line C
>>> compilers/preprocessors) Shader authors then use normal (e.g. "#ifdef
>>> FS") preprocessor directives in their (potentially combined) shaders.
>>> 4. FXTF defines a new XML or JSON package format with fragment
>>> identification specified.
>>
>> I am not sure if I got your point correctly. But the general Idea of the CSS WG is to separate styling from the content. That is why we try to void to bundle the style sheet to much to the document using it.
>
> At one time, flexibility was also a goal of the CSS WG by allowing
> cascading styles at various levels of a page. Only use #1 bundles
> shader source into the document and should probably only be used with
> inline styles. This would enable 1 network request for the entire
> HTML+CSS+WebGLSL which could then stand alone. JavaScript may also be
> used inline. Without fragment identifier support, I do not see how to
> supply a single document which uses CSS effects (inline CSS + manifest
> fetch for every VS/FS pair unless you use data blocks but then why
> mandate the XML vocab?).

If you worry about this, this is valid as well:

@filter curl {
        src: url(data:application/xml;base64,iVBORw…) format("glsl");
}

Which could be done with a minimizer.

>
> Uses #2, #3, and #4 decouple style from structure and do so in a way
> that gives authors maximum flexibility to decide how they wish to
> serve their shaders (everything from a single XML bundle with 50
> shaders and a custom vocab to a single GLSL file with preprocessor
> switches to a future manifest or module format).

Ok, I got your point. Now I understand your fragment request, in this case we just define multiple shaders in the XML file:

<shader id="curl">
        <fragment/>
        <vertex/>
</shader>
<shader id="wave">
        <fragment/>
        <vertex/>
</shader>

This sounds reasonable.

Greetings,
Dirk

>
> Many authors wish to deliver all of their page assets in their own
> manifest format. How will they take shaders from their format and load
> them into CSS filter effects? I believe <script id="..."
> type="application/webglsl"> insertion makes sense without requiring
> them to generate XML (just CSS @-rules and source block loading). data
> scheme URLs are also a possible method. In these dynamic cases, an
> intermediate XML format doesn't make sense.
>
> Regards,
>
> David

Received on Tuesday, 4 December 2012 21:33:55 UTC