Re: [filter-effects] New syntax proposal for 'custom' filter function

On Mon, Dec 3, 2012 at 3:46 PM, Dirk Schulze <dschulze@adobe.com> wrote:
> On Dec 3, 2012, at 2:30 PM, David Sheets <kosmo.zb@gmail.com> wrote:
>> Once a media subtype is chosen for the shading language, I believe
>> this parameter should mirror that string (i.e. "application/webglsl"
>> => "webglsl" or "text/essl" => "essl" or "application/glsl" => "glsl"
>> (yikes)).
>
> Dependent on the file format, it might not be necessary to rely on the MIME type. MIME types can also be a burden if the server does not support certain MIME types. The string for the format would be to determined. Any format needs to make sure that the requirements of the used language are fulfilled.

I don't understand your objection to alignment with a media type. HTML
has a media type, "text/html". CSS has a media type, "text/css".
Javascript has several media types. WebGLSL should have a media type
to refer to shading language resources that use WebGLSL's syntax and
semantics. If a package format is served, the serving host needs to
know what shading language should be served in the package. Each
language may differ in its CSS parameters.

If an author controls both page and shader server, this doesn't matter
very much. Hopefully, it is a goal of FXTF to consider third-party
shader effect hosts for which knowing as much as possible
declaratively about the interpretive capabilities of the client is
critically important.

How can accurate media types be a burden? The server is free to ignore
them. The browser should only send what it can understand in a given
context.

>>
>>> The 'src' descriptor is very generic and allows "packaging" of a feature. The feature is identified by the format passed as a string. This makes Filter Effects "extensible" by other specifications. We don't need to adapt Filter Effects to extend the support. Furthermore, we just have one HTTP request instead of multiple requests as before, which is a great benefit.
>>
>> A pipeline package/effect concept would definitely be useful in the
>> future but I am concerned about this direction for a number of
>> reasons:
>>
>> 1. A new mandatory, bespoke package format
>> 2. A new XML vocabulary that deserves careful design to support more
>> than just this use case
>> 3. Mixing and matching vertex and fragment effects becomes harder
>>
>> Can I package only a vertex effect and use a default fragment shader?
>> Custom fragment with default vertex?
>>
>> As an extension to custom CSS shaders, browser vendors may certainly
>> wish to implement package format support which may be indicated by the
>> appropriate Accept header.
>
> Ideally the package format should be determined by the string passed to the 'format()' function.

Why is a package format required? Why would the proposed XML
vocabulary be referenced by "glsl"? The platform already supports
means to request equivalent representations and extract fragments
thereof.

>> As I understand it, you have two goals with this package format:
>>
>> 1. shader bundling for semantic grouping
>> 2. shader bundling for network request reduction
>>
>> It occurs to me that previous shader effects proposals already had
>> semantic grouping in the form of separate CSS properties for separate
>> pipeline stages in a common at-rule. In this way, CSS files with
>> at-rules are essentially shader composition descriptions that may be
>> included, imported, or preprocessed into a single stylesheet or page.
>> The present proposal is essentially an XML representation of the
>> previous at-rule proposal. Is this accurate?
>>
>> In lieu of a mandatory new XML vocabulary, I propose an approach which
>> takes advantage of the already-present platform features of cascading
>> style sheet rules, fragment identifiers, media types, and/or shading
>> language analysis.
>>
>> Some facts:
>>
>> 1. CSS is modular/packaged via the cascade
>> 2. Multiple URLs with identical scheme, authority, path, and query
>> components but differing fragment parts used in a substantially
>> similar context (defined by browser session cache rules) result in a
>> single request
>> 3. HTML defines a subresource with @id that may be attached to <script>
>> 4. XML defines a subresource with @xml:id that may be attached to any element
>> 5. ESSL includes a lexical preprocessor with conditional blocks
>> 6. Shader type is detectable from and required by already deployed analysis
>>
>> Why not accept a URL for each pipeline component (vertex, fragment for
>> WebGLSL) and allow authors to refer to subresources? This gives
>> authors the freedom to use different URLs or supply a single effect
>> stage and does not require a specific XML package format unless
>> desired. This also gives each separate programmable pipeline stage a
>> unique name as a first-class resource.
>
> If we chose an XML file as manifest file, the proposed <fragment> and <vertex> elements can use xlink:href to reference external shader files:
>
> <fragment xlink:href="fragmentShader.fs"/>
>
> Is it that what you mean?

No. While this is possible and perhaps desirable for specification of
a packaging format (though requires multiple round-trips for
questionable gain), I am advocating that such a format remain optional
and is specified separately after the minimum and most flexible,
natural, and extensible support is specified and implemented.

Here are some examples to illustrate my proposed generic approach to
resource reference:

<html>
<head>
<script type="application/webglsl" id="my-vs">
// some VS code
</script>
<script type="application/webglsl" id="my-fs">
// some FS code
</script>
<script type="application/webglsl" id="other-fs">
// FS code
</script>
<style>
// @filter referencing url("#my-vs") and url("#my-fs") in vertex: and
fragment: rules for type("webglsl")
</style>
</head>
...
</html>

OR

serve a resource called "shader-pkg":

<my-app-specific-package-vocab>
<vertex xml:id="vs1">
// vs
</vertex>
<vertex xml:id="vs2">
// vs
</vertex>
<fragment xml:id="fs1">
// fs
</fragment>
<fragment xml:id="fs2">
// fs
</fragment>
<other-tag />
</my-app-specific-package-vocab>

with CSS referencing "shader-pkg#vs1" or "shader-pkg#fs2" to select
which shader.

OR

serve a resource called "combined-shader":

#ifdef VS
// some VS code
#elif defined(FS)
// some FS code
#endif

with references to "combined-shader#?D=VS" and "combined-shader#?D=FS"
for vertex shader and fragment shader respectively at the author's
discretion for their particular needs. More implicit approaches are
also possible, of course ("combined-shader#VS" might mean the macro
"VS" should be defined if no name "VS" exists in the source), but
begin to suffer from lack of flexibility and namespace collisions.

Support for this fragment identifier macro definition syntax is more
involved than the HTML/XML fragment support but has a number of useful
properties for shader customization (exactly the purpose of the
preprocessor) without sending nearly identical shaders or using
inefficient run-time conditionals in shaders.

Each of these options is specifiable and implementable separate from
the others and still allows loading each shader from a different URL
by just supplying that URL. No intermediary XML manifest format is
required (though, one may be easily supported with the xml:id method).

This approach additionally supports things like serving COLLADA files
as shader package resources with embedded shaders identified by
@xml:id.

> An XML file seems to be easier to support by browsers, since browsers already have a way to parse XML documents. This would avoid defining a parser for the description document. But we would indeed meet to define a schema for the XML file.

I am proposing to reuse the browser's CSS parsing for shader
composition, the browser's session cache for request deduplication,
the browser's HTML parser for embedding WebGLSL source, and the
browser's generic XML parser for selecting XML document fragments that
correspond to shader source. Implementation of a shader source macro
fragment query syntax would reuse the browser's shader analysis system
to insert name-value pairs into the shader's macro namespace. This
would allow functional specialization of generic shaders at load time,
something not presently possible with only uniform/attribute
parameters. This also requires coordination with Khronos in their
media type registration work to mandate macro fragment query syntax
support in resource resolution.

A new, mandatory package format appears to reduce author freedom,
increase vocabulary implementation count in browsers, and require
redefinition of concepts already present in the platform.

> At this point we didn't thought about the details of the file format, since we would like to agree on the CSS syntax first.

My point is that the details of the file format do not matter if you
totally decouple it from the CSS syntax through the use of URLs with
fragment identifiers.

The format of the manifest file can be directly represented in the CSS
syntax as with previous proposals. Instead of two places you can
specify parameters to override (XML and CSS), you only have one (CSS
at-rule). The proposed XML format will only be used by CSS unless you
are wanting to specify it for wider use among non-CSS/WebGL shader
programmers. If so, the specification task is much larger and has a
number of new constituencies. That work appears to be out of scope of
this TF and I don't see what necessary use cases it would address than
reusing the existing fragid system doesn't.

Instead of making a new vocabulary for version 1, why not reuse the
existing languages we already have: HTML, generic XML, CSS, WebGLSL?
If you'd like to transport metadata for shaders, why not propose new
#pragma directives to declare default parameters, copyright licenses,
etc so the entire WebGLSL and ESSL ecosystem can benefit regardless of
their XML manifest buy-in.

Using CSS for the effect composition format gets you:

1. built-in modularity, cascading, grouping just like any other CSS stylesheet
2. familiar syntax for designers without requiring XML (that also
works with their CSS preprocessing tools)
3. a single name to refer to an effect (the IDENT for the @-rule)
instead of both the @-rule IDENT as well as the shader package URL
4. trivial generation from XML with XSLT if you desire an XML source format
5. no new format semantics questions or hastily designed vocabularies
that only solve 50% of the problem and have slow uptake
6. fewer files, less duplication

>> Here are some simple uses:
>>
>> 1. Author writes script/@type="application/webglsl" with @id="curl" or
>> @id="my-effect-vs" in HTML and refers to it in later stylesheet or
>> from external stylesheet with fragment-identified page URL. Browser
>> sends "application/webglsl, text/html" for shader resource request (or
>> simply reads the loaded DOM if URL matches).
>> 2. Author embeds shaders in any XML dialect that they want with
>> @xml:id. Browser sends "application/webglsl, application/xml" for
>> shader resource request.
>> 3. FXTF proposes a WebGLSL fragment identifier syntax for a
>> to-be-registered WebGLSL media type that allows macros to be supplied.
>> For example: "pipeline-url#?D=VS&D=ITERATIONS=6" and
>> "pipeline-url#?D=FS". (-D for Define like command-line C
>> compilers/preprocessors) Shader authors then use normal (e.g. "#ifdef
>> FS") preprocessor directives in their (potentially combined) shaders.
>> 4. FXTF defines a new XML or JSON package format with fragment
>> identification specified.
>
> I am not sure if I got your point correctly. But the general Idea of the CSS WG is to separate styling from the content. That is why we try to void to bundle the style sheet to much to the document using it.

At one time, flexibility was also a goal of the CSS WG by allowing
cascading styles at various levels of a page. Only use #1 bundles
shader source into the document and should probably only be used with
inline styles. This would enable 1 network request for the entire
HTML+CSS+WebGLSL which could then stand alone. JavaScript may also be
used inline. Without fragment identifier support, I do not see how to
supply a single document which uses CSS effects (inline CSS + manifest
fetch for every VS/FS pair unless you use data blocks but then why
mandate the XML vocab?).

Uses #2, #3, and #4 decouple style from structure and do so in a way
that gives authors maximum flexibility to decide how they wish to
serve their shaders (everything from a single XML bundle with 50
shaders and a custom vocab to a single GLSL file with preprocessor
switches to a future manifest or module format).

Many authors wish to deliver all of their page assets in their own
manifest format. How will they take shaders from their format and load
them into CSS filter effects? I believe <script id="..."
type="application/webglsl"> insertion makes sense without requiring
them to generate XML (just CSS @-rules and source block loading). data
scheme URLs are also a possible method. In these dynamic cases, an
intermediate XML format doesn't make sense.

Regards,

David

Received on Tuesday, 4 December 2012 01:39:49 UTC