Re: [filter-effects] New syntax proposal for 'custom' filter function

Hi Dirk,

On Tue, Dec 4, 2012 at 1:33 PM, Dirk Schulze <dschulze@adobe.com> wrote:
> On Dec 3, 2012, at 5:39 PM, David Sheets <kosmo.zb@gmail.com> wrote:
>> On Mon, Dec 3, 2012 at 3:46 PM, Dirk Schulze <dschulze@adobe.com> wrote:
>>> On Dec 3, 2012, at 2:30 PM, David Sheets <kosmo.zb@gmail.com> wrote:
>>>> Once a media subtype is chosen for the shading language, I believe
>>>> this parameter should mirror that string (i.e. "application/webglsl"
>>>> => "webglsl" or "text/essl" => "essl" or "application/glsl" => "glsl"
>>>> (yikes)).
>>>
>>> Dependent on the file format, it might not be necessary to rely on the MIME type. MIME types can also be a burden if the server does not support certain MIME types. The string for the format would be to determined. Any format needs to make sure that the requirements of the used language are fulfilled.
>>
>> I don't understand your objection to alignment with a media type. HTML
>> has a media type, "text/html". CSS has a media type, "text/css".
>> Javascript has several media types. WebGLSL should have a media type
>> to refer to shading language resources that use WebGLSL's syntax and
>> semantics. If a package format is served, the serving host needs to
>> know what shading language should be served in the package. Each
>> language may differ in its CSS parameters.
>
> It is not an objection per se. Requesting documents with MIME type from a server is always a risk. I have problems with servers that don't accept "image/svg+xml" for SVG documents today. The specification is more then 12 years old. For interpretation on the client side it may work, since the server can still send it as text.

What server has the following behavior?
You request URI Q with "Accept: image/svg+xml" and the server responds
with code 406 with a body that isn't image/svg+xml.
You request URI Q with "Accept: text/plain" and the server responds
with code 200 with a body that is image/svg+xml but marked as
text/plain and interpreted by the client as image/svg+xml.

If this actually occurs, the server is clearly broken by
simultaneously being too strict and not strict enough. This does not
prohibit the specification of reasonable media type behavior, however.
E.g:

You request URI Q with "Accept: application/webglsl, text/plain" and
the server responds with something you can interpret with
"Content-type: application/webglsl" or "Content-type: text/plain".
Ideally, you request URI Q with "Accept: application/webglsl,
application/xml" or however many types you can interpret in this
context and if the server responds with "Content-type: text/plain" you
have to sniff as always.

Please explain the risk of requesting the specific media types that
the client is capable of interpreting.

>> If an author controls both page and shader server, this doesn't matter
>> very much. Hopefully, it is a goal of FXTF to consider third-party
>> shader effect hosts for which knowing as much as possible
>> declaratively about the interpretive capabilities of the client is
>> critically important.
>
> In a lot of cases the author is not under control of the server. Therefore it is hard to rely on this assumption. This actually does not block your idea of a shader server.

What assumption? I am not concerned with being unable to implement a
shader server. I am very concerned that the CSS FX specification will
not specify network requests detailing what specific languages the
client can interpret. What are the downsides of alignment with the
design principles of HTTP?

> Furthermore, the shader files are not fully compatible with GLSL or WebGL, since we have more security requirements and predefined variables.

Every valid CSS FX WebGLSL shader is also a valid WebGLSL shader. Your
security requirements may be satisfied in one of two ways:

1. a language extension to introduce new annotations for secure
textures which are easily removed by non-sensitive processors
2. a language coextension to declare certain normally-allowed texture
manipulations as globally disabled

All existing processors (WebGL and ESSL), to my knowledge, simply
ignore the coextension declaration as they have no problem "disabling"
extensions of which they are ignorant. This solution is the least
disruptive and clearly indicates CSS FX shaders are a subset of all
WebGLSL shaders which can be used interchangeably without requiring
insensitive WebGLSL processors understand a new security restriction
extension just so they can ignore it. See
<https://www.khronos.org/webgl/public-mailing-list/archives/1211/msg00132.html>
for more info.

As for predefined variables, they constitute a shading pipeline
interface just like non-CSS FX shaders have interfaces. Everything
that CSS FX mandates for shaders is emulable with WebGLSL.

>> How can accurate media types be a burden? The server is free to ignore
>> them. The browser should only send what it can understand in a given
>> context.
>
> This depends how you request the document and how flexible the client is on the MIME type send by the server. With our proposal the user does not need to worry about it, since he defines the vertex and fragment shader on his own. This is already the case with the syntax in the working draft of Filter Effects.

There will be a type interpretation task for the client under all
circumstances. This does not affect the client's request. Simple
servers may simply read a file and push it down with an
extension-based or default media type. Nothing I suggest prevents or
spoils this behavior.

In most cases, for any proposal, the client will not have to worry
about it. In the cases where it does matter (translations of a single
shader into multiple languages, translations of a package into
multiple formats), your proposal will still have to address this
issue. A request for "application/xml" is insufficient if the response
is expected to contain tags with specific semantics or encapsulate
shaders in a specific language. A request for "application/xml,
application/webglsl" is appropriate if the response may be interpreted
using generic XML features like @xml:id to yield WebGLSL source.

>>>>> The 'src' descriptor is very generic and allows "packaging" of a feature. The feature is identified by the format passed as a string. This makes Filter Effects "extensible" by other specifications. We don't need to adapt Filter Effects to extend the support. Furthermore, we just have one HTTP request instead of multiple requests as before, which is a great benefit.
>>>>
>>>> A pipeline package/effect concept would definitely be useful in the
>>>> future but I am concerned about this direction for a number of
>>>> reasons:
>>>>
>>>> 1. A new mandatory, bespoke package format
>>>> 2. A new XML vocabulary that deserves careful design to support more
>>>> than just this use case
>>>> 3. Mixing and matching vertex and fragment effects becomes harder
>>>>
>>>> Can I package only a vertex effect and use a default fragment shader?
>>>> Custom fragment with default vertex?
>>>>
>>>> As an extension to custom CSS shaders, browser vendors may certainly
>>>> wish to implement package format support which may be indicated by the
>>>> appropriate Accept header.
>>>
>>> Ideally the package format should be determined by the string passed to the 'format()' function.
>>
>> Why is a package format required? Why would the proposed XML
>> vocabulary be referenced by "glsl"? The platform already supports
>> means to request equivalent representations and extract fragments
>> thereof.
>
> I speak about a package format in a general manner. An XML file does not need to be a package of course. But a package could still be an option. Future formats can define other ways to define the feature. For "glsl" it could be XML. Again, XML is just a suggestion, since browsers already support this format natively and no extra parsing instruction is required.

I don't see what "glsl" has to do with XML. What extra parsing was
required in the previous proposal?

Definition and implementation of the specific XML vocabulary is still
required. CSS @filter rules syntax is still required. I propose the
use of fragment identifiers which removes the requirement to implement
a specific XML vocabulary and removes the mandate that authors use a
specific XML vocabulary even when it is inappropriate to their use
case.

For packaging purposes, CSS @-rule + @import can package effects
without duplicating semantics between CSS and XML.

>> Here are some examples to illustrate my proposed generic approach to
>> resource reference:
>>
>> <html>
>> <head>
>> <script type="application/webglsl" id="my-vs">
>> // some VS code
>> </script>
>> <script type="application/webglsl" id="my-fs">
>> // some FS code
>> </script>
>> <script type="application/webglsl" id="other-fs">
>> // FS code
>> </script>
>> <style>
>> // @filter referencing url("#my-vs") and url("#my-fs") in vertex: and
>> fragment: rules for type("webglsl")
>> </style>
>> </head>
>> ...
>> </html>
>>
>
> I am not objecting to this ideas either and happy to discuss them as well. But I am skeptical because it does not cleanly separate between style and content in the document. Even if a lot of WebGL content already uses this way, it is not cleanly defined to it this way.

My proposal allows authors to use ANY of these approaches (except
maybe the WebGLSL macro approach which requires fragment microsyntax
definition).

This use case is a result of reuse of existing platform features. Of
course using in-line shader is not the best way to design large
applications but neither is <style> or @style and still they are
extremely useful under certain circumstances (e.g. dynamic insertion).

Could you please elaborate on how the definition of the functionality
is not clean?

>> OR
>>
>> serve a resource called "shader-pkg":
>>
>> <my-app-specific-package-vocab>
>> <vertex xml:id="vs1">
>> // vs
>> </vertex>
>> <vertex xml:id="vs2">
>> // vs
>> </vertex>
>> <fragment xml:id="fs1">
>> // fs
>> </fragment>
>> <fragment xml:id="fs2">
>> // fs
>> </fragment>
>> <other-tag />
>> </my-app-specific-package-vocab>
>>
>> with CSS referencing "shader-pkg#vs1" or "shader-pkg#fs2" to select
>> which shader.
>
> Would that expose a separate call for fragment and vertex shader in the CSS again?

Mesh and pixel transforms are concepts that Filter Effects must
address. These concepts have direct analogs in every dialect of GLSL,
as well.

> That is something that we tried to avoid with the last syntax, so that Filter Effects and different "extensions" (like GLSL shaders) can be updated independent of each other. This would make Filter Effects more future proof for changes in GLSL as well, since a new format would be used to differ between fundamental changes in GLSL. Who know if GLSL gets a geometry shader in the future as well? If so, the format can be updated and new string for the format() function would be introduced by the new format.

This is a great reason to not call this language "glsl" as some
dialects of GLSL already have geometry shaders. I love the idea of a
language type parameter in CSS. I think it really frees Filter Effects
from a lot of lower-level details. I see this language type parameter
as analogous to the language's media type. Is it not?

I don't see any problem here with directly specifying the mesh and
pixel programs. If geometry shaders are introduced, a new "format()"
may still be minted which allows "geometry:" in the @-rule. The @-rule
is the package. The @-rule has an IDENT for the name of the applicable
effect.

If, in future, the browser understands a fragment identifier syntax
for "application/foo+json", this type may be used as a package format
and requested in a CSS FX context without changing Filter Effects or
defining a new "format()".

>> OR
>>
>> serve a resource called "combined-shader":
>>
>> #ifdef VS
>> // some VS code
>> #elif defined(FS)
>> // some FS code
>> #endif
>>
>> with references to "combined-shader#?D=VS" and "combined-shader#?D=FS"
>> for vertex shader and fragment shader respectively at the author's
>> discretion for their particular needs. More implicit approaches are
>> also possible, of course ("combined-shader#VS" might mean the macro
>> "VS" should be defined if no name "VS" exists in the source), but
>> begin to suffer from lack of flexibility and namespace collisions.
>
> I am not sure if we want to expose "compiler flags" here. This seems not to be very intuitive. Do you see it as an alternative for serving an XML file (or other web based format)?

Not every "compiler flag" would be exposed in this fragment
microsyntax. What do you find unintuitive about specifying a portion
of a shader constructed after preprocessor partial evaluation?

Fragment macros are an alternative to serving many nearly identical
shaders at significant and unnecessary bandwidth cost. Because macros
act at the syntactic level, they change the semantic interpretation of
the shader resource and it makes sense to address them as different
aspects, with different meanings, of the same identified resource.

The WebGLSL preprocessor is a fundamental part of the shading language
and should always be supported by a consuming specification at some
level.

An alternative is for the FXTF to define macro definition facilities
along-side the uniform/attribute facilities. I find this solution to
be worse for several reasons:

1. Macros may change the pipeline stage type of a single shader
resource (VS <-> FS as above).
2. Macros must be applied prior to uniforms/attributes as they may
change the shader program's interface fields and types
3. Macros can contain WebGLSL strings

For these reasons, I believe macro definition should be done as early
as possible (i.e. in the reference to a subresource of a shader).

>> Support for this fragment identifier macro definition syntax is more
>> involved than the HTML/XML fragment support but has a number of useful
>> properties for shader customization (exactly the purpose of the
>> preprocessor) without sending nearly identical shaders or using
>> inefficient run-time conditionals in shaders.
>>
>> Each of these options is specifiable and implementable separate from
>> the others and still allows loading each shader from a different URL
>> by just supplying that URL. No intermediary XML manifest format is
>> required (though, one may be easily supported with the xml:id method).
>>
>> This approach additionally supports things like serving COLLADA files
>> as shader package resources with embedded shaders identified by
>> @xml:id.
>
> The idea is to make the @filter rule generic enough, so that it could be extended with techniques like COLLADA (even if I did not thought about COLLADA).

Adobe's present proposal does not achieve this genericity by requiring
a bespoke effect format. An explicit extension for COLLADA is heavier
than necessary.

By using XML's generic @xml:id and fragment identifiers, both Adobe's
effect package format as well as COLLADA (and head/script shaders and
custom XML vocab shaders and pure GLSL shaders) may be supported with
a single "format()" definition and optional mesh/pixel processor URIs.

>>> An XML file seems to be easier to support by browsers, since browsers already have a way to parse XML documents. This would avoid defining a parser for the description document. But we would indeed meet to define a schema for the XML file.
>>
>> I am proposing to reuse the browser's CSS parsing for shader
>> composition, the browser's session cache for request deduplication,
>> the browser's HTML parser for embedding WebGLSL source, and the
>> browser's generic XML parser for selecting XML document fragments that
>> correspond to shader source. Implementation of a shader source macro
>> fragment query syntax would reuse the browser's shader analysis system
>> to insert name-value pairs into the shader's macro namespace. This
>> would allow functional specialization of generic shaders at load time,
>> something not presently possible with only uniform/attribute
>> parameters. This also requires coordination with Khronos in their
>> media type registration work to mandate macro fragment query syntax
>> support in resource resolution.
>
> I would like to limit exposing features as much as possible. Obviously shaders in CSS have needs that need to be addressed differently then for WebGL as example. In general, our positions seem not to be to far away, just the how differs.

What exposed features are you referring to here?

>> A new, mandatory package format appears to reduce author freedom,
>> increase vocabulary implementation count in browsers, and require
>> redefinition of concepts already present in the platform.
>
> No, we do not redefine any concept already used by browsers and reuse all existing concepts. We are using established technologies and formatting, like @font-face, shaders and shader parsing from WebGL and possibly XML.

@font-face src refers to a monolithic non-text format with a
well-defined interface and limited end-user re-application or
consumption. Shaders are variable modules of textual source code
without a fixed interface that are easily read, modified, and
re-appropriated by end-users.

Adobe's proposed format redefines default parameters and effect
packaging that are already required in CSS.

If this format is going to support manifests with multiple alternate
representations, Adobe's format will redefine content negotiation.

If this format is going to support selecting only a fragment shader or
only a vertex shader or a VS/FS pair that's not already paired,
Adobe's format will redefine resource fragment naming and a syntax
will need to be devised for CSS to address these subresources.

If Adobe's format does not support mix-n-match due to coupling vertex
and fragment programs, duplicate data will be transmitted.

What use cases does Adobe's proposed format support that are not
supported by URI references?

> There is nothing that is not supported by at least an alpha version of three different browsers.

Is the proposed XML vocabulary currently supported? I believe all the
semantics of CSS properties and generic URI references are already
standard.

>>> At this point we didn't thought about the details of the file format, since we would like to agree on the CSS syntax first.
>>
>> My point is that the details of the file format do not matter if you
>> totally decouple it from the CSS syntax through the use of URLs with
>> fragment identifiers.
>
> I was not talking about fragments in URLs so far. You introduced fragments in your concepts. Fragments are valid as well, but don't need to matter here.

Why don't they need to matter? They appear to solve the network
request consolidation problem as well as the semantic grouping
problem. Their use heavily reuses concepts and implementations in the
rest of the platform.

>> The format of the manifest file can be directly represented in the CSS
>> syntax as with previous proposals. Instead of two places you can
>> specify parameters to override (XML and CSS), you only have one (CSS
>> at-rule). The proposed XML format will only be used by CSS unless you
>> are wanting to specify it for wider use among non-CSS/WebGL shader
>> programmers. If so, the specification task is much larger and has a
>> number of new constituencies. That work appears to be out of scope of
>> this TF and I don't see what necessary use cases it would address than
>> reusing the existing fragid system doesn't.
>
> The idea was to decouple the CSS parsing from the current version of GLSL or WebGL. We don't know how these technologies are evolving in the future, while CSS should be downwards compatible. Introducing keywords that might be outdated in a couple of years for instance is suboptimal.

What keywords will be outdated? "vertex"? "fragment"? The Filter
Effects spec must already contain analogous concepts to define mesh
filters and pixel filters. In what way do you foresee WebGLSL evolving
enough but staying the same enough to cause a problem? How would this
evolution not be addressed with a new "format()", a new media type or
media type parameter, or a built-in shader compatibility wrapper?

>> Instead of making a new vocabulary for version 1, why not reuse the
>> existing languages we already have: HTML, generic XML, CSS, WebGLSL?
>
> That is what our proposal did.

I'm sorry if I misunderstood your proposal. Does your proposal define
semantics for new XML tags? Does your proposal use only features that
generic XML processors understand? How do I refer to a specific
fragment shader inside of an Adobe CSS Effect?

>> If you'd like to transport metadata for shaders, why not propose new
>> #pragma directives to declare default parameters, copyright licenses,
>> etc so the entire WebGLSL and ESSL ecosystem can benefit regardless of
>> their XML manifest buy-in.
>
> I am not stopping the evolution of WegGL or GLSL, and the ecosystem of these specifications is not in danger either.

I am not asserting either your interference or global damage. However,
I am asserting that Adobe's present proposal incorporates features in
a wrapper format that could profitably be included directly in shaders
with a little cooperation. The incorporation of these features would
benefit everyone and not just those who choose to use this specific
XML vocabulary. Rather than having each package format redefine these
metadata properties, we could specify a way to embed metadata directly
into programs.

Consider the case of copyright/license. This information is directly
related to the actual program expression. Why should it only be
included in one XML wrapper format? A simple xpath expression would
strip this information from the program code. In fact, I believe that
in-page submission of open source licensed shaders to a WebGLSL
processor without a license declaration inline (reaching the WebGL
API) constitutes source redistribution without copyright/license
notice. Use of the proposed XML format with license ONLY in the
manifest would then uniformly violate the license for every manifest
user.

This is one reason why copyright declarations and licenses are often
placed in source code header comments.

If you would like to solve this problem, I would love to collaborate
with you to define a way to embed programmatically actionable metadata
directly into shader code. I have been studying related problems quite
intensely for over a year now.

>> Using CSS for the effect composition format gets you:
>>
>> 1. built-in modularity, cascading, grouping just like any other CSS stylesheet
>
> This is the case with all proposals.

The present XML vocabulary proposal duplicates grouping as it is
already present in CSS.

>> 2. familiar syntax for designers without requiring XML (that also
>> works with their CSS preprocessing tools)
>
> If we can decouple it from the current state of WebGL and GLSL, then I am fine with it.

What aspects of WebGL and GLSL, that you consider to be part of the
current state, might leak?

>> 3. a single name to refer to an effect (the IDENT for the @-rule)
>> instead of both the @-rule IDENT as well as the shader package URL
>
> You need to have the possibility of fallback formats, I don't see this with just providing an IDENT, without a URL.

Yes, you absolutely need a URL. I am concerned that the @-rule and the
shader package URL effectively identify the same thing. There exists a
trivial @-rule that simply embeds the shader package into CSS. Why do
I need both the XML and the trivial injection?

>> 4. trivial generation from XML with XSLT if you desire an XML source format
>
> Indeed.
>
>> 5. no new format semantics questions or hastily designed vocabularies
>> that only solve 50% of the problem and have slow uptake
>
> This won't happen if we have a simple and clean definition of the format.

Buy-in for a new XML format in addition to a CSS module is by
definition equal to or less than buy-in for simply a new CSS module.

The format will have to handle naming, mix-n-match, and multiple
content types. How each of these are supported will require decisions
on the semantics of the XML vocabulary. These issues have already been
resolved for the platform as a whole in the form of URI references,
CSS properties, and media types.

Some people will want to use different root elements and transmit
non-shader data alongside their shaders (e.g. COLLADA).

>> 6. fewer files, less duplication
>
> Ideally yes. This is already a benefit to the new concept in comparison to the old one.

New:

1 XML file, 1 CSS file
XML vocabulary can define default parameters, CSS can override default
parameters
XML vocabulary can pair VS/FS, CSS can't pair VS/FS?

Old:

1 shader file (unless embedded), 1 CSS file
CSS defines default parameters
CSS can pair VS/FS (and only reference 1 shader file using a generic
fragment id)

>> At one time, flexibility was also a goal of the CSS WG by allowing
>> cascading styles at various levels of a page. Only use #1 bundles
>> shader source into the document and should probably only be used with
>> inline styles. This would enable 1 network request for the entire
>> HTML+CSS+WebGLSL which could then stand alone. JavaScript may also be
>> used inline. Without fragment identifier support, I do not see how to
>> supply a single document which uses CSS effects (inline CSS + manifest
>> fetch for every VS/FS pair unless you use data blocks but then why
>> mandate the XML vocab?).
>
> If you worry about this, this is valid as well:
>
> @filter curl {
>         src: url(data:application/xml;base64,iVBORw…) format("glsl");
> }
>
> Which could be done with a minimizer.

Now I'm transmitting base64-encoded text which is:
A. Unreadable
B. Splattered inside my CSS
C. Larger than necessary

Proposed steps to load a custom dynamic filter:
1. Unpack application format
2. Wrap shaders with XML
3. Base64 encode XML
4. Build @filter
---
5. Browser Base64 decodes XML
6. Browser parses XML
7. Browser parses SL

Rather than:
1. Unpack application format
2. Base64 encode shaders
3. Build @filter
---
4. Browser Base64 decodes shader
5. Browser parses shader

Or:
1. Unpack application format
2. Insert shaders into DOM with @id
3. Build @filter
---
4. Browser parses shader

What does wrapping with XML win the developer?

>> Uses #2, #3, and #4 decouple style from structure and do so in a way
>> that gives authors maximum flexibility to decide how they wish to
>> serve their shaders (everything from a single XML bundle with 50
>> shaders and a custom vocab to a single GLSL file with preprocessor
>> switches to a future manifest or module format).
>
> Ok, I got your point. Now I understand your fragment request, in this case we just define multiple shaders in the XML file:
>
> <shader id="curl">
>         <fragment/>
>         <vertex/>
> </shader>
> <shader id="wave">
>         <fragment/>
>         <vertex/>
> </shader>
>
> This sounds reasonable.

What is the root element? What if I only want to use a fragment with
default vertex or vertex with default fragment? What does it mean to
refer to an XML package resource which uses @id without a fragid
specified?

What does using this specific vocabulary allow that using any XML
vocabulary with @xml:id does not?

After all of that, I would like to reiterate that effect formats are
immensely useful and I'm sure the community would appreciate a common
effect format. However, I don't think they should be mandated or
favored and I don't see how they fall under CSS WG's mission.

Warm regards,

David

Received on Wednesday, 5 December 2012 02:25:27 UTC