Re: [filter-effects] Request: Support for convolution in CSS Shaders

Hi Roy,

On Oct 11, 2012, at 6:26 PM, Roy Williams <royw@fb.com> wrote:

> Lots of effects like reflections or distortions, etc... require
> sampling from different locations in a texture to get the desired effect.
> Effects like Blurs or edge detection require convolution.  When I first
> heard about CSS Filter Effects and sat down with the code, I tried to
> implement a water ripple originating from where the user clicked.  I found
> I was unable to do so with the current spec, I wasn't able to distort the element I was styling.
Correct, with the current specification effects like distortions are not possible. This is part of the security model. But Filter Effects specifies feGaussianBlur[1] for blurring effects and feConvolveMatrix[2] for convolutions. Some of your requests should easily be possible with these effects already.

> 
> Convolution is a superset of modifying the sample location, so I would
> propose adding two (maybe 3?) new output parameters from the fragment
> shader,
> 
> vec2[KERNEL_SIZE] css_SampleOffsets (css_SampleLocations?)
> and
> float[KERNEL_SIZE] css_SampleWeight
> (maybe mat4[KERNEL_SIZE] css_SampleColorMatrix as well?)
> 
> The first, css_SampleOffsets, would contain the offsets (or absolute
> locations?) into the texture to sample from. css_SampleColorMatrix would
> have the same functionality as css_ColorMatrix, but applied to each
> sample. css_SampleWeights would contain the weight with which each sample
> should contribute to the final color. Finally, css_SampleWeight would be
> used to weigh each sample.
> 
> The actual fragment shader executing would look something like:
> 
> void main() {
>   shaderMain();
>   float totalWeight = 0;
>   vec4 sum = vec4(0.);
>   for (int i = 0; i < KERNEL_SIZE; ++i) {
>     sum += css_SampleColorMatrix[i] *
>     texture2D(tex, css_SampleLocations[i]) * css_SampleWeight[i];
>     totalWeight += css_SampleWeight[i];
>   }
>   multColor = css_ColorMatrix * (sum / max(totalWeight, 1.));
> }
Just for understanding your request. You ask for exposing the current position of the current fragment. You don't want to access the passed texture (which is either the output of a previous effect or the filtered element itself)?

Why does css_SampleOffsets need to be a vector? You can pass vectors as parameters in your CSS syntax. I assume that css_SampleWeight is the kernel. This can be passed with parameters as well:

custom(url(vertexShader), mix(url(fragmentShader), source-atop normal), kernel mat4(1,0,0,0,  0,1,0,0,  0,0,1,0,  0,0,0,1), kerneSize 16);

But for the example above, feConvolveMatrix would work as well. You could even do it on the element texture, which would not be possible for custom filter effects.
> 
> This doesn't seem like it would disclose any additional information nor
> does it seem like it could be exploited by a timing attack, but it would
> greatly expand the kinds of effects you can achieve with the shader.
If the texture passes the requirements of CORS, it should be fine. I need to check this again.

Greetings,
Dirk

> 
> I'd make css_SampleColorMatrix optional. There are a large number of
> effects that would default to the identity matrix, and it would be
> wasteful to use all of that additional memory.  If there aren't an effects
> that would need this kind of functionality, I wouldn't bother with it.
> 
> Thanks,
> 
> Roy Williams
> Software Engineer, Facebook
> royw@fb.com

[1] https://dvcs.w3.org/hg/FXTF/raw-file/tip/filters/index.html#feGaussianBlurElement
[2] https://dvcs.w3.org/hg/FXTF/raw-file/tip/filters/index.html#feConvolveMatrixElement

Received on Tuesday, 16 October 2012 14:24:28 UTC