Re: [filter-effects] Request: Support for convolution in CSS Shaders

On Oct 16, 2012, at 7:24 AM, "Dirk Schulze" <dschulze@adobe.com> wrote:

> Hi Roy,
> 
> On Oct 11, 2012, at 6:26 PM, Roy Williams <royw@fb.com> wrote:
> 
>> Lots of effects like reflections or distortions, etc... require
>> sampling from different locations in a texture to get the desired effect.
>> Effects like Blurs or edge detection require convolution.  When I first
>> heard about CSS Filter Effects and sat down with the code, I tried to
>> implement a water ripple originating from where the user clicked.  I found
>> I was unable to do so with the current spec, I wasn't able to distort the element I was styling.
> Correct, with the current specification effects like distortions are not possible. This is part of the security model. But Filter Effects specifies feGaussianBlur[1] for blurring effects and feConvolveMatrix[2] for convolutions. Some of your requests should easily be possible with these effects already.
> 
>> 
>> Convolution is a superset of modifying the sample location, so I would
>> propose adding two (maybe 3?) new output parameters from the fragment
>> shader,
>> 
>> vec2[KERNEL_SIZE] css_SampleOffsets (css_SampleLocations?)
>> and
>> float[KERNEL_SIZE] css_SampleWeight
>> (maybe mat4[KERNEL_SIZE] css_SampleColorMatrix as well?)
>> 
>> The first, css_SampleOffsets, would contain the offsets (or absolute
>> locations?) into the texture to sample from. css_SampleColorMatrix would
>> have the same functionality as css_ColorMatrix, but applied to each
>> sample. css_SampleWeights would contain the weight with which each sample
>> should contribute to the final color. Finally, css_SampleWeight would be
>> used to weigh each sample.
>> 
>> The actual fragment shader executing would look something like:
>> 
>> void main() {
>>  shaderMain();
>>  float totalWeight = 0;
>>  vec4 sum = vec4(0.);
>>  for (int i = 0; i < KERNEL_SIZE; ++i) {
>>    sum += css_SampleColorMatrix[i] *
>>    texture2D(tex, css_SampleLocations[i]) * css_SampleWeight[i];
>>    totalWeight += css_SampleWeight[i];
>>  }
>>  multColor = css_ColorMatrix * (sum / max(totalWeight, 1.));
>> }
> Just for understanding your request. You ask for exposing the current position of the current fragment. You don't want to access the passed texture (which is either the output of a previous effect or the filtered element itself)?
> 
> Why does css_SampleOffsets need to be a vector?
Sorry, to early in the morning. I meant array. We also don't limit the access to the current position with v_texCoord.

Greetings,
Dirk

> You can pass vectors as parameters in your CSS syntax. I assume that css_SampleWeight is the kernel. This can be passed with parameters as well:
> 
> custom(url(vertexShader), mix(url(fragmentShader), source-atop normal), kernel mat4(1,0,0,0,  0,1,0,0,  0,0,1,0,  0,0,0,1), kerneSize 16);
> 
> But for the example above, feConvolveMatrix would work as well. You could even do it on the element texture, which would not be possible for custom filter effects.
>> 
>> This doesn't seem like it would disclose any additional information nor
>> does it seem like it could be exploited by a timing attack, but it would
>> greatly expand the kinds of effects you can achieve with the shader.
> If the texture passes the requirements of CORS, it should be fine. I need to check this again.
> 
> Greetings,
> Dirk
> 
>> 
>> I'd make css_SampleColorMatrix optional. There are a large number of
>> effects that would default to the identity matrix, and it would be
>> wasteful to use all of that additional memory.  If there aren't an effects
>> that would need this kind of functionality, I wouldn't bother with it.
>> 
>> Thanks,
>> 
>> Roy Williams
>> Software Engineer, Facebook
>> royw@fb.com
> 
> [1] https://dvcs.w3.org/hg/FXTF/raw-file/tip/filters/index.html#feGaussianBlurElement
> [2] https://dvcs.w3.org/hg/FXTF/raw-file/tip/filters/index.html#feConvolveMatrixElement
> 
>

Received on Tuesday, 16 October 2012 14:50:09 UTC