Re: [filter-effects] Request: Support for convolution in CSS Shaders

On 10/16/12 7:49 AM, "Dirk Schulze" <dschulze@adobe.com> wrote:

>
>On Oct 16, 2012, at 7:24 AM, "Dirk Schulze" <dschulze@adobe.com> wrote:
>
>> Hi Roy,
>> 
>> On Oct 11, 2012, at 6:26 PM, Roy Williams <royw@fb.com> wrote:
>> 
>>> Lots of effects like reflections or distortions, etc... require
>>> sampling from different locations in a texture to get the desired
>>>effect.
>>> Effects like Blurs or edge detection require convolution.  When I first
>>> heard about CSS Filter Effects and sat down with the code, I tried to
>>> implement a water ripple originating from where the user clicked.  I
>>>found
>>> I was unable to do so with the current spec, I wasn't able to distort
>>>the element I was styling.
>> Correct, with the current specification effects like distortions are
>>not possible. This is part of the security model. But Filter Effects
>>specifies feGaussianBlur[1] for blurring effects and feConvolveMatrix[2]
>>for convolutions. Some of your requests should easily be possible with
>>these effects already.

Out of curiosity, what is it about distortions that would violate the
security model?  Is it the distortion itself (e.g. Tricking a user to
think they're clicking on button X but they're actually clicking on button
Y) or the method to achieve the distortion (e.g. Sampling)?

>> 
>>> 
>>> Convolution is a superset of modifying the sample location, so I would
>>> propose adding two (maybe 3?) new output parameters from the fragment
>>> shader,
>>> 
>>> vec2[KERNEL_SIZE] css_SampleOffsets (css_SampleLocations?)
>>> and
>>> float[KERNEL_SIZE] css_SampleWeight
>>> (maybe mat4[KERNEL_SIZE] css_SampleColorMatrix as well?)
>>> 
>>> The first, css_SampleOffsets, would contain the offsets (or absolute
>>> locations?) into the texture to sample from. css_SampleColorMatrix
>>>would
>>> have the same functionality as css_ColorMatrix, but applied to each
>>> sample. css_SampleWeights would contain the weight with which each
>>>sample
>>> should contribute to the final color. Finally, css_SampleWeight would
>>>be
>>> used to weigh each sample.
>>> 
>>> The actual fragment shader executing would look something like:
>>> 
>>> void main() {
>>>  shaderMain();
>>>  float totalWeight = 0;
>>>  vec4 sum = vec4(0.);
>>>  for (int i = 0; i < KERNEL_SIZE; ++i) {
>>>    sum += css_SampleColorMatrix[i] *
>>>    texture2D(tex, css_SampleLocations[i]) * css_SampleWeight[i];
>>>    totalWeight += css_SampleWeight[i];
>>>  }
>>>  multColor = css_ColorMatrix * (sum / max(totalWeight, 1.));
>>> }

>> Just for understanding your request. You ask for exposing the current
>>position of the current fragment. You don't want to access the passed
>>texture (which is either the output of a previous effect or the filtered
>>element itself)?
>> 
>> Why does css_SampleOffsets need to be a vector?
>Sorry, to early in the morning. I meant array. We also don't limit the
>access to the current position with v_texCoord.

Sorry if my suggestion was unclear.  I don't want to access the passed
texture or v_texCoord, I want to change where it's sampled from on a per
fragment basis.  I'm suggesting allowing custom filters to do convolution
themselves by setting the convolution matrix on a per fragment basis.  I
might be abusing the concept of convolution, but at it's core I want to be
able to specify an array of offsets to sample from and the weights of
those samples.  css_SampleColorMatrix is probably just a bad idea and
overkill.  If it's a simple convolution like a blur or edge detection,
feConvolveMatrix will work just fine.  If its more complex and required
distortion, e.g. A water ripple or a fish eye effect, then this would
allow it to be done.  For example, I'd like to implement the following:
http://www.iquilezles.org/apps/shadertoy/?p=water, which requires
computing the offset on a per fragment basis.

Put another way, taking a look at:

http://code.google.com/searchframe#OAMlx_jo-ck/src/third_party/WebKit/Sourc
e/WebCore/platform/graphics/filters/CustomFilterValidatedProgram.cpp&exact_
package=chromium&q=css_ColorMatrix&l=167


I want control over css_v_texCorod at a minimum, ideally allowing me to
sample from more than one location and combine those samples in some way.

It might be easier to hack up a diff to Chromium and some samples.

Thanks!

Roy
>
>Greetings,
>Dirk
>
>> You can pass vectors as parameters in your CSS syntax. I assume that
>>css_SampleWeight is the kernel. This can be passed with parameters as
>>well:
>> 
>> custom(url(vertexShader), mix(url(fragmentShader), source-atop normal),
>>kernel mat4(1,0,0,0,  0,1,0,0,  0,0,1,0,  0,0,0,1), kerneSize 16);
>> 
>> But for the example above, feConvolveMatrix would work as well. You
>>could even do it on the element texture, which would not be possible for
>>custom filter effects.
>>> 
>>> This doesn't seem like it would disclose any additional information nor
>>> does it seem like it could be exploited by a timing attack, but it
>>>would
>>> greatly expand the kinds of effects you can achieve with the shader.
>> If the texture passes the requirements of CORS, it should be fine. I
>>need to check this again.
>> 
>> Greetings,
>> Dirk
>> 
>>> 
>>> I'd make css_SampleColorMatrix optional. There are a large number of
>>> effects that would default to the identity matrix, and it would be
>>> wasteful to use all of that additional memory.  If there aren't an
>>>effects
>>> that would need this kind of functionality, I wouldn't bother with it.
>>> 
>>> Thanks,
>>> 
>>> Roy Williams
>>> Software Engineer, Facebook
>>> royw@fb.com
>> 
>> [1] 
>>https://dvcs.w3.org/hg/FXTF/raw-file/tip/filters/index.html#feGaussianBlu
>>rElement
>> [2] 
>>https://dvcs.w3.org/hg/FXTF/raw-file/tip/filters/index.html#feConvolveMat
>>rixElement
>> 
>>

Received on Tuesday, 16 October 2012 19:56:37 UTC