[Fwd: Pre-multiplication of samples in Filter Effects]

These are good comments. The general rule is that for things that  
manipulate color independent of coverage, premultiplication is not  
convenient.  But for things that manipulate the image as a whole,  
premultiplication is more natural.  For example, I disagree with the  
analysis of the GaussianBlur case.  To correctly evaluate a  
convolution through a mask or alpha channel, I would argue that it is  
more natural to weight the contribution of each pixel by its mask  
component, and then renormalize the result by the sum of the weights.  
 This is much more complex in the non-premultiplied case.  The  
reason for this is that the alpha channel says how much of the pixel  
is "there", so any contribution to a filter or resampling kernel  
should be appropriately weighted.

Consider this example.  I have a large non-premultiplied RGBA image  
where alpha is zero everywhere except for a smaller rectangle in the  
middle of the image where alpha is one.  The color pixels are white  
in the rectangle and pink outside.  Normally when I "use" this image  
by compositing it over a background I get a white rectangle -- the  
pink pixels are not seen since alpha is zero.  In other words the  
pink pixels "don't contribute" to the image in normal useage.  Now  
suppose I blur the image.  If I don't "weight" the contribution of  
each pixel by its alpha channel, I will get pink in the resulting  
image, which arguably quite unexpected.  If I "weight" the  
contributions by the alpha channel, no pink color will contribute to  
the result which is more natural.  If the original image were  
premultiplied, I can just blur the 4 channels independently and  
everything works as expected.  These considerations also apply when  
resampling an RGBA image while scaling - if I scale up the above  
image of a white rectangle with a bicubic resampling, I certainly  
would not expect to see pink pixels in the result.

Hope this helps.

For a good reference see the article on "Compositing theory and  
practice" in  Jim Blinn's book, "Dirty Pixels".

Peter

Begin forwarded message:

> From: Chris Lilley <chris@w3.org>
> Date: 1999-06-02 18:29:26 +0200
> To: w3c-svg-wg@w3.org
> Subject: [Fwd: Pre-multiplication of samples in Filter Effects]
> X-Loop: w3c-svg-wg@w3.org
> Resent-sender: w3c-svg-wg-request@w3.org
> X-Mailer: Mozilla 4.6 [en] (WinNT; I)
> X-Mailing-List: <w3c-svg-wg@w3.org> archive/latest/732
> Resent-date: Wed, 02 Jun 1999 12:38:13 -0400 (EDT)
> Resent-from: w3c-svg-wg@w3.org
> X-Accept-Language: en
>
> This just turned up on the public (www-svg) list. I'd like to add a 
> discussion of the issues to the Thursday telcon.
>
> --
> ChrisFrom: Christian Brunschen <cb@df.lth.se>
> Date: Wed, 2 Jun 1999 16:01:25 +0200 (CEST)
> To: www-svg@w3.org
> Subject: Pre-multiplication of samples in Filter Effects
> Resent-Sender: www-svg-request@w3.org
> Resent-Date: Wed, 2 Jun 1999 10:01:31 -0400 (EDT)
> X-Mozilla-Status2: 00000000
> X-Mailing-List: <www-svg@w3.org> archive/latest/136
> X-Loop: www-svg@w3.org
> Resent-From: www-svg@w3.org
>
>
> Greetings.
>
> Having just read the latest draft SVG spec, I cannot help but  
wonder one
> thing.
>
> In section 15.5.4, the built-in Filter Effects are listed, with this 
> little paragraph as a lead-in:
>
> ``The following is a catalog of the individual processing nodes  
proposed.
> All filters operate on linear premultiplied RGBA samples. Filters which 
> work more naturally on non premultiplied data (ColorMatrix and
> ComponentTransfer) will temporarily undo and redo premultiplication as 
> specified.''
>
> Since RGBA rasters can not store sample values with infinite  
resolution,
> the combination of premultipluying and the un-premultiplying pixels is 
> almost guaranteed to introduce inaccuracies at some places. Assuming an 
> 8-bit-per-sample representation, a simple example is one where the 
> original pixel is painted with an RGBA value of 3,5,7,127 - a very dark 
> shade, with a 50% coverage. The resulting pre-multiplied pixel would 
> contain values something like 1,2,3,127 - and un-pre-multiplying would 
> result in 2,4,6,127, with each of the RGB samples being markedly  
different
> from the original. Basically, pre-multiplication guarantees a loss of 
> information.
>
> I would further like to suggest that pre-multiplied alpha is not  
the best
> choice for the default alpha handling for, for instance, the
> 'DisplacementMap' Filter:
>
> -- begin quote --
>
> NodeType         DisplacementMap
>
> Image Inputs     2
>
> Attributes       scale
>                  x-channel-selector 	one of R,G,B or A.
>                  y-channel-selector 	one of R,G,B or A.
>
> Description      Uses Input2 to spatially displace Input1,
>                  similar to the Photoshop displacement filter. This is 
>                  the transformation to be performed:
>
>                   P'(x,y) <- P( x + scale * XC(x,y), y + scale *  
YC(x,y) )
>
>                  where P(x,y) is the source image, Input1, and  
P'(x,y) is
>                  the destination. XC(x,y) and YC(x,y) are the component 
>                  values of the designated by the x-channel-selector and 
>                  y-channel-selector. For example, to use the R  
component
>                  of Image2 to control displacement in x and the G  
component
>                  of Image2 to control displacement in y, set
>                  x-channel-selector to "R" and y-channel-selector  
to "G".
>
> Comments         The displacement map defines the inverse of the  
mapping
>                  performed.
>
> Implementation
> issues           This filter can have arbitrary non-localized
>                  effect on the input which might require substantial 
>                  buffering in the processing pipeline. However  
with this
>                  formulation, any intermediate buffering needs can be 
>                  determined by scale which represents the maximum
>                  displacement in either x or y.
>
> -- end quote --
>
> If I want to displace something according to the 'R' and 'G'  
channels of
> an image I have constructed, it is not necessarily obvious that I  
want the
> pre-multiplied-alpha Red and Green channels.
>
> Furthermore, if I definitely want to use the  
_non_premultiplied-alpha Red
> and Green channels from my image, I cannot get at them easily.
>
>
> The lead-in paragraph states that ``Filters which
> work more naturally on non premultiplied data (ColorMatrix and
> ComponentTransfer) will temporarily undo and redo premultiplication as 
> specified'' which appears based in the assumption that for _all other 
> filters_, pre-multiplied-alpha is the `more natural' state of  
affairs. So
> let's take a look at the different filters defined (these are in my 
> personal opinion, of course):
>
> Filter	            'Most Natural'          Comment
>
> ColorMatrix         non-premultiplied       explicitly stated in the 
>                                             description
>
> Color               N/A                     has no inputs
>
> ComponentTransfer   non-premultiplied	    explicitly stated in the 
>                                             description
>
> Composite           premultiplied?          could easily be written 
>                                             to handle non-premultiplied 
>                                             while maintaining fidelity 
>                                             of output; or could be  
stated
>                                             to require premultiplied 
>                                             alpha
>
> DiffuseLighting     N/A                     uses only the Alpha  
channel,
>                                             does not care about  
wether RGB
>                                             are premultiplied or not 
>
> DisplacementMap     I'm not sure            As I discussed above,  
I don't
>                     but it feels like       see how premultiplied  
could be
>                     non-premultiplied       seen as 'more natural' here 
>                     would be better anyway
>
> GaussianBlur        non-premultiplied       for maximum fidelity in the 
>                                             resulting blurred image 
>
> Image               N/A                     has no inputs
>
> Merge               premultiplied?          premultiplied preferred for 
>                                             performance reasons,  
but could
>                                             use non-premultiplied  
with the
>                                             same fidelity (like  
Composite)
>
> Morphology          non-premultiplied       to ensure fidelity in the 
>                                             resulting image
>
> Offset              N/A                     doesn't care about the  
colors
>                                             of the pixels it shifts 
>
> SpecularLighting    N/A                     only uses the Alpha channel 
>
> Tile                N/A                     doesn't care
>
> Turbulence          non-premultiplied       to ensure fidelity, or else 
>                                             the values to be added also 
>                                             have to be  
premultiplied, lest
>                                             non-opaque pixels  
receive an
>                                             unproportionally large  
change
>
>
> So, in summary, of the 14 filters in the spec, 2 _definitely_ need 
> non-premultiplied Alpha (already in the spec), 4 more seem to me to 
> probably be better off with non-premultiplied, 6 simply don't care, and 
> only 2 can really make a demand on premultiplied - Composite and  
Merge. So
> I would interpret this to mean that non-premultiplied would be overall 
> 'more natural' than premultiplied.
>
> Again, this is all in my own opinion.
>
>
> Anyway, my main argument for _not_ premultiplying RGN with A is that of 
> maintaining fidelity, of not losing resolution in the RGB channels just 
> because they happen to have a not-fully-opaque A value.
>
> The main argument for premultiplied Alpha would be performance,  
unless I
> am mistaken: Compositing an image with pre-multiplied Alpha onto  
another
> saves one multiplication per pixel over Compositing the same image 
> non-premultiplied.
>
> However, consider the case of a graphic that gets offset, a ColorMatrix 
> filter applied, then offset again (stupid example, but bear with me): 
>
> Premultiplied Alpha
>
> *  Graphic renders to RGBA matrix : premultiplications take place here 
> *  Offset: shifts pixels around
> *  ColorMatrix: _Un_premultiply, perform the filter, then  
_Re_premultiply
> *  Offset: shift pixels around
> *  Merge: composite to destination
>
> Multiplications take place in three places: Once when the graphic is 
> rendered to the RGBA matrix, then when Un-premultiplying before the 
> ColorMatrix filter can do its job, then re-premultiplying afterwards. 
>
>
> Now, with non-premultiplied Alpha:
>
> *  Graphic renders to RGBA matrix
> *  Offset: shifts pixels around
> *  ColorMatrix: perform the filter
> *  Offset: shift pixels around
> *  Merge: multiply RGB with A, then composite to destination
>
> There's only one place where RGB samples are multiplied with A,  
and that's
> at the very end when the resulting image is to be composited onto the 
> background.
>
>
> Note also that in the non-premultiplied case, the ColorMatrix  
filter can
> access the actual RGB values rendered by the Graphic, rather than the 
> 'pre-multiplied, then un-pre-multiplied' values offered with the  
current
> scheme.
>
>
> For those cases where a certain image will be used pre-multiplied more 
> than once, it would be easy to define a filter which premultiplies RGB 
> with A on an image:
>
> -- begin suggestion --
>
> NodeType         PreMultiply
>
> Image Inputs     1
>
> Attributes       none
>
> Description      multiplies each of the R, G and B channels with the A 
>                  channel
>
> Comments         a very common operation
>
> Implementation
> issues           none, really
>
> -- end suggestion --
>
> which could be used whenever needed, for whatever reason.
>
>
> So, in summary, I am questioning the use of premultiplied alpha as the 
> default representation for RGBA images in filter effects.
>
>
> Best regards
>
> // Christian Brunschen
>

Received on Wednesday, 2 June 1999 14:09:50 UTC