RE: Properties of Custom Paint

Hi Ian & Domenic,

 

> We agree that it'd be nice you have something that could 

> eventually just run on the GPU. Something like a shader 

> as Domenic suggested.

 

Great :-)

 

 

> You could imagine in the future, being able to write a 

> shader in script (or a subset of javascript like asm.js? :) 

> as a custom paint callback.

> 

> This would require a lot of spec work, and wouldn't build

> on existing primitives like the 2d canvas api.

 

Well, obviously :) 

 

I wasn't arguing for something so radical. I was thinking about WebGL Shaders which are already implemented by browsers and used broadly in the industry.

 

 

> [moving to self-quoting from here]

> The magic about Web Audio is that the whole thing can be 

> described as a graph of operations. [...] In short, I propose 

> to serialize the painting step of the browser in a graph, let 

> a script change the resulting graph, and then unserialize 

> the modified graph and send it to the graphical pipeline.

 

I'm self-quoting here as I think my point wasn't made clear in the previous mail; I'm not much arguing for a specific implementation of the drawing operations themselves (though I believe WebGL Shaders should be the "go-to" solution whenever it is possible) but I was making the comparison with Web Audio mainly for the "graph-of-operations" nature of their technology.

 

 

INITIAL SITUATION FOR THE EXAMPLES:

Here's an example of what a Custom Paint Controller may receive as input graph for an element which displays three lines of text and has an inline-block child:

 



 

 

EXAMPLE 1: TEXT-VISIBILITY

For our first example, let’s imagine we want a custom property (named “text-visibility: visible|hidden”) that controls the rendering of text runs inside the element, the custom painter may remove from the paint graph the nodes about the text run, like this:

 



 

In this case, the only thing that needs to be run on the JS thread is a graph manipulation code that is very cheap. The browser can proceed painting in a very usual way as no non-native operation has been introduced in the graph.

 

 

EXAMPLE 2: TEXT-JITTER

In this second example, we want to add a “jitter” effect to the text. We do this by exposing a “Jitter” WebGL shader, and adding it to the draw pipeline of text run nodes:

 



 

 

EXAMPLE 3: AERO-LIKE EFFECT

As a third example, I think it would be neat to think about how we would implement an Aero-like effect behind a <dialog> element, for instance.

In this case, we would introduce a blur between the blending phase of the element and the underneath content.

 



 

In some cases, just adding the blur may not do the trick (if you element draws outside the box in which you want to enable the aero look, in which case you may need a “custom-blender” which runs JavaScript code and return a WebGL canvas. This is similar to the custom-painter like the one of the next example.

 

 

EXAMPLE 4: PROCEDURAL-TEXTURE BACKGROUNDS

Finally, let’s imagine a case where we want a custom property to indicate a background should be drawn a custom procedural texture. We may achieve this by replacing the background painter by a custom painter using a 2d canvas:

 



 

 

WHY I THINK GRAPHS ARE AWESOME:

Firstly, they allow to express things in a way that enable you to do simple things without taking over the whole painting.

Secondly, they favor composition. It would be a no brainer to run each of the previous examples after each other on the graph generated by the previous custom paint controller. If the first doesn’t hide the text, the second ones appends a shader, the third one setup the aero effect and the last one modifies the background. It’s very hard to face conflicts which would be harder if custom painter had to take over the painting entirely.

 

 

François