[3d-transforms] Behavior of intersecting layers with transform-style: preserve-3d is undefined and inconsistent across implementations

(note: this is an offshoot of the www-style thread "[css-shaders]
subdivision for transparency"
http://lists.w3.org/Archives/Public/www-style/2011Oct/0117.html - please
refer to that thread for the original context)

CSS 3D Transforms defines a transform-style property preserve-3d defines the
behavior of its children to "maintain their position in 3D space".  (I'm
referring to the Editor's Draft
http://dev.w3.org/csswg/css3-3d-transforms/ dated
March 21, 2011).  This property is used extensively in demos - for example
the Poster Circle demo (
http://www.webkit.org/blog-files/3d-transforms/poster-circle.html) uses it -
but the rendering and interactive behavior is not well defined and
implementations today differ considerably in how they interpret this
property.  For example, in this demo made by Gregg there are 3 rotating
intersecting semitransparent <div>s inside a preserve-3d ancestor:

http://greggman.com/downloads/examples/intersecting-elements-3d-css.html

The rendering produced by Safari is:
http://greggman.com/downloads/examples/correct-3d-css-polygon-sorting-subdivisions-safari.png

In Google Chrome:
http://greggman.com/downloads/examples/correct-3d-css-polygon-sorting-subdivisions-safari.png

Recent Firefox nightlies produce a rendering that is closer to Google
Chrome's.  The Safari rendering implies that they are subdividing each layer
into two portions and rendering the rearmost portions first.  A related
behavior is hit testing - in Safari or Google Chrome, the portion of the
<div>s that appear visual closer in the Safari rendering are selectable.  In
Firefox, the hit testing order appears to ignore the preserve-3d style and
is the same regardless of the rotation state.

I think the Safari behavior makes the most sense here for both rendering and
hit testing, but this should be clearly defined in the specification so that
vendors can produce interoperable implementations and developers can know
what to expect.  This would be somewhat novel for CSS in that instead of
having a global paint / hit testing order of CSS boxes we would have an
order of convex polygons within CSS boxes, but there isn't any real
alternative for this feature.  This feature also complicates implementations
and requires more computation effort, but again I don't think there is a
valid alternative.  We also need to define this behavior in relation to the
CSS shaders proposal (
https://dvcs.w3.org/hg/FXTF/raw-file/tip/custom/index.html).  From the
www-style thread it sounds like the consensus there is to treat
transformations within a shader as if that element were isolated in its own
transform-style: flat section, which seems fine at first glance, but it
needs to be clearly defined.

I do not think it is acceptable to leave this behavior
implementation-defined.  Having objects share a 3d space and thus having
their paint order defined by their position in that space is the point of
having transform-style: preserve-3d, if that behavior is undefined then we
might as well not have the feature in the spec at all.

- James

PS: Safari will sometimes allow elements within a transform-style:
preserve-3d subtree to clip elements that are outside that subtree, but I
think that's an implementation bug that they intend to fix and do not think
it should be specified.