Re: WHLSL Compatibility with HLSL

Hey Myles, thanks for this report! Great to see things laid out.

> Many files in the corpus use HLSL’s syntax for sampler literals, but
those aren’t supported in SPIR-V, so I think we can safely ignore those. I
don’t know what the SPIR-V Cross guys are doing about that.

Just to clarify, by SPIR-V Cross did you mean DXC
<https://github.com/Microsoft/DirectXShaderCompiler/blob/master/docs/SPIR-V.rst#samplers>?
SPIRV-Cross doesn't compile HLSL to SPIR-V, it compiles SPIR-V to HLSL.

On Tue, Nov 27, 2018 at 2:30 PM Myles C. Maxfield <mmaxfield@apple.com>
wrote:

> Over the past few days, I’ve collected a large corpus of HLSL files so we
> can determine what we need to do to be source-compatible with existing HLSL
> source.
>
> *The Corpus*
>
> I wrote a GitHub crawler which looked for repositories that had many HLSL
> files in them. I looked over the results of this crawler and hand-picked a
> few repositories that are from respectable sources. In total, we ended up
> with 2099 HLSL files.
>
> The list of repositories:
>
>    - Microsoft/DirectX-Graphics-Samples
>    - vvvv/vvvv-sdk
>       - Of limited use, because most of the source is written in another
>       language (Effects) which includes HLSL snippets. GitHub classifies this as
>       HLSL.
>    - Unity-Technologies/ScriptableRenderPipeline
>       - Of limited use, because most of the source is written in another
>       language (Shaderlab) which includes HLSL snippets. GitHub classifies this
>       as HLSL.
>    - Microsoft/Windows-universal-samples
>    - OGRECave/ogre
>    - EpicGames/UnrealEngine
>       - Of limited use, because most of the source is written in another
>       language (Unreal Shader Format) which includes HLSL snippets. GitHub
>       classifies this as HLSL.
>    - ConfettiFX/The-Forge
>    - AtomicGameEngine/AtomicGameEngine
>    - NVIDIAGameWorks/D3DSamples
>    - EpicGames/UnrealTournament
>       - Of limited use, because most of the source is written in another
>       language (Unreal Shader Format) which includes HLSL snippets. GitHub
>       classifies this as HLSL.
>    - urho3d/Urho3D
>    - NVIDIAGameWorks/HairWorks
>    - NVIDIAGameWorks/WaveWorks
>       - Of limited use, because most of the source is written in another
>       language (Effects) which includes HLSL snippets. GitHub classifies this as
>       HLSL.
>    - NVIDIAGameWorks/FleX
>    - Unity-Technologies/PostProcessing
>       - Of limited use, because most of the source is written in another
>       language (Shaderlab) which includes HLSL snippets. GitHub classifies this
>       as HLSL.
>    - NVIDIAGameWorks/Falcor
>    - NVIDIAGameWorks/FaceWorks
>    - NVIDIAGameWorks/HBAOPlus
>    - GPUOpen-LibrariesAndSDKs/GPUParticles11
>    - NVIDIAGameWorks/VolumetricLighting
>    - GPUOpen-Effects/ShadowFX
>    - GPUOpen-LibrariesAndSDKs/LiquidVR
>    - NVIDIAGameWorks/NvCloth
>    - GPUOpen-Effects/DepthOfFieldFX
>    - NVIDIAGameWorks/PhysX-3.4
>    - GPUOpen-Effects/GeometryFX
>    - GPUOpen-LibrariesAndSDKs/TiledLighting11
>    - NVIDIAGameWorks/Flow
>    - GPUOpen-LibrariesAndSDKs/ForwardPlus11
>    - Microsoft/Win2D
>    - GPUOpen-LibrariesAndSDKs/SSAA11
>    - Microsoft/Win2D-Samples
>    - PixarAnimationStudios/OpenSubdiv
>
> We could potentially figure out how to compile Effects, Shaderlab and
> Unreal Shader Format to HLSL (because that’s what their engines do). If we
> did this, we could grow the repository by 13% + 8% + 15% (respectively) =
> 36%. I didn’t want to get bogged down doing this, though.
>
> *Preprocessor*
>
> HLSL Source files make heavy use of the preprocessor. Each file includes
> an average of 9.61 uses of the preprocessor (lines that begin with “#”) and
> the preprocessor is used on average every 11.68 lines.
>
>
> As you can see above, most of the users of the preprocessor are not to
> include files, but are instead to enable / disable features. Therefore,
> this is a situation where compatibility with existing HLSL source is
> directly in conflict with simplicity of the language.
>
> I proceeded by running the corpus through the Microsoft HLSL preprocessor,
> and investigated the preprocessed files. My analysis is just based on the
> parsing stage of the language, not name resolution or type checking.
> Out-of-the-box, we parse 5.9% of the corpus.
>
> *Language Features*
>
> From investigating the source, I found some language features that HLSL
> depends on.
>
> In MSL, if you want to pass some data to your shader, you make a struct,
> and pass a reference to that struct as an argument of the main function.
> Then, in the main function, you reference the data by saying
> theReference.field. This approach is possible in HLSL, but there’s another
> more common way to do it. Instead of making a struct, you make a “cbuffer”
> which lists a set of fields, but those fields are treated as global
> variables. The cbuffer is given a “semantic” so the API can attach memory
> to back the cbuffer.
>
> cbuffer Camera : register(b0) // The API assigns memory to this block by
> using the “b0” handle
> {
>     float4x4 viewProjection;
>     float4x4 projectionInv;
>     float3 viewPos;
> };
>
> Output main() {
>     output.foo = viewProjection; // viewProjection, projectionInv, and
> viewPos are in the global scope.
>     return output;
> }
>
> About 1/3 of the files in the corpus use cbuffers.
>
> HLSL has two flavors of global variables:
>
>    1. Resources, like RWTexture2D<float2> dstTexture : register(u0);.
>    These work just like entry point parameters, except they are in the global
>    scope and therefore can be accessed from any function, without passing
>    around a pointer to them.
>    2. Literal data, like static const float convolutionWeights[] = {1, 2,
>    3};.
>
>
> About 1/5 of the files in the corpus use global variables.
>
> HLSL supports default arguments in function parameters and cbuffers, so
> you can say void foo(int x = 3);. I would imagine specifying this would
> be tricky because we have to mention which variables and functions the
> initial value can refer to.
>
> Many files in the corpus use HLSL’s syntax for sampler literals, but those
> aren’t supported in SPIR-V, so I think we can safely ignore those. I don’t
> know what the SPIR-V Cross guys are doing about that.
>
> *New Syntax*
>
> There are lots of changes to the syntax of the language that shouldn’t
> have much of an effect on the language itself, but are required if we want
> to claim compatibility with lots of HLSL sources.
>
>
>    - Removing the entry point keywords (vertex, fragment, compute) is a
>    requirement for any shader to compile. Instead, we should require that
>    compilation of a WHLSL file state which function names are the entry points.
>    - It appears that HLSL allows any arbitrary semantic for stage in/out
>    parameters. Around 30% of the corpus uses a semantic that WHLSL doesn’t
>    currently accept.
>    - Some functions in the HLSL standard library use
>    member-function-syntax, like texture.Sample(sampler, location) instead
>    of Sample(texture, sampler, location).
>    - There are a whole collection of variable modifiers that HLSL sources
>    use that WHLSL doesn’t accept. E.g. row_major float4x4 mvpMatrix;
>    - HLSL has a few function modifiers like [ RootSignature(…stuff goes
>    here…) ] void foo(…) { … } that are irrelevant for WebGPU. This
>    includes information about the D3D root signature, but also things like how
>    geometry shaders and tessellation work, which WebGPU doesn’t have.
>    - HLSL arrays put the brackets after the variable name, like float
>    myArray[40];
>    - Arrays and structs can be initialized using brackets, like float
>    myArray[3] = { 1.0, 2.0, 3.0 };
>    - HLSL has some hints about how the compiler should treat branches and
>    loops. They look like [ unroll ] for (…) and [ branch ] if ( …). I
>    don’t think these have semantic meaning so we can probably just swallow
>    them.
>    - HLSL uses C-style casts instead of C++-style casts. So, we need to
>    support (float)x instead of float(x).
>    - HLSL accepts float literals with exponents, like 1e-3.
>    - Functions can be forward-declared in HLSL.
>
>
> After doing all that, we get up to around 90% compatibility with parsing
> (not resolving names nor type checking) the HLSL corpus. The biggest wins
> are member-function syntax, allowing every semantic name, C-style casting,
> and C-style array syntax.
>
> —Myles
>