HLSL to GLSL: Parsing Shader Dissasembly

http://www.sebbylive.com/2011/10/02/hlsl-to-glsl-parsing-shader-dissasembly

Finally got around to writing my first in-depth post about my HLSL to GLSL translation work. In this first post, I go over some of the basics and the generation of a grammar that will take basic dissasembled HLSL into a more usable binary representation. To serve as an example, here is a basic HLSL sample taken from the DirectX SDK...

//--------------------------------------------------------------------------------------
// File: BasicHLSL.fx
//
// The effect file for the BasicHLSL sample.
//
// Copyright (c) Microsoft Corporation. All rights reserved.
//--------------------------------------------------------------------------------------

//--------------------------------------------------------------------------------------
// Global variables
//--------------------------------------------------------------------------------------
float4 g_MaterialAmbientColor;      // Material's ambient color
float4 g_MaterialDiffuseColor;      // Material's diffuse color
int g_nNumLights;

float3 g_LightDir[3];               // Light's direction in world space
float4 g_LightDiffuse[3];           // Light's diffuse color
float4 g_LightAmbient;              // Light's ambient color

texture g_MeshTexture;              // Color texture for mesh

float    g_fTime;                   // App's time in seconds
float4x4 g_mWorld;                  // World matrix for object
float4x4 g_mWorldViewProjection;    // World * View * Projection matrix

//--------------------------------------------------------------------------------------
// Texture samplers
//--------------------------------------------------------------------------------------
sampler MeshTextureSampler =
sampler_state
{
    Texture = <g_MeshTexture>;
    MipFilter = LINEAR;
    MinFilter = LINEAR;
    MagFilter = LINEAR;
};

//--------------------------------------------------------------------------------------
// Vertex shader output structure
//--------------------------------------------------------------------------------------
struct VS_OUTPUT
{
    float4 Position   : POSITION;   // vertex position
    float4 Diffuse    : COLOR0;     // vertex diffuse color (note that COLOR0 is clamped from 0..1)
    float2 TextureUV  : TEXCOORD0;  // vertex texture coords
};

//--------------------------------------------------------------------------------------
// This shader computes standard transform and lighting
//--------------------------------------------------------------------------------------
VS_OUTPUT RenderSceneVS( float4 vPos : POSITION,
                         float3 vNormal : NORMAL,
                         float2 vTexCoord0 : TEXCOORD0,
                         uniform int nNumLights,
                         uniform bool bTexture,
                         uniform bool bAnimate )
{
    VS_OUTPUT Output;
    float3 vNormalWorldSpace;
	
    float4 vAnimatedPos = vPos;
	
    // Animation the vertex based on time and the vertex's object space position
    if( bAnimate )
		vAnimatedPos += float4(vNormal, 0) * (sin(g_fTime+5.5)+0.5)*5;
		
    // Transform the position from object space to homogeneous projection space
    Output.Position = mul(vAnimatedPos, g_mWorldViewProjection);
	
    // Transform the normal from object space to world space
    vNormalWorldSpace = normalize(mul(vNormal, (float3x3)g_mWorld)); // normal (world space)
	
    // Compute simple directional lighting equation
    float3 vTotalLightDiffuse = float3(0,0,0);
    for(int i=0; i<nNumLights; i++ )
        vTotalLightDiffuse += g_LightDiffuse[i] * max(0,dot(vNormalWorldSpace, g_LightDir[i]));
		
    Output.Diffuse.rgb = g_MaterialDiffuseColor * vTotalLightDiffuse +
                         g_MaterialAmbientColor * g_LightAmbient;
    Output.Diffuse.a = 1.0f; 
	
    // Just copy the texture coordinate through
    if( bTexture )
        Output.TextureUV = vTexCoord0;
    else
        Output.TextureUV = 0; 
		
    return Output;
}

//--------------------------------------------------------------------------------------
// Pixel shader output structure
//--------------------------------------------------------------------------------------
struct PS_OUTPUT
{
    float4 RGBColor : COLOR0;  // Pixel color
};

//--------------------------------------------------------------------------------------
// This shader outputs the pixel's color by modulating the texture's
//       color with diffuse material color
//--------------------------------------------------------------------------------------
PS_OUTPUT RenderScenePS( VS_OUTPUT In,
                         uniform bool bTexture )
{
    PS_OUTPUT Output;
	
    // Lookup mesh texture and modulate it with diffuse
    if( bTexture )
        Output.RGBColor = tex2D(MeshTextureSampler, In.TextureUV) * In.Diffuse;
    else
        Output.RGBColor = In.Diffuse;
		
    return Output;
}

Since several of the uniform constants, are not set, the shader does simplify somewhat and the generated dissasembly is included below.

The actual structure of the dissasembly file is not documented although the DirectX documentation does include most of the documentation for the meaning of the ASM instructions. However, the structure of the assembly file is straightforward and was easy to reverse engineer.

Using C#, I've decided to settle on Antlr as my parser generator. Antly can generate grammars and lexers in different languages including C#. Although Antlr's syntax deviates slightly from the standard Lexx/Yacc grammar, it is still in a similar BNF form along with additional annotations which can serve as syntaxic and semantic predicated. I decided to include the grammar for reference but since it is fairly lengthy, I've included it at the end of the post to avoid clutter.

With Antlr, I am not translating on the fly but rather using a AST grammar to generate a syntax tree which in essence is a binary representation of the source code which is obviously easier to handle in code than the text form. This tree is further processed into a usable form (next post) and is then translated into GLSL form.

Here is a snapshot of what the tree looks like from within my test application.

http://www.sebbylive.com/wp-content/uploads/2011/10/sm3AST.png

The generated AST is still fairly general with items such as register references being leaf nodes of the tree. To make the translation process less cumbersome, I decided to add another processing step before the translation occurs. This step takes the general AST and compacts it into a more specialized tree where each node is represented by a class for each type of operation, this in turns makes the translation process less centralized and easier to approach. The creation of the compact tree will be the topic of my next post...

Finally, as promised here is the grammar I use so far to process the shader dissasembly into an AST. Note that there is some Shader Model 4/5 constructs in the grammar as I am working on adding support for DX10 shaders in my tool. But as of right now, only Shader Model 3 support fully works.

PreviousPixel ShaderNextGrayscale

Last updated

Was this helpful?