HLSL geom shader fails to compile, then crashes glslangValidator in linker pass

[hrydgard]

I'm trying to create a shader pipeline where I start out with HLSL, run it through glslang to obtain SPIR-V, then use SPIRV-Cross to generate shaders for the non-D3D platforms I want to support (initially GLSL and Vulkan/SPIR-V, maybe later Metal etc). However, my geometry shader fails to compile, and then causes a crash in the linker. It crashes here, currently line 599 in linkValidate.cpp, because treeRoot is nullptr:

TIntermSequence& globals = treeRoot->getAsAggregate()->getSequence();

Here are the error messages that the compiler produces:

ERROR: draw2d.gs:8: ')' : Expected
ERROR: draw2d.gs:8: ';' : Expected
draw2d.gs(8): error at column 17, HLSL parsing failed.
ERROR: 3 compilation errors.  No code generated.

And here's the HLSL shader (D3DCompile() happily chews it up):

struct VertexData {
	float4 position : POSITION;
	float4 color : COLOR0;
	float2 uv : TEXCOORD0;
} vin;

[maxvertexcount(4)]
void gshader(line VertexData vin[2], inout TriangleStream<VertexData> outStream) {
	VertexData vout;

	vout.color = vin[0].color;
	vout.uv = vin[0].uv;
	vout.position = vin[0].position;
	outStream.Append(vout);

	vout.color = vin[1].color;
	vout.uv = float2(vin[0].uv.x, vin[1].uv.y);
	vout.position = float4(vin[0].position.x, vin[1].position.yzw);
	outStream.Append(vout);

	vout.color = vin[0].color;
	vout.uv = float2(vin[1].uv.x, vin[0].uv.y);
	vout.position = float4(vin[1].position.x, vin[0].position.y, vin[1].position.zw);
	outStream.Append(vout);

	vout.color = vin[1].color;
	vout.uv = vin[1].uv;
	vout.position = vin[1].position;
	outStream.Append(vout);
}

[ghost]

@hrydgard - there is a WIP PR #599 to add GS support. It will not compile your shader above verbatim, but not because of the GS features. Currently there is a limitation related to flattening arrayed struct inputs - it's a separate issue. If I change the shader to use non-struct inputs like so, with corresponding changes to the references:

[maxvertexcount(4)]
void gshader(line float4 position[2],
             line float4 color[2],
             line float2 uv[2],
             inout TriangleStream<VertexData> outStream) {

Then it compiles (with #599).

[hrydgard]

@steve-lunarg That's great! Could easily live with that limitation for now.

[johnkslang]

@hrydgard This is merged now. If it's acceptable, please close, or make a more specific issue. Thanks.

[hrydgard]

Great! I'm happy to close this with that one merged.

As the missing struct array input flattening is a known problem, I don't know if it makes sense for me to create an issue about it.

[hrydgard]

Though, if there are other ways compilation can fail in a way that leaves treeRoot a nullptr, crashing the "linker" step, that might be worth fixing too.

[ghost]

Hi Hrydgard - yeah, I've noticed that same treeRoot crash in other situations. Somewhere, I have some local code that prevents it for all potential causes of a null value. I'll try to hunt that down and submit a PR for it.

I'm going to work on the flattening of struct arrays soon.

[johnkslang]

The treeRoot fix is in #577.