Shader unit test framework

[bhouston]

Description

We have many many GLSL functions that use all over the place in our shaders.

These functions can have subtle bugs or problems with their extremes.

We really do not check if these functions operate correctly via unit tests because we do not have unit tests that work with shaders.

Solution

For threeify, I implemented a quick and dirty shader unit test system.

You could write tests in this fashion:

#pragma import "../tests/fragment.glsl"
#pragma import "./normalPacking.glsl"

void testEquivalency(inout TestSuite suite, int testId, vec3 normal) {
  vec3 rgb = normalToRgb(normal);
  vec3 normal2 = rgbToNormal(rgb);
  assert(suite, testId, eqAbs(normal, normal2, 0.0001));
}

void tests(inout TestSuite suite) {
  vec3 px = vec3(1.0, 0.0, 0.0);
  vec3 py = vec3(0.0, 1.0, 0.0);
  vec3 pz = vec3(0.0, 0.0, 1.0);

  testEquivalency(suite, 3, px);
  testEquivalency(suite, 4, -px);
  testEquivalency(suite, 5, py);
  testEquivalency(suite, 6, -py);
  testEquivalency(suite, 7, pz);
  testEquivalency(suite, 8, -pz);

}

Another example here: https://github.com/bhouston/threeify/blob/main/packages/core/src/shaders/math/unitIntervalPacking.test.glsl

And then it would be executed so that each test would write to a pixel in an output texture 0 or 1. And then you could read back that texture and see if all unit tests pass -- all 1s:

  const passMaterial = new ShaderMaterial(
        'index',
        vertexSource,
        glslUnitTest.source
      );
      const unitProgram = await shaderMaterialToProgram(context, passMaterial);

      framebuffer.clear(BufferBit.All);
      renderPass({
        framebuffer,
        program: unitProgram,
        uniforms: unitUniforms
      });

      const result = frameBufferToPixels(framebuffer) as Uint8Array;

      for (let i = 0; i < result.length; i += 4) {
        const runResult = result[i + 2];
        const id = i / 4;
        switch (runResult) {
          case 0:
            failureIds.push(id);
            break;
          case 1:
            passIds.push(id);
            break;
          case 3:
            duplicateIds.push(id);
            break;
        }
      }

Full source code here: https://github.com/bhouston/threeify/blob/main/examples/src/units/glsl/index.ts

Adding this to Three.js would extend the robustness of Three to the shader level. It would also help identify GPU hardware / drivers that are misbehaving in a much faster way than we currently do. Basically you will see specific tests fail for GPU X and know exactly what you have to do to fix it.

Alternatives

Unsure of alternatives, maybe something new with TSL?

Additional context

We identified recently that some of our packing / unpacking code for GLSL is buggy and has been for many years: #28692

[mrdoob]

At this point it would make more sense to do this for TSL instead.