Project 5: Wenli Zhao

.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+    "git.ignoreLimitWarning": true
+}

README.md

@@ -3,25 +3,51 @@ WebGL Clustered Deferred and Forward+ Shading
 
 **University of Pennsylvania, CIS 565: GPU Programming and Architecture, Project 5**
 
-* (TODO) YOUR NAME HERE
-* Tested on: (TODO) **Google Chrome 222.2** on
-  Windows 22, i7-2222 @ 2.22GHz 22GB, GTX 222 222MB (Moore 2222 Lab)
+* Wenli Zhao
+* Tested on: **Google Chrome**
+  Mac OS X El Capitan 10.11.6 , 2.6 GHz Intel Core i5, Intel Iris 1536 MB
 
 ### Live Online
 
+Coming soon. I will put this online when I work out some bugs!
+
 [![](img/thumb.png)](http://TODO.github.io/Project5B-WebGL-Deferred-Shading)
 
 ### Demo Video/GIF
 
 [![](img/video.png)](TODO)
 
-### (TODO: Your README)
+Also coming soon. In the meantime, here is an image of a slightly buggy clustered forward+ renderer:
+
+![](img/image2.png)
+
+### ReadMe
+The goal of this project was to implement a clustered forward + and clustered deferred renderer.
+
+Clustered forward+ is a rendering technique that divides the scene visible to the camera into "clusters". Imagine chopping up the camera frustum into smaller pieces. When rendering the scene, rather than checking against all the lights, we only check against the lights influencing the cluster that the geometry is in. This leads to huge performance gains when there are lots of lights in the scene.
+
+Clustered deferred shading also uses the same cluster technique, but uses two passes for shading. In this implementation, it goes through two fragment shaders. The first stores relevant information about the fragment in a g-buffer, which is actually just a texture. The second shader reads from the g-buffer and uses the attributes to actually calculate the light and shading. 
+
+I implemented both the clustered forward+ and the clustered deferred renderer with blinn phong shading. However, I was not able to properly debug the clustered forward+, which is why the images of tile-outlines. Additionally, after adding the blinn-phong shading to the clustered deferred, it stopped working. So to demonstrate that I properly implemented the shading, I included a screenshot of the blinn-phong shading applied to clustered forward+, even though it doesn't look that pretty.
+
+![](img/blinn.png)
+
+### Analysis
+
+![](img/barchart.png)
+
+<img src="/img/chart1.png" alt="test image size" height="50%" width="50%">
+
+After 500 lights, the normal forward rendering stopped running on my machine. Unfortunately, I wasn't able to work out some bugs in my cluster deferred renderer so I couldn't do a thorough analysis on it. Furthermore, the numbers I measured were not extremely accurate. I got a different measurement everytime I ran the program even for the same number of lights.
+
+Overall, the clustered forward+ was a huge optimization over regular forward rendering. It could still run on my machine at 10,000 lights, whereas forward froze my browser. For smaller numbers of lights, the performance gain was smaller. Naively running through all the lights in the scene is not as costly. Additionally there is a lot of overhead in computing the clusters and the light indexing itself. However, it pays off greatly as the number of lights in the scene increases. 
 
-*DO NOT* leave the README to the last minute! It is a crucial part of the
-project, and we will not be able to grade you without a good README.
+### Debug views
+![](img/normal.png)
+![](img/depth.png)
+![](img/image.png)
 
-This assignment has a considerable amount of performance analysis compared
-to implementation work. Complete the implementation early to leave time!
+I could render the normals, uvs, and depth of the cluster deferred renderer, but it not render the correct shading. I think something is wrong with the light sampling, but I still need to debug it!
 
 
 ### Credits

src/init.js

@@ -1,5 +1,5 @@
 // TODO: Change this to enable / disable debug mode
-export const DEBUG = true && process.env.NODE_ENV === 'development';
+export const DEBUG = false && process.env.NODE_ENV === 'development';
 
 import DAT from 'dat-gui';
 import WebGLDebug from 'webgl-debug';
@@ -60,7 +60,7 @@ stats.domElement.style.top = '0px';
 document.body.appendChild(stats.domElement);
 
 // Initialize camera
-export const camera = new PerspectiveCamera(75, canvas.clientWidth / canvas.clientHeight, 0.1, 1000);
+export const camera = new PerspectiveCamera(75, canvas.clientWidth / canvas.clientHeight, 0.1, 50);
 
 // Initialize camera controls
 export const cameraControls = new OrbitControls(camera, canvas);

src/renderers/clustered.js

@@ -4,6 +4,44 @@ import TextureBuffer from './textureBuffer';
 
 export const MAX_LIGHTS_PER_CLUSTER = 100;
 
+function getIndex(lightPos, min, max, numSlices) {
+  let z = lightPos;
+  let step = (max - min)/numSlices;
+  if (z < min) {
+    return -1;
+  } else if (z > max) {
+    return 15;
+  } else {
+    let bound = numSlices - 1;
+    //return Math.floor((z - min)/ step);
+    return (Math.max(Math.min(Math.floor((z - min)/ step), bound), 0 ) );
+  }
+}
+
+// Unused attempt at getting z in NDC
+function getZIndex(z, lPos, camera, viewmat, numSlices) {
+  let min = camera.near;
+  let max = camera.far;
+  if (z < min) {
+    return -1;
+  } else if (z > max) {
+    return 15;
+  } else {
+    let ndc = vec4.create();
+    let pMat = mat4.create();
+    mat4.copy(pMat, camera.projectionMatrix.elements);
+    ndc[0] = lPos[0];
+    ndc[1] = lPos[1];
+    ndc[2] = lPos[2];
+    ndc[3] = lPos[3];
+    vec4.transformMat4(ndc, ndc, pMat);
+
+    let ndcZ = (ndc[2]/ndc[3])/2;
+
+    return(Math.floor(ndcZ * numSlices));
+  }
+}
+
 export default class ClusteredRenderer {
   constructor(xSlices, ySlices, zSlices) {
     // Create a texture to store cluster data. Each cluster stores the number of lights followed by the light indices
@@ -27,6 +65,84 @@ export default class ClusteredRenderer {
       }
     }
 
+    // For each light
+    // 1. Get bounding indices for x, y z
+    // 2. Add light to clusterTexture for indices between
+    for (let l = 0; l < NUM_LIGHTS; l++) {
+      let lightPos = vec4.create();
+      lightPos[0] = scene.lights[l].position[0];
+      lightPos[1] = scene.lights[l].position[1];
+      lightPos[2] = scene.lights[l].position[2];
+      lightPos[3] = 1;
+
+      let r = scene.lights[l].radius;
+
+      // Light position in camera space
+      vec4.transformMat4(lightPos, lightPos, viewMatrix);
+
+      let projection = camera.projectionMatrix;
+
+      let posZ = -lightPos[2]; 
+      let posY = lightPos[1]; 
+      let posX = lightPos[0];
+      let minZ = getIndex(posZ - r, camera.near, camera.far, this._zSlices);
+      let maxZ = getIndex(posZ + r, camera.near, camera.far, this._zSlices);
+
+      let zStride = (camera.far - camera.near)/this._zSlices;
+
+      for (let p = minZ; p <= maxZ; ++p) {
+        // y = z * tan(FOV)
+        // max yPosition at z plane of intersection
+        let pZ = camera.near + p * zStride;
+        let maxPosY = pZ * Math.tan(Math.PI/180 * camera.fov * 0.5);
+        let minPosY = - maxPosY;
+
+        let maxPosX = maxPosY* camera.aspect;
+        let minPosX = -maxPosX;
+
+        let c = r;
+        let b = posZ - pZ;
+        let height = Math.sqrt(c * c - b * b);
+        let width = height * camera.aspect;
+
+        let minX = getIndex(posX - width, minPosX, maxPosX, this._xSlices);
+        let maxX = getIndex(posX + width, minPosX, maxPosX, this._xSlices);
+
+        let minY = getIndex(posY - height, minPosY, maxPosY, this._ySlices);
+        let maxY = getIndex(posY + height, minPosY, maxPosY, this._ySlices);
+
+        if (minX == 15 || minY == 15 || minZ == 15 || maxX == -1 || maxY == -1 || maxZ == -1) {
+          continue;
+        }
+
+        minX = Math.max(minX, 0);
+        maxX = Math.min(maxX, this._xSlices - 1);
+        minY = Math.max(minY, 0);
+        maxY = Math.min(maxY, this._ySlices - 1);
+        minZ = Math.max(minZ, 0);
+        maxZ = Math.min(maxZ, this._zSlices - 1);
+
+        for (let q = minY; q <= maxY; ++q) {
+          for (let r = minX; r <= maxX; ++r) {
+            let idx = r + q * this._xSlices + p * this._xSlices * this._ySlices;
+            let lightCount = this._clusterTexture.buffer[this._clusterTexture.bufferIndex(idx, 0)];
+            lightCount++;            
+            if (lightCount <= MAX_LIGHTS_PER_CLUSTER) {
+              let pixelNum = Math.floor(lightCount / 4.0);
+              let pixelIdx = lightCount % 4;
+              let clusterIdx = this._clusterTexture.bufferIndex(idx, pixelNum) + pixelIdx;
+              this._clusterTexture.buffer[this._clusterTexture.bufferIndex(idx, 0)] = lightCount;
+              this._clusterTexture.buffer[clusterIdx] = l;
+            }
+          }
+        }
+      }
+
+    }
+
     this._clusterTexture.update();
   }
+
+
+
 }

src/renderers/clusteredDeferred.js

@@ -8,6 +8,7 @@ import QuadVertSource from '../shaders/quad.vert.glsl';
 import fsSource from '../shaders/deferred.frag.glsl.js';
 import TextureBuffer from './textureBuffer';
 import ClusteredRenderer from './clustered';
+import {MAX_LIGHTS_PER_CLUSTER} from './clustered';
 
 export const NUM_GBUFFERS = 4;
 
@@ -28,8 +29,14 @@ export default class ClusteredDeferredRenderer extends ClusteredRenderer {
     this._progShade = loadShaderProgram(QuadVertSource, fsSource({
       numLights: NUM_LIGHTS,
       numGBuffers: NUM_GBUFFERS,
+      xSlices: this._xSlices, 
+      ySlices: this._ySlices, 
+      zSlices: this._zSlices,
+      maxLights: MAX_LIGHTS_PER_CLUSTER,      
     }), {
-      uniforms: ['u_gbuffers[0]', 'u_gbuffers[1]', 'u_gbuffers[2]', 'u_gbuffers[3]'],
+      uniforms: ['u_gbuffers[0]', 'u_gbuffers[1]', 'u_gbuffers[2]', 'u_gbuffers[3]',
+      'u_lightbuffer', 'u_clusterbuffer', 
+      'u_viewMatrix', 'u_near', 'u_far', 'u_viewProjectionMatrix'],
       attribs: ['a_uv'],
     });
 
@@ -153,7 +160,23 @@ export default class ClusteredDeferredRenderer extends ClusteredRenderer {
     // Use this shader program
     gl.useProgram(this._progShade.glShaderProgram);
 
+    // Upload the camera matrix
+    gl.uniformMatrix4fv(this._progShade.u_viewProjectionMatrix, false, this._viewProjectionMatrix);
+
+    // Set the light texture as a uniform input to the shader
+    gl.activeTexture(gl.TEXTURE2);
+    gl.bindTexture(gl.TEXTURE_2D, this._lightTexture.glTexture);
+    gl.uniform1i(this._progShade.u_lightbuffer, 2);
+
+    // Set the cluster texture as a uniform input to the shader
+    gl.activeTexture(gl.TEXTURE3);
+    gl.bindTexture(gl.TEXTURE_2D, this._clusterTexture.glTexture);
+    gl.uniform1i(this._progShade.u_clusterbuffer, 3);
+
     // TODO: Bind any other shader inputs
+    gl.uniformMatrix4fv(this._progShade.u_viewMatrix, false, this._viewMatrix);
+    gl.uniform1f(this._progShade.u_near, camera.near);
+    gl.uniform1f(this._progShade.u_far, camera.far);
 
     // Bind g-buffers
     const firstGBufferBinding = 0; // You may have to change this if you use other texture slots
@@ -164,5 +187,6 @@ export default class ClusteredDeferredRenderer extends ClusteredRenderer {
     }
 
     renderFullscreenQuad(this._progShade);
+
   }
 };

src/renderers/clusteredForwardPlus.js

@@ -6,6 +6,7 @@ import vsSource from '../shaders/clusteredForward.vert.glsl';
 import fsSource from '../shaders/clusteredForward.frag.glsl.js';
 import TextureBuffer from './textureBuffer';
 import ClusteredRenderer from './clustered';
+import {MAX_LIGHTS_PER_CLUSTER} from './clustered';
 
 export default class ClusteredForwardPlusRenderer extends ClusteredRenderer {
   constructor(xSlices, ySlices, zSlices) {
@@ -15,9 +16,11 @@ export default class ClusteredForwardPlusRenderer extends ClusteredRenderer {
     this._lightTexture = new TextureBuffer(NUM_LIGHTS, 8);
 
     this._shaderProgram = loadShaderProgram(vsSource, fsSource({
-      numLights: NUM_LIGHTS,
+      numLights: NUM_LIGHTS, xSlices: this._xSlices, ySlices: this._ySlices, zSlices: this._zSlices,
+      maxLights: MAX_LIGHTS_PER_CLUSTER,
     }), {
-      uniforms: ['u_viewProjectionMatrix', 'u_colmap', 'u_normap', 'u_lightbuffer', 'u_clusterbuffer'],
+      uniforms: ['u_viewProjectionMatrix', 'u_colmap', 'u_normap', 'u_lightbuffer', 'u_clusterbuffer', 
+        'u_viewMatrix','u_near', 'u_far'],
       attribs: ['a_position', 'a_normal', 'a_uv'],
     });
 
@@ -76,7 +79,10 @@ export default class ClusteredForwardPlusRenderer extends ClusteredRenderer {
     gl.uniform1i(this._shaderProgram.u_clusterbuffer, 3);
 
     // TODO: Bind any other shader inputs
-
+    gl.uniformMatrix4fv(this._shaderProgram.u_viewMatrix, false, this._viewMatrix);
+    gl.uniform1f(this._shaderProgram.u_near, camera.near);
+    gl.uniform1f(this._shaderProgram.u_far, camera.far);
+
     // Draw the scene. This function takes the shader program so that the model's textures can be bound to the right inputs
     scene.draw(this._shaderProgram);
   }

src/scene.js

@@ -8,7 +8,7 @@ export const LIGHT_RADIUS = 5.0;
 export const LIGHT_DT = -0.03;
 
 // TODO: This controls the number of lights
-export const NUM_LIGHTS = 100;
+export const NUM_LIGHTS = 1000;
 
 class Scene {
   constructor() {

src/shaders/clusteredForward.frag.glsl.js

@@ -12,6 +12,11 @@ export default function(params) {
   // TODO: Read this buffer to determine the lights influencing a cluster
   uniform sampler2D u_clusterbuffer;
 
+  uniform mat4 u_viewMatrix;
+  uniform mat4 u_viewProjectionMatrix;
+  uniform float u_near;
+  uniform float u_far;
+
   varying vec3 v_position;
   varying vec3 v_normal;
   varying vec2 v_uv;
@@ -74,22 +79,59 @@ export default function(params) {
     }
   }
 
+  ivec3 getClusterIndex(vec3 position) {
+    vec4 pos = vec4(position, 1.0);
+    vec4 zVec = u_viewMatrix * pos;
+    vec4 viewPos = u_viewProjectionMatrix * pos;
+    viewPos = viewPos / viewPos.w;
+
+    int x = int(min(floor( (viewPos.x + 1.0) * float(${params.xSlices}) / 2.0) , 14.0));
+    int y = int(min(floor( (viewPos.y + 1.0) * float(${params.ySlices}) / 2.0) , 14.0));
+
+    float zPos = - zVec[2];
+    zPos = zPos - u_near;
+
+    float zStep = (u_far - u_near)/float(${params.zSlices});
+
+    int z = int( min(floor(zPos/zStep), 14.0));    
+    return ivec3(x,y,z);
+  }
+
   void main() {
     vec3 albedo = texture2D(u_colmap, v_uv).rgb;
     vec3 normap = texture2D(u_normap, v_uv).xyz;
     vec3 normal = applyNormalMap(v_normal, normap);
 
     vec3 fragColor = vec3(0.0);
 
-    for (int i = 0; i < ${params.numLights}; ++i) {
-      Light light = UnpackLight(i);
+    ivec3 index = getClusterIndex(v_position);
+
+    int clusterIdx = index.x + index.y * ${params.xSlices} + index.z * ${params.xSlices} * ${params.ySlices};
+
+    float clusterTotal = float(${params.xSlices} * ${params.ySlices} * ${params.zSlices});
+    float clusterU = float(clusterIdx + 1) / (clusterTotal + 1.0);
+    int texHeight = int(float(${params.maxLights})*0.25);   
+
+    float lightCount = ExtractFloat(u_clusterbuffer, int(clusterTotal), texHeight, clusterIdx, 0);
+
+    float numLights = float(${params.numLights});
+    // fragColor = vec3(lightCount/ numLights, lightCount/ numLights , lightCount/ numLights);
+
+    for (int i = 0; i < ${params.numLights}; i++) {
+
+      if (i >= int(lightCount)) {
+        break;
+      }
+      int l = int (ExtractFloat(u_clusterbuffer, int(clusterTotal), texHeight, clusterIdx, i + 1));
+
+      Light light = UnpackLight(l);
       float lightDistance = distance(light.position, v_position);
       vec3 L = (light.position - v_position) / lightDistance;
 
       float lightIntensity = cubicGaussian(2.0 * lightDistance / light.radius);
       float lambertTerm = max(dot(L, normal), 0.0);
 
-      fragColor += albedo * lambertTerm * light.color * vec3(lightIntensity);
+      fragColor += albedo * lambertTerm * light.color * vec3(lightIntensity);      
     }
 
     const vec3 ambientLight = vec3(0.025);