Sometimes projection skips some triangles/faces

[trusktr]

I'm not sure why, but in some cases the projection will skip some triangles of a mesh, and also will render on the other side when it shouldn't. I'm not sure yet how exactly I trigger this case (I'm using my TS fork, but the code is the same, and I submitted the same bug fixes here as there).

Here is a video. Notice that half of the sphere never receives the projection, and also the projection seems to happen on both sides (unlike any of the examples).

wth.mp4

Do you have any ideas what could cause this?

---

I don't have a simple reproduction yet because I've made some custom elements for LUME that use ProjectedMaterial internally, allowing this to be possible (the above video is from the following HTML code):

<style>
	html,
	body {
		width: 100%;
		height: 100%;
		margin: 0;
		padding: 0;
		background: #222;
		touch-action: none;
	}
</style>

<script src="global.js"></script>

<lume-scene id="scene" perspective="800" webgl>
	<lume-point-light position="200 -200 200" intensity="0.6" color="white"></lume-point-light>
	<lume-ambient-light color="white" intensity="0.6"></lume-ambient-light>
	<lume-camera-rig active="false" initial-distance="500" max-distance="1500" min-distance="100"></lume-camera-rig>

	<lume-points
		id="model"
		has="ply-geometry"
		src="./models/shelby-gt350.ply"
		point-size="0.3"
		rotation="90 0 0"
		position="-250 0 0"
		size="0 0 0"
		scale="50 50 50"
	></lume-points>

	<lume-node position="100 100 100" rotation="0 30 0">
		<lume-camera-rig active initial-distance="500" max-distance="1500" min-distance="100"></lume-camera-rig>
		<lume-sphere
			has="projected-material"
			projected-textures="#tex"
			size="50 50 50"
			color="white"
			mount-point="0.5 0.5 0.5"
		>
		</lume-sphere>
		<lume-node id="rotator" align-point="0.5 0.5 0.5">
			<lume-projected-texture
				id="tex"
				size="50 50 50"
				src="./images/monalisa-2.jpg"
				fitment="contain"
				position="0 0 100"
				mount-point="0.5 0.5 0.5"
			>
				<lume-plane
					wireframe
					size="1 1"
					has="basic-material"
					size-mode="proportional proportional"
					color="skyblue"
					sidedness="double"
				></lume-plane>
			</lume-projected-texture>
		</lume-node>
	</lume-node>
</lume-scene>

<script>
	LUME.defineElements()

	rotator.rotation = (x, y, z) => [x, y + 0.3, z]
</script>

It seems to me, and I definitely don't doubt because I've encountered issues numerous times, that THREE.WebGLRenderer has state bugs that may affect this somehow.

How do I know Three.js has some fairly bad bugs? Well for example, and in my experience porting Three.js to WebAssembly (with AssemblyScript, a TypeScript to Wasm compiler) I have found, fixed, and added unit tests for multiple bugs that TypeScript would have otherwise caught, so I know from experience the code base is not in as good of a shape as it can be.

[trusktr]

Would you be willing to have a session to debug this? I added you on Twitter where we can arrange something over DM if you'd be interested.

I discovered that if I call material.project(mesh) too many times, I get the above problem. If I don't call it too many times, I get the expected result but only on first frame (unable to animate). Notice the projection does not go to both sides of the sphere, as we would expect:

correct.mp4

[trusktr]

I'm not yet sure what's causing this on my end. LUME sets matrixAutoUpdate to false for all objects controlled by the custom elements, then it updates their world matrices at a certain time. Maybe this has to do with the issue.

In the meantime, I found the following temporary workaround on my end, in the GLSL code:

					// bool isFacingProjector = dotProduct > 0.0000001;
					bool isFacingProjector = true; // always apply it, for now

Somehow, the direction of the camera that ProjectedMaterial sees is different than what I actually have in my scene, which is why the above sphere front faces doesn't match with actual projection direction. Not sure what's up yet, but this workaround works for my current case.

[marcofugaro]

Looks like some matrix isn't getting updated. My guess would be the mesh.matrixWold used here:

three-projected-material/src/ProjectedMaterial.js

Lines 309 to 311 in af105f1

	// we save the object model matrix so it's projected relative
	// to that position, like a snapshot
	this.uniforms.savedModelMatrix.value.copy(mesh.matrixWorld)

Don't have any other useful information since I don't know your setup. Doesn't seem an issue with the library to me.

[trusktr]

Hmmm, yeah I'm not sure what is going on yet, but I am quite sure that I am updating the world matrices because I am moving the objects around with animation, however the material isn't updating.

In the above video, the camera used for projection is a child of an object in the scene (the blue square) which is being animated, and we can tell from this that the world matrices are in fact being updated.

I'll report back once I find out.