KHR_environment_map #1956
KHR_environment_map #1956
Conversation
This is a continuation of PR #1850 made into KHR extension and based on Khronos gltf repo (not forked)
|
With regards to names, I prefer not "environment", rather Sky Dome, Dome Light or Sky Light as I think that is more descriptive. Here is a survey of what other renderers and engines use:
|
Clarification of cubemap generation and use of spherical harmonics. Remove duplicate section. Added images and textures to json example. Added implementation note for specular and irradiance sections. Updated contributors.
Elaborate on how KTX v2 is supported, add supported formats.
Specify that this extension requires KHR_texture_ktx. Clarify that no pre-filtering shall be done for uncompressed formats. Add open issues.
Add section for KTX v2 images, update json to include extensionsUsed
|
Starting to look into how best to represent environment map data for scenes in Mozilla Hubs. This definitely looks pretty close to our needs. Its likely for our usecase we will want to be able to define multiple environment maps per scene, either on a per material level, or perhaps in more of a "light probe" way where environment maps will be effective in some spatial region, and possibly be mixed. The later certainly sounds out of scope for this extension, but allowing references on materials may make sense (the currently specified one on the scene being the default for materials not overriding it). In any case we can certainly at least share the actual We also have a |
|
Hi netpro2k and thanks for your comments!
A good idea to be able to specify multiple environment maps! Do you have a suggestion on how to spatially override the scene environment map? |
Good point - maybe I could add a value indicating the contribution/use of the Do you think this could work? |
This is non-physical, advanced and I would suggest we do not do that. This is similar to the concept to advanced light linking, where only certain objects are affected by certain lights. This is not supported in the punctual light extension either because it is really advanced. |
|
I think that if we removed SH's from the current definition and had the mipmaps optional (I prefer removed, but it isn't that important), this could get approved by the committee. I think because we haven't done that this extension is not getting support from the standards committee and we are moving around laterally as a result. I think there is a clear way forward to getting this approved. I would suggest we just go ahead and get it done? |
While I definitely appreciate the tooling simplicity this would provide,I think my main concern with this extension essentially being just "a way to associate cubemaps with a gltf" would be that it is then a bit under specified what that model should look like at runtime. I am not particularly tied to including SHs but I do think this extension needs to at least mostly specify how whatever data it does include is to be used. Re mips, I am still unclear on what the suggested solution is for compressed textures if we do not require including the full mip chain other than decompressing it, generating the mip chain, and then uploading the entire thing uncompressed. I suppose this is not completely unacceptable but certainly seems undesirable from a "I know what the runtime costs of this asset I am creating are going to be" perspective.
I would definitely agree on the point of light linking being out of scope for this, not sure about including something to indicate how the cubemap should be used. Especially if dropping SHs (since previously it was discussed that you could omit the cubemap or SH if you only wanted specular or diffuse components) it does seem like it may be useful to be able to specify multiple ways to use the cubemap data. Don't feel particularly strong about this point either though, I can see a good argument for this being only a "purely physical" lighting representation to keep things simple.
Hmm yeah its a bit conceptually weird to associate it with a material, though this is what three.js does. Associating with nodes could also make sense, though I do think any sort of spatial mapping (like specifying a bounding area of effect, falloff, etc) is probably getting out of scope for this and should be done in another extension. Specifying just 1 environment map for the entire gltf seems a bit restricted to the sort of "model viewer" usecase, where I am thinking more along the lines of a "level" asset which will likely need multiple. I guess I am trying to come up with something in between single environment map per scene and a full "light probe" extension, but that might end up being too awkward. |
|
Hi @johguenther
I don't really understand the usecase here - could you please elaborate?
I would say yes, since we do want the implementation to be accessible. Hope you understand what I mean? Best regards |
It probably depends on the point of view: I see it mainly as a light source (after all it's a
Fair enough. Is there already such a "light probes on nodes" extension in the making (a PR)? |
The number of authoring tools that can export light probe data in any format is — as best as I can tell — zero, despite efforts to do so:
That being the case, I'm unfortunately not sure where to start defining a standard light probe representation. Ideally we want standards to be informed by some kind of implementation, even if it's only proof-of-concept. It's a compelling feature, but I think it probably needs concrete workflows and interest from potential implementors first. |
|
I was thinking just of the HDRI environment texture, to be placed as a light at nodes. An implementation / renderer could derive the SH coefficients or other pre-filtering from that in a preprocessing step, if needed. |
This is what this extension aims to do for the scene - as opposed to node level. |
This extension shall be standalone without the dependency to some other extension to define cubemap texture support.
Fixes to schema, add schema for cubemap
| "uri": "cubemap0.ktx2", | ||
| "mimeType": "image/ktx2" | ||
| } | ||
| ], |
There was a problem hiding this comment.
I'm curious, why create a new images array under the extension rather than using the root images array? Similarly, it might be more natural to use the root textures array rather than adding cubemaps here. For example:
// root
{
"textures": [{
"source": 0,
"extensions": {"KHR_lights_environment":{"layer": 0}} // optional
}],
"images": [{
"uri": "cubemap.ktx2",
"mimeType": "image/ktx2"
}],
"extensions": [{
"lights": [{
"name": "SceneEnvironment",
"cubemap": 0,
"irradianceCoefficients": [...],
...
}]
}],
...
}There was a problem hiding this comment.
Hi @donmccurdy and thanks for your comment
I agree that it is not the obvious choice to put the cubemaps inside the light extension.
The reason for this is backwards compatibility and separation.
By backwards compatibility I mean that since we introduce a new fileformat (ktx) and new texture formats this may break current implementations if they go through the main images and/or textures arrays and treat them like they are jpg/png.
This is a behavior which I think must be allowed with the glTF 2.0 spec - there simply is no other formats for images/textures.
If the cubemaps are included in the root images/textures arrays - how are implementations that does not support this extension supposed to handle those resources?
By separation I mean that from a data packaging perspective it may make sense to put the cubemaps inside the extension to make it abundantly clear that these resources are only ever used if the extension is supported (and used)
I understand that the behavior to load the cubemaps will be slightly different than to normal resources - but I think it's worth it due to the above reasons.
Please let me know what you think!
There was a problem hiding this comment.
IMO, I don't think this would break any existing implementation in the same way that the KHR_texture_basisu extension didn't by adding image/ktx2 mime type in the images. Otherwise an implementation that didn't support KHR_texture_basisu should run into the same issue you described.
In practice I think most implementation would load what they support and either ignore the rest or load them generically.
For example, in my image loading implementation (WebGPU/WebGL based), for jpg/png or any web loadable image type I load it using an Image element and use createImageBitmap to send it to the GPU. For everything else I simply load it as an array buffer, working under the assumption that any spec compliant object that references that image will know what to do it with it from there. As is the case with KHR_texture_basisu, and now this KHR_lights_environment would as well. In my case with the current spec of this extension, I reused the same Image loading logic I just had to call it again from the extension specific logic rather during the root load logic.
There was a problem hiding this comment.
Aaah, now I see what you mean.
That would mean that the expected behavior at load time is to check what textures are used and then proceed to load image resources based on that usage.
I will move the images from being included in the extension to be at root level images array instead.
| ## Specular Radiance Cubemaps | ||
|
|
||
| The cubemap used for specular radiance is defined as a cubemap containing separate images for each cube face. | ||
| The data in the maps represents illuminance in candela per square meter. |
There was a problem hiding this comment.
I'm not sure how to interpret cd / m² — wouldn't we need to know the physical area represented by a pixel to use that? Perhaps cd / sr (sr = steradians)? I notice Unreal Engine uses cd / m², but it also requires a reference size (in meters) used for projection onto the scene. It would be helpful to know if other formats (.hdr or .exr) have chosen any existing precedent here.
There was a problem hiding this comment.
The way I think about it is that the output uses the same units meaning that the area represented is only resolved in the final step, usually in the display.
Modern displays knows how many candelas they can output and the same units are used for instance in HDR content.
Does this answer your question?
On a side note, this is one reason why I think it is important to proceed with #2083
| The data in the maps represents illuminance in candela per square meter. | ||
| Using this information it is possible to calculate the non-direct light contribution to the scene if irradiance coefficients are not included. | ||
|
|
||
| Cube faces are defined in the KTX2 format specification, implementations of this extension must follow the KTX2 cubemap specification. |
There was a problem hiding this comment.
It's probably necessary to say something about the color space. For material textures, the embedded color space can (and typically should) be ignored. For environment cubemaps I'm not sure – given the specified units of cd / m² is this always Linear-sRGB? Or do we need to allow other options?
There was a problem hiding this comment.
My assumption was that the colorspace is given by the vkFormat.
According to the KTX V2 spec the dataformat colorspace information is only needed when the format is VK_FORMAT_UNDEFINED
This would give:
VK_FORMAT_R16G16B16_SFLOAT being a linear format in the default colorspace
VK_FORMAT_R8G8B8_SRGB being nonlinear in the default colorspace
The default colorspace being define by the ITU BT 709 color primaries.
Do you think this makes sense?
|
I just wanted to provide some feedback on this extension as I recently implemented the specular cubemap and irradiance coefficient portion (I haven't done the localized bit yet). Schema layoutI do think this extension would be simpler if it reused the root image and texture collections for the cubemap. As it stands now I had to put in extra extension specific loading in place to load the cubemap, which really just includes fetching and parsing the KTX. Normally though, I can just loop through the images / textures and load from there since there is already logic in place for loading ktx2 image types. Now I have to go check if there are khr_lights_environment.lights to load as well. It's not the end of the world but it just seemed a bit unnecessary. I think @donmccurdy's suggestion in the review comments makes sense there. Irradiance CoefficientsSo let me preface by saying my mathematics is not the strongest but I have read through the referenced papers on spherical harmonics and looked through the various code samples included on deriving SH. What I found was that there are slight variations that led to some confusion about what the "correct" approach would be. For example, there was one example that used the texel solid angle for applying weight to the coefficients and another that simply divided by the number of samples rather than keeping a weighted sum. So because of this discrepancy I went looking for other papers and came across Stupid SH Tricks which did a good job of breaking it down but it also used another slight variation in that it used an approximation for the solid angle. Another major thing to consider is that if the irradianceCoefficients are supplied in the gltf file, they must match what the runtime derivation would be if they weren't. I know this seems obvious but with the above I think this may vary more than expected. In my opinion, because the gltf file should be universal, it kind of forces us to pick one rather than leaving it up to the runtime implementation. In addition, while looking for implementations I came across bablyonjs's implementation of spherical harmonics. Specifically here. You can see that their coefficients have integrated the reconstructions coefficients directly and have divided by pi to simulate Lambertian (more on this bit below). Should this spec take a similar approach? At least the reconstruction portion might make sense. If so the included irradianceCoefficients need to specify this or they definitely won't match runtime derivation or the shader code. How irradiance coefficients fit into the GGX/Lambertian BRDF modelPerhaps you can just ignore this section. I think this is just my own misunderstanding of the math because after I read what I wrote here I realized that perhaps the texels ARE each channel which would explain the off by PI if the code examples assume a 4 channel format (i.e. 4 * pi / sum). I spent an embarrassingly long time trying to figure out why my diffuse portion was so washed out. I am using the same GGX/Lambertian BRDF model from the glTF-Sample-Viewer. I had assumed that I could swap out the Lambertian portion with the reconstructed irradiance. It did not look good. So I came across another implementation which did the weight a little bit differently here. And I thought, that can't be right. They are summing each color channel, rather than each texel. So I thought it must be off by PI somewhere as that is pretty close to 3 (the number of channels). So I divided the weight by PI and voila it looked perfect. But I could not understand why I needed to do this. Perhaps I missed it but I couldn't find it anywhere in the referenced papers. It wasn't until later that I came across the bablyonjs implementation and I realized that this is required in the ggx/lambertian model. In any case, my point is that it would be helpful if this extension called this out somewhere or provided reference to a paper which described the relationship between SH and GGX/Lambertian. Cubemap Texture FormatsI think it's important to note that WebGPU does not support 3 channel texture formats (with the exception of packed formats). It also does not provide a way to write 3 channel buffers to 4 channel formats. This means that you have to pad the buffer before offloading it to the GPU resulting in extra CPU side preprocessing. I don't know if that is a problem for this spec so much as a problem for WebGPU. However, the list of supported texture formats here seemed somewhat arbitrary. It seems like we should at least also support the 4 channel variations. |
|
Hi @shannon and thank you so much for your great effort and comments - greatly appreciated! Schema LayoutI totally see your point and have been thinking about where to include the cubemaps. Irradiance CoefficientsYes, there are slight differences 'out there' as to how the coefficients are calculated and used. I would assume the divide by PI depends on if a solid angle is used when calculating the coefficients or not. Cubemap Texture FormatsThis texture format limitation is nothing strange to me at all. Or do you see a need to actually provide the alpha channel in the cubemap? Again - thank you so much for your comments and effort! :-) |
|
@rsahlin more than happy to contribute where I can. I don't really have a preference for which algorithm to use. I ended up using the Stupid SH Tricks algorithm because it was the simplest to implement and since it used an approximation I assumed it must be slightly faster. My understanding is that this is just the same thing but with an approximation of the solid angle. I did replace it with an actual solid angle calculation and it made no visible difference. In both cases I needed to keep the 1/PI or it was very blown out. After more reading on it, I do think it really just needs to be divided by pi to fit into the Lambertian BRDF formula. In either case, if this spec defined that the full solid angle should be calculated I would just drop the approximation. It only affects the preprocess time and not render time anyways. I've included the two relevant JS and WGSL shader snippets here for reference: async deriveIrradianceCoefficients(commandEncoder, cubemap) {
const start = performance.now();
//Downsample to SPHERICAL_HARMONICS_SAMPLE_SIZE (128x128 seems to be sufficient)
//Downsampling also ensures that the data is in rgba32float format regardless of the input ktx format.
const data = await this.downsampleCubemap(commandEncoder, cubemap);
const size = SPHERICAL_HARMONICS_SAMPLE_SIZE;
const coefficients = [...new Array(9)].map(() => new Float32Array(3));
const texel = (x, y, f) => {
const i = ((f * size * size) + ((size * y) + x)) * 4;
return new Float32Array(data.buffer, data.byteOffset + (i * 4), 3);
}
let weightSum = 0;
for(let f = 0; f < 6; f++) {
for (let y = 0; y < size; y++) {
for (let x = 0; x < size; x++) {
const u = ((x + 0.5) / size) * 2.0 - 1.0;
const v = ((y + 0.5) / size) * 2.0 - 1.0;
const temp = 1.0 + u * u + v * v;
const weight = 4.0 / (Math.sqrt(temp) * temp);
// const weight = texelSolidAngle(x, y, size, size);
const [dx, dy, dz] = cubeCoord(u, v, f);
const color = texel(x, y, f);
for(let c = 0; c < 3; c++) { //this is faster than vec3 methods
const value = color[c] * weight;
//band 0
coefficients[0][c] += value * 0.282095;
//band 1
coefficients[1][c] += value * 0.488603 * dy;
coefficients[2][c] += value * 0.488603 * dz;
coefficients[3][c] += value * 0.488603 * dx;
//band 2
coefficients[4][c] += value * 1.092548 * dx * dy;
coefficients[5][c] += value * 1.092548 * dy * dz;
coefficients[6][c] += value * 0.315392 * (3.0 * dz * dz - 1.0);
coefficients[7][c] += value * 1.092548 * dx * dz;
coefficients[8][c] += value * 0.546274 * (dx * dx - dy * dy);
}
weightSum += weight;
}
}
}
for(let c = 0; c < 9; c++) {
vec3.scale(coefficients[c], coefficients[c], 4 * Math.PI / weightSum);
}
console.log(`Derived in ${performance.now() - start}ms`, coefficients);
return coefficients;
}fn getIrradiance(n: vec3<f32>) -> vec3<f32>{
let c1 = 0.429043;
let c2 = 0.511664;
let c3 = 0.743125;
let c4 = 0.886227;
let c5 = 0.247708;
var L00 = lighting.irradianceCoefficients[0];
var L1_1 = lighting.irradianceCoefficients[1];
var L10 = lighting.irradianceCoefficients[2];
var L11 = lighting.irradianceCoefficients[3];
var L2_2 = lighting.irradianceCoefficients[4];
var L2_1 = lighting.irradianceCoefficients[5];
var L20 = lighting.irradianceCoefficients[6];
var L21 = lighting.irradianceCoefficients[7];
var L22 = lighting.irradianceCoefficients[8];
return (
c1 * L22 * (n.x * n.x - n.y * n.y) +
c3 * L20 * n.z * n.z +
c4 * L00 -
c5 * L20 +
2.0 * c1 * (L2_2 * n.x * n.y + L21 * n.x * n.z + L2_1 * n.y * n.z) +
2.0 * c2 * (L11 * n.x + L1_1 * n.y + L10 * n.z)
) / vec3<f32>(${M_PI});
}For the cubemap texture formats. I don't really feel strongly about this, I just noticed that while trying to get an early estimation of the time it would take to prefilter and derive the coefficients, I found that padding the buffer from 3 channels to 4 channels before sending it to the GPU was a large portion of the time spent. About 200ms of blocking CPU time on a decent desktop. This leads to a lot of jank on startup. I'm still optimizing this so it's possible I can get rid of this in some other way. |
|
Great work @shannon - thanks :-) |
|
I am still spending my time completing the KHR_displaymapping_pq extension and will hopefully get around to updating this extension within a couple of weeks. |
|
I am finally gearing up to get time to spend on this extension - hope to have time in the coming weeks |
|
I just wanted to share a running implementation of this extension in case anyone is interested. This is not a finished engine so a couple of things to note:
A good portion of the shader code was ported or referenced from the glTF-Sample-Viewer project. I wrote a deno script to generate the sample environments from polyhaven HDRs. (Yay for WebGPU in deno!). I did implement the localized portion of this extension but I don't have any good sample models to really test it fully. I will try to make a better one at some point as I was really just adding the bounding boxes to existing environments. Which is not ideal since the scene doesn't really line up with anything like the scene from the reference paper. |
|
Thank you @rsahlin and everyone that's working on this. I would love to see this in glTF and am curious what the status is. I'm asking because we're looking at proposing something that would fit very nicely with this. It's based on an Unreal feature called HDRI Backdrop, which is a more generalized version of a skybox that is more suited for product visualization. Regarding
|
|
Hi @shannon, @aidinabedi and all - thanks for the interest and support for this extension! In answer to your questions @aidinabedi:
Unless there are some very specific (current) needs that cannot be fulfilled by using an equirectangular -> ktx cubemap tool I would prefer to keep it simple (and only support KTX V2 cubemap format) - is this ok with you? Can you think of any reason why this would not work in the future? Best regards |
|
That's excellent to hear. Will you be at Khronos AWE event next week? Would be a great opportunity to discuss some of these details there as well.
In my limited experience, I haven't encountered multiple cubemaps in a single KTX file (and can't really imagine much use case for it). Additionally, since KHR_texture_basisu itself doesn't provide any
I would like to argue that panoramic images are the de facto transmission/delivery format for environment maps. You'll find endless panoramic HDRIs available online, but few (if any) cubemaps. Additionally, I believe one of the goals of glTF (at least in my view) is to standardize features already implemented - and I have yet to encounter a 3D engine (whether web or native) that doesn't support panoramic images out-of-the-box. For some it's also the only way to input an environment (like Unreal, Sketchfab, and even the glTF sample viewer). On top of that it would additionally removes the extension's reliance on a single image format, which makes it significantly more attractive for engines and platforms that don't prioritize supporting KTX. Sincerely, |
The use case could probably be related to switching environments similar to
The BasisU extension provides a drop-in alternative to the core PNG and JPEG formats so it deliberately does not expose any extra KTX features.
glTF texture objects are defined as image/sampler tuples, which would have little value over referring to images directly. Nevertheless, this extension should put its images in the global
glTF's goal is finding efficient (and sometimes new) ways of standardizing existing features, not necessarily copying popular solutions as-is. Since this extension is in early draft, it's expected that glTF-Sample-Viewer has no support for its design. Besides, using KTX to store equirectangular panoramic images could be evaluated as well.
Relying and standardizing on KTX for image transmission is one of the main 3D Formats WG priorities. |
|
I have been trying to optimize my implementation of this a bit to get rid of the jank on startup. Something I am noticing is that the prefilter shader is not cheap and it needs to run faces x roughnessLevel x (ggx+Charlie), which leads to a total of 120 draw calls. On some environments this is pretty fast but on others it can be pretty slow and in webgl2 this leads to blocking the main UI thread (compositor). Does it make sense to store all of this in the same KTX file ahead of time? With roughness stored as mip levels along with a separate LUT. Before I implemented this extension, I stored the ggx and Charlie as two separate cubemaps already prefiltered and loaded them directly. Perhaps this could be a use case for the layers in KTX. I'm really just thinking out loud as I am not an expert in this area. My understanding is this would require defining a specific BRDF for this extension. For now I am just going to break up the prefiltering to something like running one roughness level per frame. Instead of all in one frame. |
|
Hi I agree with @lexaknyazev - the reason we do not see any current usage of cubemaps in KTX is due to lack of tools.
I think the normal way of releasing extensions is to first expose the functionality then decide to break down into multiple extensions if there is need for it. @shannon Thanks for the info - it would be great if we could provide 'implementation notes' for how to efficiently and/or accurately do the prefiltering. |
|
I have created a couple of test models - one with the intention of checking roughness values and how the reflection changes with roughness going from 0 to 1 The spheres are 1m in diameter, there are two rows - one close to the viewer (at Z=0) and one row at a distance (Z = 15m) |
What is specified here is not lights in the sense that are otherwise defined in glTF - for instance KHR_llights_punctual. One major difference is that the incoming light contribution from an environment lacks distance. This extension shall not be seen as a replacement for spot or directional lights - it shall be seen as an extension that may provide reflection lookup and some non-directed light contribution. But not as a replacement for 'real' light extensions - hence the decision to rename.
|
And here is a test model to show what the reflection will be with different IOR values - going from IOR = 1 to the left up to IOR = 3 on the right.
|
Moved images array to root of glTF. Added properties for cubemap intensity, irradiance factor and ior.
This is a continuation of #1850 which seems to have gone stale and is based on a fork of glTF in the UX3D repo.
The purpose of this extension is to add support for defining an environment map (cubemap or spherical harmonics) to a glTF scene.
I have done my best to update according to comments in #1850
This extension is based on EXT_lights_image_based, with the major difference being that KTX2 is used as a container format for cubemap textures.
Cubemaps may choose to include prefiltered mip-levels , if not included this shall be done by implementations.
This extension specifies a set of texture formats that are allowed for the cubemaps.