NV_materials_mdl

[tblutNV]

The NV_materials_mdl extensions enables glTF assets to store and transfer materials defined in the NVIDIA Material Definition Language (MDL). The intention of this extension is to enable higher quality rendering based on MDL materials instead of glTF 2.0 PBR materials. The material definitions are mutually exclusive, therefore, if a renderer does not support this extension, then closely matching standard PBR materials can be used as a fallback to achieve a representative rendering of the glTF asset.

Two additional vendor extensions, NV_image_exr and NV_image_vdb, that enable images to be stored and transferred in the OpenEXR and OpenVDB formats, respectively, are added as well. The intention of defining these extensions only on images and not textures, like other image format extensions do, is that these formats are only needed for consumption in MDL and there is no obvious use-case for them in regular glTF textures.

We're looking forward to further discussions.

[donmccurdy]

The intention of defining [EXR and VDB] only on images and not textures, like other image format extensions do, is that these formats are only needed for consumption in MDL and there is no obvious use-case for them in regular glTF textures.

One notable reason for extending the texture rather than the image: it provides a fallback mechanism. The example below would require support for OpenEXR, while allowing clients to optionally choose a KTX2 version instead, for purposes of upload time or VRAM cost.

{
    "asset": {
        "version": "2.0"
    },
    "extensionsUsed": [
        "KHR_texture_basisu",
        "NV_texture_exr"
    ],
    "extensionsRequired": [
        "NV_texture_exr"
    ],
    "textures": [
        {
            "extensions": {
                "NV_texture_exr": {
                    "source": 0
                },
                "KHR_texture_basisu": {
                    "source": 1
                },
            }
        }
    ],
    "images": [
        {
            "mimeType": "image/x-exr",
            "bufferView": 1
        },
        {
            "mimeType": "image/ktx2",
            "bufferView": 2
        }
    ]
}

The example above is hypothetical at the moment — new extensions would be required to use float16 or float32 KTX2 files in glTF, as well. But this is worth considering, particularly if any of the the proposals here evolved into EXT_ or KHR_ extensions.

EXT_lights_image_based, KHR_materials_emissive_strength, and KHR_lights_environment (draft proposal) extensions could plausibly benefit from OpenEXR textures. Or, implementations of NV_materials_mdl might plausibly choose to support BC6H KTX2 textures with fallback to OpenEXR textures.

[tblutNV]

The fallback mechanism is a valid point. This would also allow us to use the OpenEXR images that are referenced in MDL materials in the standard PBR materials which we use a fallback if NV_materials_mdl isn't supported. OpenVDB doesn't have any obvious use case outside of NV_materials_mdl though, given that it's volume data. In MDL we use it for 3D textures which are not supported by glTF 2.0 currently, but this might change in the future of course, so it wouldn't really hurt to allow OpenVDB resources to be used in textures, I guess?

[donmccurdy]

I agree it wouldn't hurt! KTX2 does support 3D textures as well, though glTF materials do not allow that usage. I've seen KTX2 3D textures applied to glTF materials in custom 3D asset pipelines. But that was a very specific performance optimization, not any sort of volumetric effect.

[fire]

Does gltf have a extension for loading openexr images? That could be a good addition similar to BPTC bc6 and webp extensions for hdr and regular images.

[donmccurdy]

I'm not aware of a previous EXT_texture_exr proposal. For realtime rendering in glTF materials, I would really lean toward KTX2's float16, float32, or BC6H capabilities though.

[fire]

There is a gap since BASISU doesn't support float16 or float32 hdr texture, and KTX2's float16, float32 is lossless. BC6H is only for PC platforms and we're missing the mobile platforms.

[donmccurdy]

OpenEXR also offers float16 and float32 with lossless compression, to my understanding, but it does currently include more lossless compression methods than KTX2. While KHR_texture_basisu focuses on Basis Universal formats, not all of KTX2's functionality, that choice is downstream of the current needs in the glTF ecosystem. Defining a more flexible KHR_texture_ktx2 would be trivial, if the need were there. I don't think OpenEXR fills a gap that KTX2 cannot already fill, other than interoperability with MDL, Blender, etc.

That said — I have no objection to an NV_ or EXT_ prefixed extension for OpenEXR, but would have some misgivings about KHR_. I don't love the idea of shipping textures that need to be parsed pixel by pixel at runtime by the viewer, this is not a good thing for mobile platforms especially.

[donmccurdy]

Correction to my earlier post — one related proposal:

• EXT_hdr_textures #1365

I believe @hybridherbst and Needle Tools might also be using OpenEXR textures in glTF extensions through a vendor extension, for lightmaps.

[hybridherbst]

That's correct. Ours is modelled pretty much 1:1 after EXT_texture_webp. The reason for going that route is that we're definitely seeing future use of EXR textures as emissive textures, besides the usage as IBL, lightmap or reflection probe (our current usage for the extension).

I agree with what @donmccurdy mentioned regarding the possibility of defining fallbacks for textures, too.

Example glTF structure

{
  "asset": {
    "generator": "Needle Engine",
    "version": "2.0"
  },
  "extensionsUsed": [
    "NEEDLE_gameobject_data",
    "EXT_texture_exr",
    "NEEDLE_lightmaps",
    "NEEDLE_lighting_settings"
  ],
  ...
  "images": [
    {
      "mimeType": "image/exr",
      "bufferView": 6,
      "name": "Skybox"
    }
  ],
  ...
  "textures": [
    {
      "sampler": 0,
      "source": 0,
      "name": "Skybox",
      "extensions": {
        "EXT_texture_exr": {
          "source": 0
        }
      }
    }
  ],
  ...
}

I can put up a proposal PR for EXT_texture_exr if that helps. I believe the main point of discussion last time was that EXR is a relatively complex format, and while the "core" of it for "I just want a single layer of floating point data" is pretty agreed upon there's numerous edge cases that one wouldn't really want to impose on loaders.

[tblutNV]

We decided to change the image extensions to texture extensions. I'm assuming our proposal for NV_texture_exr looks very similar to your extension @hybridherbst ? We don't explicitly mention any edge cases though. Our only requirement is basically that OpenEXR images are allowed to be listed in the images array for consumption by NV_materials_mdl.

[lexaknyazev]

glTF images are barely resource pointers so adding new media types there does not hurt because core glTF textures are allowed to refer only to PNG and JPEG images. On the other hand, glTF textures are referenced by materials (both core and extended) and must conform to various context-dependent rules.

Conceptually, glTF textures should be regarded only as a tuple of a 2D image (with mip levels in case of KTX) and an associated sampler state. The fallback mechanism for WebP and BasisU extensions ensures that new formats do not change anything beyond storage and/or VRAM consumption.

The original MDL proposal was well-scoped because only MDL internal objects would be allowed to use EXR and VDB images. Now with both formats proposed as texture extensions, there are multiple cases that dramatically increase the scope because now any texture slot in any material may refer to texture objects with EXR and/or VDB payloads. Therefore, these texture extensions now have to meticulously define EXR and VDB for non-MDL use cases.

[tblutNV]

NV_texture_exr also defines a fallback mechanism like the WebP extension does @lexaknyazev. However, NV_texture_vdb doesn't since it isn't possible to fall back to a 2D PNG or JPG for volumetric data. If the extensions must exactly define how the features of the formats should be used in textures, then we'd rather stick to our original proposal of only extending image and keeping everything contained in MDL internal objects, given our tight time constraint. We won't be able to invest too much time into this at this moment. We would be willing to help define a EXT_texture_exr in the future though, but wouldn't be able lead the initiative.

[donmccurdy]

glTF textures should be regarded only as a tuple of a 2D image (with mip levels in case of KTX) and an associated sampler state

I'm not sure I agree on this point. Should 3D textures, array textures, or animated textures someday have a purpose in glTF, I would expect to find them in this array.

...there are multiple cases that dramatically increase the scope because now any texture slot in any material may refer to texture objects with EXR and/or VDB payloads. Therefore, these texture extensions now have to meticulously define EXR and VDB for non-MDL use cases.

This doesn't need to be the case, correct? The *_texture_exr extension could say, for example, "This extension may not be not be attached to any texture unless that texture is unused. Other extensions may loosen this requirement."

EDIT: I'm mainly interested in consistency among extensions here. Different entrypoints for texture formats, with different fallback mechanisms, seems to needlessly complicate client implementation from my perspective.

[hybridherbst]

@lexaknyazev in our case we exactly based the extension on how EXT_texture_webp is defined. Would you say that does it wrong too? Personally I agree with @donmccurdy here, some notes on expected usage would work but having different formats defined very differently feels wrong from a consistency standpoint.

My takeaway here is that I'll open a PR for EXT_texture_exr so that the discussion here isn't derailed too much and we can continue there.

[lexaknyazev]

Should 3D textures, array textures, or animated textures someday have a purpose in glTF, I would expect to find them in this array.

That would be true for glTF 1.0 that directly binds textures to custom shaders.

Texture objects of glTF 2.0 have to remain compatible with materials that use them. Since there is no unambiguous way to use, e.g., a texture object with VDB payload as a material's base color, a VDB texture extension has either to define it, which is clearly out of NV scope, or completely ban it, which makes putting such payloads in glTF's texture objects much less valuable.

Different entrypoints for texture formats, with different fallback mechanisms, seems to needlessly complicate client implementation from my perspective.

They are not so different after all

• Actual images go to the images array anyway and MDL objects would refer to them directly without being restricted by glTF's textures.
• A restricted EXR texture extension for some other use cases would still be possible and it may even refer to the same images as MDL objects do as long as the images are compatible.

[donmccurdy]

A restricted EXR texture extension for some other use cases would still be possible and it may even refer to the same images as MDL objects do as long as the images are compatible.

That's a very interesting idea... in that case, possibly the NV_materials_mdl extension does not need to be paired with these VDB or OpenEXR extensions at all, it could just reference normal images containing these file formats? Does MDL already define sampler and texCoord properties, if those are needed?

Then if we someday need glTF materials referencing OpenEXR textures, EXT_texture_exr or another extension would provide that access point, with appropriate restrictions.

[tblutNV]

That's a very interesting idea... in that case, possibly the NV_materials_mdl extension does not need to be paired with these VDB or OpenEXR extensions at all, it could just reference normal images containing these file formats?

I don't think simply including such file formats in an asset without specifying extensions for them is a valid option. How would the loader know which formats are required to be supported otherwise?

Does MDL already define sampler and texCoord properties, if those are needed?

Yes, texture coordinates and certain sampler properties can be defined in MDL. The texture coordinates of the glTF mesh can be accessed in MDL, but there is no restriction on which values are passed to the texture lookup functions since they are simply float2/float3 values.

[lexaknyazev]

I don't think simply including such file formats in an asset without specifying extensions for them is a valid option. How would the loader know which formats are required to be supported otherwise?

The NV_materials_mdl extension itself may say something like:

When this extension is used, image objects may refer to EXR and VDB resources using image/x-exr and application/x-vdb media types respectively.

This implies that all implementations of NV_materials_mdl would have to support EXR and VDB but that's likely already the case.

[tblutNV]

We decided to update the NV_materials_mdl extension with your suggestion @lexaknyazev, so support for recognizing OpenEXR and OpenVDB image objects is now described in NV_materials_mdl and the NV_image_* extensions have been removed.

[lkettnerNV]

Hi everybody. There was no time in yesterdays call to discuss this extension. However, I think we are pretty far along with this one and we adopted the improvements suggested here on how to handle the image format extensions. I would be happy if the can finalize this pull request in this thread also without meeting. I can also be available for the next call next week, but would not be available after that until late August. We are of course available and happy to answer all questions you might have here, and very grateful for all the great feedback we received here.

[emackey]

@lexaknyazev When you get a chance can you post the last round of review comments for the JSON schemas here? We'll status again at the PBR meeting on Monday. Thanks!