Minimal HDR Canvas proposal #95

palemieux · 2023-02-24T07:14:26Z

Refactor the HDR Canvas proposal to a minimal baseline focused on today's HDR imagery and practice without precluding future optimizations and extensions.

Highlights:

add support for HLG and PQ color spaces, including explicit recommendations for:
- tone-mapping from HLG and PQ to sRGB
- mapping from sRGB to HLG and PQ
- mapping between HLG and PQ
add support 16-bit floats bitmaps

hdr_html_canvas_element.md

chrisn

Just reviewing today, so I added a few more comments

hdr_html_canvas_element.md

svgeesus

A few minor comments

svgeesus · 2023-03-27T16:41:29Z

hdr_html_canvas_element.md

-A color is said to be within a screen's extended range if, when that color is converted to the screen's native linear color space (using relative colorimetric intent), all of its color components are within the `[0, highDynamicRangeHeadroom]` interval.
-When displaying content produced by a rendering context with `CanvasHighDynamicRangeMode` set to `'extended'`, all colors that are within the screen's extended range must be displayed without any clamping or roll-offs (neither in chrominance nor luminance).
+A color is said to be within a screen's *extended* range if, when that color is converted to the screen's native linear color space
+(using relative colorimetric intent), all of its color components are within the `[0, highDynamicRangeHeadroom]` interval. When


That definition carefully excludes out of gamut colors, which would have some negative values, while leaving in-gamut HDR values. Nice.

svgeesus · 2023-03-27T16:42:14Z

hdr_html_canvas_element.md

+A color is said to be within a screen's *extended* range if, when that color is converted to the screen's native linear color space
+(using relative colorimetric intent), all of its color components are within the `[0, highDynamicRangeHeadroom]` interval. When
+displaying content produced by a rendering context with `CanvasHighDynamicRangeMode` set to `'extended'`, all colors that are within
+the screen's extended range must be displayed without any clamping or roll-offs (neither in chrominance nor luminance).


That precludes any soft rolloff or tonemapping. Is that intentional?

svgeesus · 2023-03-27T16:44:07Z

hdr_html_canvas_element.md


-* Connection matrix: matrix to convert the [sRGB primaries](https://www.w3.org/TR/css-color-4/#valdef-color-srgb) to the primary colours specified in Rec. ITU-R BT.2100
+In general, application should avoid conversions between color spaces and maintain imagery in its original color space: conversions


But canvas always converts on input from the specified colorspace to that of the canvas.

hdr_html_canvas_element.md

svgeesus · 2023-03-27T16:46:10Z

hdr_html_canvas_element.md

-_Note 1:_ As `rec2100-hlg` is a relative format, the brightness of the virtual monitor used for mathematical transforms can be chosen to be any level. In this transform it is chosen to be 302 cd/m2 so that the brightness of diffuse white matches the diffuse white of sRGB (80 cd/m2). Using the extended range gamma formula in footnote 2 of BT.2100, this also sets the HLG Inverse OOTF to be unity. The value 0.265 is calculated by taking the inverse OETF of 0.75, the `rec2100-hlg` diffuse white level.
-
-_Note 2:_ See section 5.3 in ITU-R BT.2408-4 relating to negative transfer functions in format conversions.
+1. Apply HLG EOTF to convert the non-linear `rec2100-hlg` Signal to linear Pseudo-Display Light with Lw = 302 cd/m2 - See Note 1


Typo: 302 should be 203

I think you are looking at an old version of the document.

svgeesus · 2023-03-27T16:47:18Z

hdr_html_canvas_element.md


 _Output:_ Full-range non-linear floating-point `rec2100-hlg` pixel with black at 0.0 and diffuse white at 0.75. Values may exist outside the range 0.0 to 1.0.

 _Process:_
-1. Convert from `extended-linear-srgb` color space to `rec2100-hlg` color space
+
+1. Convert from `extended-srgb-linear` color to `rec2100-hlg` color
 2. Scale pixel values - See Note 1
 3. Apply HLG Inverse EOTF to convert to HLG from Pseudo-Display Light with Lw = 302 cd/m2 - See Note 1


Again should be 203

I think you are looking at an old version of the document.

svgeesus · 2023-03-27T16:47:37Z

hdr_html_canvas_element.md

-_Input:_ Full-range non-linear floating-point `extended-srgb` pixel with black at 0.0 and diffuse white at 1.0. Values may exist outside the range 0.0 to 1.0.
-
-_Output:_ Full-range non-linear floating-point `rec2100-pq` pixel with black at 0.0 and diffuse white at ???. Values may exist outside the range 0.0 to 1.0.
+_Note 1:_ As `rec2100-hlg` is a relative format, the brightness of the virtual monitor used for mathematical transforms can be chosen to be any level. In this transform it is chosen to be 302 cd/m2 so that the brightness of diffuse white matches the diffuse white of sRGB (80 cd/m2). Using the extended range gamma formula in footnote 2 of BT.2100, this also sets the HLG Inverse OOTF to be unity. The value 0.265 is calculated by taking the inverse OETF of 0.75, the `rec2100-hlg` diffuse white level.


svgeesus · 2023-03-27T16:50:02Z

hdr_html_canvas_element.md


-_Output:_ Full-range non-linear floating-point `extended-srgb` pixel with black at 0.0 and diffuse white at 1.0. Values may exist outside the range 0.0 to 1.0.
+_Output:_ Full-range non-linear floating-point `rec2100-pq` pixel with black at 0.0 and diffuse white at ???. Values may exist outside the range 0.0 to 1.0.


What to say for ???
PQ diffuse white is at whatever the creative intent places it at, so maybe instead say peak small-area white (10,000 cd/m2) at 1.0?

RafaelCintron · 2023-04-18T20:45:07Z

hdr_html_canvas_element.md

+* drawing of visual elements that are related to an HDR video presentation, with
+  the expectation that the visual elements match the look of the HDR video
+  presentation.


Nit: This third bullet talks about "HDR video presentation" even though HDR images are also covered by the proposal. I think we can drop the word "video" and simply refer to an "HDR presentation".

Is the following clearer?

drawing of visual elements that are related to an HDR presentation in a <video> element, with the expectation that the visual elements match the look of the <video> element presentation

RafaelCintron · 2023-04-18T20:49:38Z

hdr_html_canvas_element.md

-    'rec2100-hlg',
-    'rec2100-pq',
-  }
+  partial dictionary ImageDataSettings {


Nit: For this and other partial definitions in the spec, please include all of the members of the type, with the new ones in bold. Saves me the trouble of hunting these down in the original spec.

RafaelCintron · 2023-04-18T20:51:02Z

hdr_html_canvas_element.md

-    'rec2100-pq',
-  }
+  partial dictionary ImageDataSettings {
+    ImageDataColorType colorType = "Uint8ClampedArray";


colorType is assigned to Uint8ClampedArray but Uint8ClampedArray does not appear in the ImageDataColorType enum defined below.

Did you mean for this to be "unorm8"? Or did you mean for ImageDataColorType to have "Uint8ClampedArray" as a choice?

It should be "unorm". Thanks.

RafaelCintron · 2023-04-18T21:05:28Z

hdr_html_canvas_element.md

+```idl
+  enum CanvasColorType {
+    "unorm8",
+    "float16",


I see the highest bit depth of CanvasColorType is float16 but ImageDataColorType goes all the way to float32.

Is this a hedge against float16 not getting approved by the JS standards body? If not, seems like we should unify these or provide a use case for float32 in the spec.

Is this a hedge against float16 not getting approved by the JS standards body?

I think it is saying that it is not an essential part of this spec, and that either would work -- at least initially. I do not like however the idea of a placeholder, so listing the two seemed the least terrible approach. What is your preference?

RafaelCintron · 2023-04-18T22:56:15Z

hdr_html_canvas_element.md

-  return [r4, g4, b4];
-}
+```idl
+  enum CanvasColorType {


I wouldn't really call unorm8 and float16 "color types". I think "CanvasDataType" or "CanvasFormatType" would be a better name for this enum.

I agree. Done.

RafaelCintron · 2023-04-18T23:26:08Z

hdr_html_canvas_element.md


-#### srgb-linear
+Add a new CanvasColorMetadata dictionary:


How will we help developers provide good values for the dataype and colorspace of the canvas? I originally thought the spec would have recommendations based on the metadata but the metadata is only available after the canvas is created.

Do we expect developers to first make an sacrificial canvas, see what metadata is returned, and then make the "real" canvas afterwards? Seems like the metadata should be available independent of an individual canvas instance.

In general, I think the spec should provide use cases for the metadata such as how the metadata guides what DOMString values to pass for 2D canvas colors. Without use cases, this is going to draw the ire of those who oppose any fingerprinting bits be exposed to web developers for informational purposes. I, personally, think we can make the case for having these but we need to be more clear why it is needed in the spec.

Yes, we definitely need more information -- similar to what is provided in the PNG 3ED spec: https://w3c.github.io/PNG-spec/#mDCv-chunk

There are two basic use cases:

the application receives the ST 2086 metadata, e.g. from metadata within the image file or from an API call. In this case, the application simply copies the information as in the CanvasSmpteSt2086Metadata structure

the application generates the ST 2086 metadata, in which case, there are best practices we should be able to point to

See added prose

RafaelCintron · 2023-04-18T23:43:04Z

hdr_html_canvas_element.md

-      return 1 / scale;
-    }
+```idl
+// SMPTE ST 2086 color volume metadata.


Please define what "SMPTE ST 2086" stands for somewhere in the spec.

Removed "ST 2086" from symbol names and added it as a reference in the prose.

RafaelCintron · 2023-04-18T23:44:30Z

hdr_html_canvas_element.md

-    }
+```idl
+// SMPTE ST 2086 color volume metadata.
+dictionary CanvasSmpteSt2086Metadata {


Does "SmpteSt2086" need to be part of the enum name? Is there some other metadata we plan to add in the future that needs its own dictionary? I think calling this CanvasMetadata should suffice.

Removed "ST 2086" from symbol names and added it as a reference in the prose.

michaeldsmith · 2023-04-19T16:23:34Z

The ST2094-10 Annex B method mentioned in the section "rec2100-pq to srgb"

needs parameters min/avg/max values.

The example code at https://www.sandflow.com/public/tone-mapping/index.html performs image analysis to determine the min/avg/max values. This is a great way to obtain the min/avg/max parameters for a single image.

I think that it should be suggested that the good results will come from obtaining the min/avg/max parameters via image analysis.

If image analysis is not performed, then the min/avg/max values need to be provided as parameters to 2094-10 Annex B another way. Default min and max values could come from ST2086 mastering display metadata min luminance and max luminance, default max value could come from MaxCLL metadata. The average will depend on if the scene is bright, average or dark. Using a default average value between 5-15 nits could be a good starting place.

…e on the same HTML Canvas

ccameron-chromium · 2023-04-21T20:41:52Z

This removes the features related to rendering content in an SDR-relative, application-tone-mapped canvas (in particular, the extended mode, and the ScreenDetails parts.

Splitting these into two proposals is probably the right thing to do -- there is too much misunderstanding between the two worlds that both call themselves HDR.

I would also suggest separating the float16 support out from this proposal. It already has its own proposal here and lengthy-and-stalled discussion here.

A remaining core part of this proposal (with this PR applied) is a standardization of how to render PQ and HLG content on phone/tablet/laptop/desktop operating systems.

These operating systems parameterize their displays by two parameters: HDR headroom and color gamut (this encompasses of Windows, iOS, macOS, Android, and ChromeOS, including the built-into-a-TV variants). None of the existing standards address this problem (they have no concept of a time-varying HDR headroom, which is fundamental to all operating systems' implementations of HDR).

There is also this issue about standardizing how PQ and HLG images are to be rendered as HTMLImageElement.

palemieux · 2023-04-25T04:31:46Z

I would also suggest separating the float16 support out from this proposal.

I do not think this is separable since HDR cannot exist in an 8-bit world, as described in the introduction.

there is too much misunderstanding between the two worlds that both call themselves HDR.

I do not see a misunderstanding.

HLG and PQ have been iterated for years and are used by all HDR TV and movie content. The web has become an important source of TV and movie consumption. It makes sense for the web platform to support them.

This does not preclude/conflict with other approaches, as far as I know.

ccameron-chromium · 2023-04-25T21:15:31Z

I would also suggest separating the float16 support out from this proposal.

I do not think this is separable since HDR cannot exist in an 8-bit world, as described in the introduction.

Changes to PredefinedColorSpace for HLG and PQ is valuable on its own, because PredefinedColorSpace is used by 2D canvas, ImageData, WebGL, and WebGPU.

There are some issues to be worked out for 2D canvas and ImageData with respect to float16 support, but WebGPU is already shipping support for it, and WebGL has a proposal that doesn't have any issues that I know of.

I wouldn't want us to block HLG and PQ support in WebGPU because of unrelated 2D canvas issues (which may take a long time to resolve).

I also don't think we should explicitly disallow 8-bit HDR (it's not unreasonable for HLG -- see these 8-bit JPEG HLG images).

palemieux · 2023-04-25T21:22:52Z

I also don't think we should explicitly disallow 8-bit HDR

Have you read the article at https://ieeexplore.ieee.org/document/7514362 ? It lays out the fact that 8-bit is not even sufficient for sRGB at low luminance.

P.S.: The images show up HDR in my Chrome browser! :)

palemieux · 2023-04-25T21:23:53Z

I wouldn't want us to block HLG and PQ support in WebGPU because of unrelated 2D canvas issues (which may take a long time to resolve).

I argue we should use HDR as a way to drive the resolution of these issues.

I am not disagreeing with you that there is an obstacle.

kenrussell · 2023-04-25T23:40:27Z

We could almost replace references to Float16Array with Float32Array and not only would no web developer notice, but also the references could be changed back to Float16Array once that has been shipped in browsers with no application impact. Since reads from Float16Array will immediately convert to 64-bit float (JavaScript's native number representation) and writes to Float16Array silently truncate from that representation, it will behave very similarly to how Float32Array does today.

However - since this might be contentious, getting this in to have a spec browsers can start implementing would be preferable in my opinion.

svgeesus · 2023-04-27T14:45:40Z

I also don't think we should explicitly disallow 8-bit HDR

I disagree. Don't give people a footgun, and instead, use this to drive the higher-bit-depth canvas upgrade.

palemieux added 6 commits January 30, 2023 09:08

Initial refactoring

7220dbd

Add rec2100-pq HDR conversions

3a07c6c

Color space conversion illustration

aebfc6f

In progress

e5bb8a4

Simplify strawman

0269ed1

Address feedback comments

33758e9

chrisn reviewed Mar 13, 2023

View reviewed changes

palemieux added 2 commits March 13, 2023 15:49

Fix typos

c663547

Further simplify the proposal

3981262

chrisn reviewed Mar 15, 2023

View reviewed changes

palemieux added 3 commits March 19, 2023 13:01

Address comments

c6ba457

Clarify the scope of the proposal

9d02917

Address feedback from Mike Smith

5b16baf

palemieux changed the title ~~Simplification of the HDR Canvas proposal~~ Minimal HDR Canvas proposal Mar 20, 2023

chrisn reviewed Mar 20, 2023

View reviewed changes

hdr_html_canvas_element.md Outdated Show resolved Hide resolved

hdr_html_canvas_element.md Outdated Show resolved Hide resolved

Fix typo

c0b6e55

palemieux marked this pull request as draft March 20, 2023 16:45

palemieux mentioned this pull request Mar 24, 2023

WebGPU, WCG and HDR #8

Open

palemieux marked this pull request as ready for review March 24, 2023 23:27

svgeesus reviewed Mar 27, 2023

View reviewed changes

RafaelCintron reviewed Apr 18, 2023

View reviewed changes

palemieux added 3 commits April 18, 2023 22:36

Change CanvasColorType to CanvasDataType

eb3581a

Fix default value of ImageDataSettings::dataType

3995615

Add details on CanvasMasteringDisplayMetadata

bf420a5

palemieux added 3 commits April 19, 2023 09:37

Addressed w3c#95 (comment)

514a1e6

Address feedback

62debfe

Add note re: practices when drawing images with different color volum…

380d4a7

…e on the same HTML Canvas

palemieux requested a review from svgeesus April 25, 2023 17:15

chrisn mentioned this pull request Apr 25, 2023

Support for HDR images and video on the web w3c/media-and-entertainment#96

Open

palemieux merged commit 45bdcce into w3c:main May 2, 2023


		* Connection matrix: matrix to convert the [sRGB primaries](https://www.w3.org/TR/css-color-4/#valdef-color-srgb) to the primary colours specified in Rec. ITU-R BT.2100
		In general, application should avoid conversions between color spaces and maintain imagery in its original color space: conversions


		_Output:_ Full-range non-linear floating-point `extended-srgb` pixel with black at 0.0 and diffuse white at 1.0. Values may exist outside the range 0.0 to 1.0.
		_Output:_ Full-range non-linear floating-point `rec2100-pq` pixel with black at 0.0 and diffuse white at ???. Values may exist outside the range 0.0 to 1.0.

Minimal HDR Canvas proposal #95

Minimal HDR Canvas proposal #95

Conversation

palemieux commented Feb 24, 2023 • edited by svgeesus Loading

chrisn left a comment

Choose a reason for hiding this comment

svgeesus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

palemieux Apr 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaeldsmith commented Apr 19, 2023 • edited Loading

ccameron-chromium commented Apr 21, 2023

palemieux commented Apr 25, 2023

ccameron-chromium commented Apr 25, 2023

palemieux commented Apr 25, 2023

palemieux commented Apr 25, 2023 • edited Loading

kenrussell commented Apr 25, 2023

svgeesus commented Apr 27, 2023

palemieux commented Feb 24, 2023 •

edited by svgeesus

Loading

palemieux Apr 19, 2023 •

edited

Loading

michaeldsmith commented Apr 19, 2023 •

edited

Loading

palemieux commented Apr 25, 2023 •

edited

Loading