feat(ext/canvas): enhance `createImageBitmap` specification compliance #25517

Hajime-san · 2024-09-08T22:17:24Z

Feature

resolves #26032

basic support for decoding image format

jpeg, png, gif, bmp, ico, webp
- animated image
- 16-bit image
- Grayscale image

issues for supporting AVIF

The AVIF image format is supported by all browsers today.
However, the standardization of sniffing mime type seems to have hard going.

Missing image codecs whatwg/mimesniff#143

support `colorSpace` for `ImageData`

~~There can convert color space from srgb to display-p3 with createImageBitmap.~~

This was a misunderstanding on my part, so I reverted it.
The conversion using the ImageData colorSpace option was probably the responsibility of CanvasRenderingContext2D.
I've left the implementation commented out as it may be useful as a reference for implementing CanvasRenderingContext2D or OffscreenCanvasRenderingContext2D.

support `colorSpaceConversion` option

The specifications are not particularly clear about the implementation details, and I had a hard time understanding its behavior, but I eventually do reverse engineering the results of wpt and implemented it to be consistent with those results.

support receives `ImageBitmap`

createImageBitmap: Clarify treatment of colorSpaceConversion:"none" ImageBitmaps whatwg/html#10578

port to Rust

To simplify code, almost logic moved to Rust side.

added a simple architecture document for README

It is basically based on the image implementation, and I wrote roughly how you can handle data effectively.

performance impact

I was curious to see if the increased amount of Rust code, especially static dispatch, was affecting the bootstrap time of the runtime.
I'm not very confident about the suitability of this benchmark.

bootstrap

console.log('Hello, world!');

this pr fb6b87d

% hyperfine --warmup 5 'target/release/deno run bootstrap.ts'
Benchmark 1: target/release/deno run bootstrap.ts
  Time (mean ± σ):      21.5 ms ±   3.6 ms    [User: 14.2 ms, System: 5.5 ms]
  Range (min … max):    18.7 ms …  54.5 ms    102 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

main b1b72a8

% hyperfine --warmup 5 'target/release/deno run bootstrap.ts'
Benchmark 1: target/release/deno run bootstrap.ts
  Time (mean ± σ):      22.2 ms ±   4.7 ms    [User: 14.8 ms, System: 5.4 ms]
  Range (min … max):    18.8 ms …  66.8 ms    97 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

API benchmark

const imageData = new ImageData(new Uint8ClampedArray([255, 0, 0, 255]), 1, 1);

Deno.bench("createImageBitmap", async () => {
  await createImageBitmap(imageData);
})

this pr acc4119

% target/release/deno bench bench.ts
    CPU | Apple M1
Runtime | Deno 2.0.0-rc.2 (aarch64-apple-darwin)

file:///path/to/deno/bench.ts

benchmark           time/iter (avg)        iter/s      (min … max)           p75      p99     p995
------------------- ----------------------------- --------------------- --------------------------
createImageBitmap            1.6 µs       615,000 (  1.6 µs …   1.7 µs)   1.6 µs   1.7 µs   1.7 µs

main b1b72a8

% target/release/deno bench bench.ts
    CPU | Apple M1
Runtime | Deno 1.46.1 (aarch64-apple-darwin)

file:///path/to/deno/bench.ts

benchmark           time/iter (avg)        iter/s      (min … max)           p75      p99     p995
------------------- ----------------------------- --------------------- --------------------------
createImageBitmap            3.5 µs       287,200 (  3.3 µs …   3.6 µs)   3.5 µs   3.6 µs   3.6 µs

extra

The benchmarks below are for when the argument to op_create_image_bitmap is a struct except for the first argument of buf.
It seems that serializing the argument to op as a struct has a fairly large impact on performance.
This may apply to things other than op_create_image_bitmap as well.

% target/release/deno bench bench.ts
    CPU | Apple M1
Runtime | Deno 2.0.0-rc.2 (aarch64-apple-darwin)

file:///path/to/deno/bench.ts

benchmark           time/iter (avg)        iter/s      (min … max)           p75      p99     p995
------------------- ----------------------------- --------------------- --------------------------
createImageBitmap            4.2 µs       235,900 (  4.1 µs …   4.5 µs)   4.3 µs   4.5 µs   4.5 µs

plans for enhancement in another PRs

improve performance
There is known to that processing image is a cpu-bound task. So, op_create_image_bitmap consider to be data-parallelism with using rayon.
align error message
- Normalize error message grammar (TS/JS) #25269
- refactor(ext): align error messages #25310

…xcept for JPG and animated image - https://mimesniff.spec.whatwg.org/#matching-an-image-type-pattern

- adding `DOMExceptionInvalidStateError` in global due to handling not supported image format - animation image is not supported currently

Hajime-san · 2024-10-06T08:09:54Z

Updated: The image supports various bit depth now.

Avif YUV decoder, drop dcv, high bit depth image-rs/image#2373

Note for supporting AVIF: I've implemented AVIF support using the AvifDecoder here 111f00b, although it only supports 8-bit currently.
As mentioned in the overview of this PR, sniffing MIME about AVIF is an ongoing issue, but I believe this implementation could support it without deviating from the spec.
The AvifDecoder depends on dav1d and seems to require additional tools to build locally and in CI.

Build Failure log

% cargo b   
   ...
   Compiling dav1d-sys v0.8.2
error: failed to run custom build command for `dav1d-sys v0.8.2`

Caused by:
  process didn't exit successfully: `/path/to/deno/target/debug/build/dav1d-sys-7acb1adeb047a692/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=DAV1D_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_aarch64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_aarch64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_aarch64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_aarch64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_aarch64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_aarch64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at /path/to/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dav1d-sys-0.8.2/build.rs:82:10:
  called `Result::unwrap()` on an `Err` value: PkgConfig(
  pkg-config exited with status code 1
  > PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 pkg-config --libs --cflags dav1d dav1d >= 1.3.0

  The system library `dav1d` required by crate `dav1d-sys` was not found.
  The file `dav1d.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.
  The PKG_CONFIG_PATH environment variable is not set.

  HINT: if you have installed the library, try setting PKG_CONFIG_PATH to the directory containing `dav1d.pc`.
  )
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

% uname -a
Darwin user.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:16:51 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8103 arm64

% brew install dav1d
% cargo b

petamoriken · 2024-10-15T17:52:17Z

ext/canvas/01_image.js

+hint: When you want to get a "Blob" from "fetch", make sure to go through a file server that returns the appropriate content-type response header,
+      and specify the URL to the file server like "await(await fetch('http://localhost:8000/sample.png').blob()".
+      Alternatively, if you are reading a local file using 'Deno.readFile' etc.,
+      set the appropriate MIME type like "new Blob([await Deno.readFile('sample.png')], { type: 'image/png' })".\n`,


Suggestions may be better written in runtime/fmt_errors.rs.

Ref #26252

Thanks.

@crowlKats
Could I move this hint to runtime/fmt_errors.rs?

Thank you for your suggestion and permission, but I reverted this commit.
The runtime/fmt_errors.rs is exactly as implemented, suggesting hints from error messages in the runtime stack trace.
I think it's preferable to provide hints for specific APIs, as in this implementation, in the implementation itself from a maintenance perspective.
There are other places where hints are implemented, for example, here.

Regarding this issue, I think it would be better to devise a common system for displaying hints later.

petamoriken · 2024-10-15T17:52:45Z

ext/canvas/01_image.js

+info: The following MIME types are supported:
+      https://mimesniff.spec.whatwg.org/#image-type-pattern-matching-algorithm\n`,


crowlKats · 2024-10-24T08:34:24Z

@Hajime-san is it ready for review or still working on this?

Hajime-san · 2024-10-24T09:00:32Z

@crowlKats
Thanks, there were a lot of changes in this PR though it's ready for review.
From now on, I will not be making any new changes until I receive your review.👍

crowlKats

Great PR, thanks a lot for this.
Am really sorry it took so long to review this.
Left a few comments and changes, but overall looks extremely good!

ext/canvas/image_ops.rs

ext/canvas/01_image.js

crowlKats · 2024-11-23T05:10:18Z

ext/canvas/01_image.js

+    // TODO(Hajime-san): this should be real async
+    const processedImage = op_create_image_bitmap(


i think it should be fine if its sync; afterall it makes sense for browsers to be async since they are handling graphics stuff outside of the js apis sandbox, but for us we dont have that so its fine.

ext/canvas/op_create_image_bitmap.rs

ext/canvas/image_decoder.rs

ext/canvas/op_create_image_bitmap.rs

- denoland#25517 (comment)

- denoland#25517 (comment) - denoland#25517 (comment)

- denoland#25517 (comment)

ext/canvas/lib.rs

crowlKats · 2024-11-28T01:32:48Z

Overall looking very good, though one nitpick: in the newly added error messages, could you remove the trailing .? This is just a design decision that we have made to have a unified style for error messages

- denoland#25517 (comment)

Hajime-san added 21 commits August 29, 2024 17:35

basic support all image type pattern of unsigned 8bit when its blob e…

198c90a

…xcept for JPG and animated image - https://mimesniff.spec.whatwg.org/#matching-an-image-type-pattern

support 8bit JPG

b9fe5e8

handling unsuportted situation

77703e8

- adding `DOMExceptionInvalidStateError` in global due to handling not supported image format - animation image is not supported currently

fmt

22cf2f9

cleanup premultiplyAlpha and adding test

32d71f5

handling unsupported bit depth

827e7b1

port to rust

0de8a9b

support animated PNG

d0da79e

support animated WebP

d255518

support animated GIF

396c09c

support colorspaceConversion for ImageData

7d3e39d

support colorspaceConversion for Blob

a6604c1

clean up unit test

1e21467

clean up

1f7524a

add unit test

98e9304

update README

4bc517e

Merge remote-tracking branch 'upstream/main' into update-imagebitmap

dbac50d

fix error handling

f0895c6

update README

86f0000

revert "support colorspaceConversion for ImageData"

75d26df

switch to static link for lcms2

1de527f

Hajime-san force-pushed the update-imagebitmap branch from a234586 to 1de527f Compare September 17, 2024 01:56

Hajime-san added 2 commits September 17, 2024 12:47

Merge branch 'main' into update-imagebitmap

825515e

add comments for the reference of image mimesniff

bd77fb9

Hajime-san force-pushed the update-imagebitmap branch from 055395a to bd77fb9 Compare September 21, 2024 07:04

marvinhagemeister requested a review from littledivy September 23, 2024 13:37

Hajime-san added 4 commits September 29, 2024 10:19

clean up process_icc_profile_conversion

2d58fd1

clean up

d258914

fmt

fa01320

avoid copying vector

8d8d726

Hajime-san force-pushed the update-imagebitmap branch from 111f00b to 69addc8 Compare October 6, 2024 07:35

perf: pass u8 to op

acc4119

petamoriken reviewed Oct 15, 2024

View reviewed changes

Hajime-san added 4 commits October 20, 2024 20:17

support imageOrientation correctly

83c40c4

remove unnecessary boilerplate

cb72235

add comment

3ef4cf9

Merge branch 'main' into update-imagebitmap

529c3f0

Hajime-san added 2 commits October 24, 2024 17:40

update error

50bee86

map lcms error

1429013

crowlKats requested changes Nov 23, 2024

View reviewed changes

Hajime-san added 10 commits November 26, 2024 10:06

remove TODO

40c89cc

- denoland#25517 (comment)

move suggestion to runtime/format_errors.rs

ad67750

- denoland#25517 (comment) - denoland#25517 (comment)

remove comment out

3fcc3ab

- denoland#25517 (comment)

remove comment

24306b3

- denoland#25517 (comment)

remove boilerplate

332fc41

- denoland#25517 (comment)

use an unique variant of CanvasError

d764f62

- denoland#25517 (comment)

implement the case of ImageBitmap

4b9cf86

- denoland#25517 (comment)

better error handling

5b11282

fix typo

e59be4a

Merge branch 'main' into update-imagebitmap

40cee98

Hajime-san requested a review from crowlKats November 27, 2024 23:18

crowlKats reviewed Nov 28, 2024

View reviewed changes

ext/canvas/lib.rs Outdated Show resolved Hide resolved

Hajime-san added 2 commits November 28, 2024 11:41

use ImageError instead of string

7039401

- denoland#25517 (comment)

align error message

4cca9ae

- denoland#25517 (comment)

Hajime-san requested a review from crowlKats November 28, 2024 22:33

Revert move suggestion to runtime/fmt_errors.rs

6453b7f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ext/canvas): enhance `createImageBitmap` specification compliance #25517

feat(ext/canvas): enhance `createImageBitmap` specification compliance #25517

Hajime-san commented Sep 8, 2024 •

edited

Loading

Hajime-san commented Oct 6, 2024 •

edited

Loading

petamoriken Oct 15, 2024

Hajime-san Nov 25, 2024

crowlKats Nov 25, 2024

Hajime-san Dec 7, 2024

petamoriken Oct 15, 2024

crowlKats commented Oct 24, 2024

Hajime-san commented Oct 24, 2024

crowlKats left a comment

crowlKats Nov 23, 2024

crowlKats commented Nov 28, 2024

		info: The following MIME types are supported:
		https://mimesniff.spec.whatwg.org/#image-type-pattern-matching-algorithm\n`,

		// TODO(Hajime-san): this should be real async
		const processedImage = op_create_image_bitmap(

feat(ext/canvas): enhance createImageBitmap specification compliance #25517

Are you sure you want to change the base?

feat(ext/canvas): enhance createImageBitmap specification compliance #25517

Conversation

Hajime-san commented Sep 8, 2024 • edited Loading

Feature

basic support for decoding image format

issues for supporting AVIF

support colorSpace for ImageData

support colorSpaceConversion option

support receives ImageBitmap

port to Rust

added a simple architecture document for README

performance impact

bootstrap

API benchmark

extra

plans for enhancement in another PRs

Hajime-san commented Oct 6, 2024 • edited Loading

petamoriken Oct 15, 2024

Choose a reason for hiding this comment

Hajime-san Nov 25, 2024

Choose a reason for hiding this comment

crowlKats Nov 25, 2024

Choose a reason for hiding this comment

Hajime-san Dec 7, 2024

Choose a reason for hiding this comment

petamoriken Oct 15, 2024

Choose a reason for hiding this comment

crowlKats commented Oct 24, 2024

Hajime-san commented Oct 24, 2024

crowlKats left a comment

Choose a reason for hiding this comment

crowlKats Nov 23, 2024

Choose a reason for hiding this comment

crowlKats commented Nov 28, 2024

feat(ext/canvas): enhance `createImageBitmap` specification compliance #25517

feat(ext/canvas): enhance `createImageBitmap` specification compliance #25517

Hajime-san commented Sep 8, 2024 •

edited

Loading

support `colorSpace` for `ImageData`

support `colorSpaceConversion` option

support receives `ImageBitmap`

Hajime-san commented Oct 6, 2024 •

edited

Loading