WinML backend for wasi-nn #7807

jianjunz · 2024-01-24T06:50:28Z

This change adds Windows Machine Learning backend for wasi-nn. Since WinML is a built-in feature of Windows, this change makes it possible to run wasi-nn on Windows without a third-party machine learning framework.

WinML backend is only available on Windows.

WinML: https://learn.microsoft.com/en-us/windows/ai/windows-ml/

crates/test-programs/Cargo.toml

Like the rest of the `windows-*` crates published by Kenny Kerr, this change also adds the `windows` crate itself to the trusted list. This is necessary for use in bytecodealliance#7807.

Like the rest of the `windows-*` crates published by Kenny Kerr, this change also adds the `windows` crate itself to the trusted list. This is necessary for use in #7807.

abrown · 2024-01-30T18:20:40Z

@jianjunz:now that #7846 is merged you should be able to rebase on that and get cargo vet to pass.

crates/test-programs/Cargo.toml

crates/wasi-nn/Cargo.toml

crates/wasi-nn/src/backend/winml.rs

abrown · 2024-01-30T18:47:28Z

crates/wasi-nn/tests/all.rs

+        #[cfg(feature = "openvino")]
+        let mut backend = backend::openvino::OpenvinoBackend::default();
+        #[cfg(feature = "winml")]
+        let mut backend = backend::winml::WinMLBackend::default();


This won't work when both features are enabled. Features are additive, so both backends can be present at the same time.

We run openvino and winml backend in two different test cases, with different inputs, so only one backend is specified here for testing purpose.

backend.list() is able to list all available backends.

wasmtime/crates/wasi-nn/src/backend/mod.rs

Lines 21 to 31 in 9732234

pub fn list() -> Vec<crate::Backend> {

let mut backends = vec![];

#[cfg(feature = "openvino")]

{

backends.push(Backend::from(OpenvinoBackend::default()));

}

#[cfg(feature = "winml")]

{

backends.push(Backend::from(WinMLBackend::default()));

}

backends

What I mean is: if someone runs cargo test --all-features, both the OpenVINO and WinML statements that assign to let mut backend will be present in the code. Some background: this all.rs file is compiled once by cargo test into a single binary that runs multiple tests, so that compilation will be incorrect because the second let mut backend will shadow the first.

Thanks for the background. It's fixed.

abrown · 2024-01-30T18:50:17Z

crates/wasi-nn/tests/all.rs

+#[cfg_attr(not(all(target_arch = "x86_64", target_os = "windows")), ignore)]
+#[test]
+fn nn_image_classification_winml() {
+    run(NN_IMAGE_CLASSIFICATION_WINML, true).unwrap()


You will want to double-check that this test is actually run. Right now in Wasmtime's CI, only a subset of jobs run to preserve runner time. The Windows "test" job does not run by default but if you add the string prtest:full in any commit, then all jobs will run. Then we can check the logs for the Windows job to ensure this test actually ran.

With prtest:full, this test failed on Windows Server. I guess GitHub Windows Server runner doesn't have Desktop Experience enabled.

I'm afraid we need to disable this test at this time. GitHub Actions Windows Server image doesn't have Desktop Experience available, so it cannot be installed.

List of installed or available features:
https://github.com/jianjunz/wasmtime/actions/runs/7752997447/job/21143496106#step:9:1

Yeah, that's unfortunate. Why don't we add a function like check_openvino_is_installed but for WinML: check_winml_is_available. There's probably a way to check with the Windows API if ML system calls are possible and that function could return false if they are not.

That extra runtime check would be in addition to the #[cfg_attr(not(feature = "winml"), ignore)]. That can stay there for the case where the winml feature isn't even enabled.

check_winml_is_available is added. It returns ok for my Windows 11, but I'll need to find an environment to check if it returns an error when WinML is not available.

In bytecodealliance#7846 I added `cargo-vet` entries to trust the `windows` crate, just like we already trust several related `windows-*` crates. I did not, however, update the lockfile, which means that bytecodealliance#7807 continued to fail the `cargo vet --locked` CI check. This change is the result of simply running `cargo vet`.

In #7846 I added `cargo-vet` entries to trust the `windows` crate, just like we already trust several related `windows-*` crates. I did not, however, update the lockfile, which means that #7807 continued to fail the `cargo vet --locked` CI check. This change is the result of simply running `cargo vet`.

jianjunz · 2024-02-11T12:41:46Z

Thanks for auditing windows-* crates, but cargo vet still fails after #7900. Only 1 unvetted dependency now.

1 unvetted dependencies:
  windows:0.52.0 missing ["safe-to-deploy"]

abrown

@jianjunz, this looks good to me with a couple of final points:

let's resolve the shadowing for the cargo test --all-features case
let's add the check_winml_is_available dynamic check
can you rebase on main? If it's not too much trouble, I might suggest squashing some of the commits since the entire commit history will get dropped into the merge commit (but this is a personal preference thing: feel free to leave as-is).

ExecutionTarget::Gpu is mapped to LearningModelDeviceKind::DirectX.

App knows input and target data range, so it's better to let app to handle scaling.

To avoid involving too many external dependencies, input image is converted to tensor data offline.

Only FP32 is supported right now. Reject other tensor types.

It aligns with openvino backend. prtest:full

jianjunz · 2024-02-28T09:02:17Z

The code is rebased. No conflict. However, cargo vet reported more unvetted dependencies. I'll check if there is any unnecessary change for cargo.lock.

jianjunz · 2024-03-01T00:30:33Z

Thanks for your review and merging this PR.

jianjunz requested review from a team as code owners January 24, 2024 06:50

jianjunz requested review from pchickey and removed request for a team January 24, 2024 06:50

jianjunz force-pushed the winml branch from 139dbfc to 950a165 Compare January 24, 2024 07:37

jianjunz commented Jan 24, 2024

View reviewed changes

crates/test-programs/Cargo.toml Outdated Show resolved Hide resolved

jianjunz marked this pull request as draft January 24, 2024 08:43

jianjunz force-pushed the winml branch 2 times, most recently from 54de278 to a51d2b0 Compare January 29, 2024 02:45

abrown mentioned this pull request Jan 30, 2024

vet: trust the windows crates #7846

Merged

abrown reviewed Jan 30, 2024

View reviewed changes

jianjunz force-pushed the winml branch 2 times, most recently from 1262255 to 9732234 Compare February 1, 2024 08:20

jianjunz marked this pull request as ready for review February 5, 2024 04:51

This was referenced Feb 8, 2024

vet: update lock file for trusted windows crates #7900

Merged

[WASI-NN] Add support for a ONNXruntime backend using ort #7691

Merged

alexcrichton mentioned this pull request Feb 10, 2024

vet: prune lockfile #7910

Merged

jianjunz force-pushed the winml branch from 4b456ec to 2a48656 Compare February 11, 2024 09:50

abrown approved these changes Feb 27, 2024

View reviewed changes

jianjunz added 5 commits February 28, 2024 10:11

Add WinML backend for wasi-nn.

200d716

Log execution time.

2624feb

WinML backend supports execution target selection.

31a5f6d

ExecutionTarget::Gpu is mapped to LearningModelDeviceKind::DirectX.

Limit WinML backend on Windows only.

e3d4f45

Move wasi-nn WinML example to new test infra.

ca7ef3a

jianjunz and others added 12 commits February 28, 2024 10:11

Scale tensor data in test app.

bab7508

App knows input and target data range, so it's better to let app to handle scaling.

Remove old example for wasi-nn WinML backend.

1d6611a

Update image2tensor link.

26119ad

Format code.

d6b6294

Upgrade image2tensor to 0.3.1.

8a7297b

Upgrade windows to 0.52.0

6a2c813

Use tensor data as input for wasi-nn WinML backend test.

2447e7a

To avoid involving too many external dependencies, input image is converted to tensor data offline.

Restore trailing new line for Cargo.toml.

18166fe

Remove unnecessary features for windows crate.

c4f7ce2

Check input tensor types.

0e31cef

Only FP32 is supported right now. Reject other tensor types.

Rename default model name to model.onnx.

99c33c0

It aligns with openvino backend. prtest:full

Run nn_image_classification_winml only when winml is enabled.

ca792aa

jianjunz force-pushed the winml branch from f81c59d to 264ff6b Compare February 28, 2024 08:40

abrown and others added 3 commits February 28, 2024 20:17

vet: add trusted windows crate to lockfile

0dd6e06

Fix wasi-nn tests when both openvino and winml are enabled.

64ddce9

Add check for WinML availability.

2de787c

jianjunz force-pushed the winml branch from 264ff6b to 2de787c Compare February 28, 2024 12:21

vet: reapply vet lock

4cd42ed

abrown added this pull request to the merge queue Feb 29, 2024

Merged via the queue into bytecodealliance:main with commit 10a9f9e Feb 29, 2024
43 checks passed

jianjunz deleted the winml branch March 1, 2024 00:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WinML backend for wasi-nn #7807

WinML backend for wasi-nn #7807

jianjunz commented Jan 24, 2024

abrown commented Jan 30, 2024

abrown Jan 30, 2024

jianjunz Feb 1, 2024

abrown Feb 27, 2024

jianjunz Feb 28, 2024

abrown Jan 30, 2024

jianjunz Feb 1, 2024

jianjunz Feb 2, 2024

abrown Feb 27, 2024

jianjunz Feb 28, 2024

jianjunz commented Feb 11, 2024

abrown left a comment

jianjunz commented Feb 28, 2024 •

edited

Loading

jianjunz commented Mar 1, 2024

	pub fn list() -> Vec<crate::Backend> {
	let mut backends = vec![];
	#[cfg(feature = "openvino")]
	{
	backends.push(Backend::from(OpenvinoBackend::default()));
	}
	#[cfg(feature = "winml")]
	{
	backends.push(Backend::from(WinMLBackend::default()));
	}
	backends

WinML backend for wasi-nn #7807

WinML backend for wasi-nn #7807

Conversation

jianjunz commented Jan 24, 2024

abrown commented Jan 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianjunz commented Feb 11, 2024

abrown left a comment

Choose a reason for hiding this comment

jianjunz commented Feb 28, 2024 • edited Loading

jianjunz commented Mar 1, 2024

jianjunz commented Feb 28, 2024 •

edited

Loading