[Feature][UDF][WASM] support wasm udf #24504

taptao · 2023-09-17T13:38:34Z

Proposed changes

Issue Number: close #20267

Support wasm udf via wasmtime library.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions

clang-tidy made some suggestions

be/test/util/wasm_manager_test.cpp

be/test/util/wasm_test.cpp

dataroaring · 2023-10-03T13:05:35Z

run buildall

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 37. Check the log or trigger a new build to see more.

be/src/runtime/exec_env.h

be/src/util/wasm_manager.cpp

be/src/util/wasm_manager.h

be/src/vec/functions/function_wasm.h

function_json.cpp

github-actions

clang-tidy made some suggestions

function_json.cpp

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 35. Check the log or trigger a new build to see more.

be/src/runtime/user_function_cache.cpp

be/src/util/wasm_manager.cpp

function_json.cpp

github-actions · 2023-12-13T12:24:53Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In thirdparty/vars.sh line 494:
WASMTIME_C++_DOWNLOAD="https://github.com/bytecodealliance/wasmtime-cpp/archive/refs/tags/v9.0.0.tar.gz"
^-- SC2276 (error): This is interpreted as a command name containing '='. Bad assignment or comparison?


In thirdparty/vars.sh line 495:
WASMTIME_C++_NAME=v9.0.0.tar.gz
^-----------------------------^ SC2276 (error): This is interpreted as a command name containing '='. Bad assignment or comparison?


In thirdparty/vars.sh line 496:
WASMTIME_C++_SOURCE=wasmtime-cpp-9.0.0
^-- SC2276 (error): This is interpreted as a command name containing '='. Bad assignment or comparison?


In thirdparty/vars.sh line 497:
WASMTIME_C++_MD5SUM="b440d7efd615f1b018cd326876d8ead7"
^-- SC2276 (error): This is interpreted as a command name containing '='. Bad assignment or comparison?

For more information:
  https://www.shellcheck.net/wiki/SC2276 -- This is interpreted as a command ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions

clang-tidy made some suggestions

be/test/util/wasm_test.cpp

taptao · 2023-12-13T13:52:25Z

run buildall

taptao · 2023-12-13T14:23:06Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/vec/functions/function_wasm.cpp

taptao · 2023-12-14T01:43:49Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/vec/functions/function_wasm.cpp

fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java

github-actions

clang-tidy made some suggestions

be/src/util/wasm_manager.cpp

be/src/vec/functions/function_wasm.cpp

github-actions

clang-tidy made some suggestions

be/src/vec/functions/function_wasm.cpp

HappenLee · 2024-01-23T17:00:58Z

be/src/util/wasm_manager.h

+class WasmFunctionManager {
+private:
+    // wasmtime
+    wasmtime::Engine* engine;


why not use std::unique_ptr ?

zhangstar333 · 2024-01-23T02:43:52Z

fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java

+            try (Store<Void> store = Store.withoutData()) {
+                try (Engine engine = store.engine();
+                        Module module = Module.fromFile(engine, url.getFile())) {
+                    return;


could get the function name, input and return params from module?
Then compare it with the create statement to see if it is consistent

I tried it before, but wasmtime-java jar doesn't support this method. I asked the wasmtime-java author this question, but there has been no reply so far.

taptao · 2024-01-29T03:11:34Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-01-29T03:21:38Z

be/src/util/wasm_manager.cpp

+bool WasmFunctionManager::DeleteFunction(std::string functionName) {
+    auto funcBody = funcs.find(functionName);
+    if (funcBody == funcs.end()) {
+        return false;


warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]

be/src/util/wasm_manager.cpp:80:

- if (funcBody == funcs.end()) { - return false; - } - funcs.erase(functionName); - return true; + return !funcBody == funcs.end();

github-actions · 2024-01-29T03:21:38Z

be/src/vec/functions/function_wasm.cpp

+    return Status::OK();
+}
+
+Status FunctionWasm::execute(FunctionContext* context, Block& block, const ColumnNumbers& arguments,


warning: function 'execute' exceeds recommended size/complexity thresholds [readability-function-size]

Status FunctionWasm::execute(FunctionContext* context, Block& block, const ColumnNumbers& arguments, ^

Additional context

be/src/vec/functions/function_wasm.cpp:69: 116 lines including whitespace and comments (threshold 80)

Status FunctionWasm::execute(FunctionContext* context, Block& block, const ColumnNumbers& arguments, ^

github-actions · 2024-01-29T03:21:38Z

be/src/vec/functions/function_wasm.h

+    Status execute(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+                   size_t result, size_t input_rows_count, bool dry_run = false) override;
+
+    bool is_deterministic() const override { return false; }


warning: method 'is_deterministic' can be made static [readability-convert-member-functions-to-static]

Suggested change

bool is_deterministic() const override { return false; }

static bool is_deterministic() override { return false; }

github-actions · 2024-01-29T03:21:38Z

be/src/vec/functions/function_wasm.h

+
+    bool is_deterministic() const override { return false; }
+
+    bool is_deterministic_in_scope_of_query() const override { return false; }


warning: method 'is_deterministic_in_scope_of_query' can be made static [readability-convert-member-functions-to-static]

Suggested change

bool is_deterministic_in_scope_of_query() const override { return false; }

static bool is_deterministic_in_scope_of_query() override { return false; }

taptao · 2024-01-30T01:20:59Z

run buildall

amorynan · 2024-02-01T02:45:11Z

run buildall

taptao · 2024-02-01T08:42:29Z

reopen

taptao · 2024-02-01T08:55:25Z

run buildall

taptao · 2024-02-01T11:26:04Z

run buildall

taptao · 2024-02-01T15:45:06Z

run buildall

taptao · 2024-02-01T23:41:58Z

run buildall

taptao · 2024-02-04T09:18:56Z

Move to #30789 to view WASM PR

github-actions bot reviewed Sep 17, 2023

View reviewed changes

be/test/util/wasm_manager_test.cpp Show resolved Hide resolved

be/test/util/wasm_test.cpp Outdated Show resolved Hide resolved

github-actions bot added the meta-change label Dec 9, 2023

github-actions bot reviewed Dec 9, 2023

View reviewed changes

taptao force-pushed the wasm-udf branch from 869d533 to 073998e Compare December 9, 2023 05:53

github-actions bot reviewed Dec 9, 2023

View reviewed changes

taptao force-pushed the wasm-udf branch 2 times, most recently from 805d34a to 34dfaf3 Compare December 10, 2023 08:59

github-actions bot reviewed Dec 10, 2023

View reviewed changes

taptao force-pushed the wasm-udf branch from 34dfaf3 to 8b78264 Compare December 13, 2023 12:23

taptao force-pushed the wasm-udf branch from 8b78264 to 815979a Compare December 13, 2023 12:29

github-actions bot reviewed Dec 13, 2023

View reviewed changes

be/test/util/wasm_test.cpp Show resolved Hide resolved

taptao force-pushed the wasm-udf branch from f7c9565 to 12718cf Compare December 13, 2023 13:38

taptao force-pushed the wasm-udf branch 2 times, most recently from 21c82e5 to 6896d89 Compare December 13, 2023 14:09

taptao force-pushed the wasm-udf branch 2 times, most recently from 347f632 to 61daae1 Compare December 13, 2023 15:07

github-actions bot reviewed Dec 13, 2023

View reviewed changes

be/src/vec/functions/function_wasm.cpp Outdated Show resolved Hide resolved

taptao force-pushed the wasm-udf branch from 61daae1 to 7222af7 Compare December 15, 2023 12:51

github-actions bot reviewed Dec 15, 2023

View reviewed changes

be/src/vec/functions/function_wasm.cpp Outdated Show resolved Hide resolved

taptao force-pushed the wasm-udf branch from 7222af7 to 8ce0da5 Compare December 16, 2023 02:09

zhangstar333 reviewed Dec 19, 2023

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java Show resolved Hide resolved

taptao force-pushed the wasm-udf branch from 8ce0da5 to 7a05541 Compare January 8, 2024 10:20

github-actions bot reviewed Jan 8, 2024

View reviewed changes

be/src/util/wasm_manager.cpp Show resolved Hide resolved

be/src/vec/functions/function_wasm.cpp Show resolved Hide resolved

be/src/vec/functions/function_wasm.cpp Show resolved Hide resolved

taptao force-pushed the wasm-udf branch from 7a05541 to bd45f1a Compare January 14, 2024 07:10

github-actions bot reviewed Jan 14, 2024

View reviewed changes

be/src/vec/functions/function_wasm.cpp Show resolved Hide resolved

HappenLee reviewed Jan 23, 2024

View reviewed changes

zhangstar333 reviewed Jan 24, 2024

View reviewed changes

taptao requested review from zhangstar333 and HappenLee January 24, 2024 13:45

taptao force-pushed the wasm-udf branch from 06e306e to a020817 Compare January 29, 2024 03:11

github-actions bot reviewed Jan 29, 2024

View reviewed changes

taptao mentioned this pull request Jan 30, 2024

[dep](thirdparty) import wasmtime library #30591

Merged

dataroaring closed this in #30591 Feb 1, 2024

dataroaring reopened this Feb 1, 2024

taptao added 7 commits February 1, 2024 16:54

[feature][udf](wasm) add wasmtime-manager

93f6378

[feature][udf](wasm) support wasm udf in FE

6f5c72f

[feature][udf](wasm) support wasm in be

7462a13

[feature-wip][udf] regression

910e59c

[feature][udf](wasm) add doc

8b676ce

[feature][udf](wasm) fix review comment

a79e664

[feature][udf] add TODO check wasm udf param

c8cb630

taptao force-pushed the wasm-udf branch from 6dbc046 to c8cb630 Compare February 1, 2024 08:55

[fix](wasm): fix build error

5a1a1f1

Merge branch 'master' into wasm-udf

2427661

Merge branch 'master' into wasm-udf

b51ab67

taptao closed this Feb 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][UDF][WASM] support wasm udf #24504

[Feature][UDF][WASM] support wasm udf #24504

taptao commented Sep 17, 2023

github-actions bot left a comment

dataroaring commented Oct 3, 2023

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot commented Dec 13, 2023

github-actions bot left a comment

taptao commented Dec 13, 2023

taptao commented Dec 13, 2023

github-actions bot left a comment

taptao commented Dec 14, 2023

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

HappenLee Jan 23, 2024

taptao Jan 24, 2024

zhangstar333 Jan 23, 2024

taptao Jan 24, 2024

zhangstar333 Jan 25, 2024

taptao commented Jan 29, 2024

github-actions bot left a comment

github-actions bot Jan 29, 2024

github-actions bot Jan 29, 2024

github-actions bot Jan 29, 2024

github-actions bot Jan 29, 2024

taptao commented Jan 30, 2024

amorynan commented Feb 1, 2024

taptao commented Feb 1, 2024

taptao commented Feb 1, 2024

taptao commented Feb 1, 2024

taptao commented Feb 1, 2024

taptao commented Feb 1, 2024

taptao commented Feb 4, 2024

	bool is_deterministic() const override { return false; }
	static bool is_deterministic() override { return false; }


		bool is_deterministic() const override { return false; }

		bool is_deterministic_in_scope_of_query() const override { return false; }

	bool is_deterministic_in_scope_of_query() const override { return false; }
	static bool is_deterministic_in_scope_of_query() override { return false; }

[Feature][UDF][WASM] support wasm udf #24504

[Feature][UDF][WASM] support wasm udf #24504

Conversation

taptao commented Sep 17, 2023

Proposed changes

Further comments

github-actions bot left a comment

Choose a reason for hiding this comment

dataroaring commented Oct 3, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 13, 2023

sh-checker report

github-actions bot left a comment

Choose a reason for hiding this comment

taptao commented Dec 13, 2023

taptao commented Dec 13, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

taptao commented Dec 14, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

HappenLee Jan 23, 2024

Choose a reason for hiding this comment

taptao Jan 24, 2024

Choose a reason for hiding this comment

zhangstar333 Jan 23, 2024

Choose a reason for hiding this comment

taptao Jan 24, 2024

Choose a reason for hiding this comment

zhangstar333 Jan 25, 2024

Choose a reason for hiding this comment

taptao commented Jan 29, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Jan 29, 2024

Choose a reason for hiding this comment

github-actions bot Jan 29, 2024

Choose a reason for hiding this comment

github-actions bot Jan 29, 2024

Choose a reason for hiding this comment

github-actions bot Jan 29, 2024

Choose a reason for hiding this comment

taptao commented Jan 30, 2024

amorynan commented Feb 1, 2024

taptao commented Feb 1, 2024

taptao commented Feb 1, 2024

taptao commented Feb 1, 2024

taptao commented Feb 1, 2024

taptao commented Feb 1, 2024

taptao commented Feb 4, 2024

`sh-checker report`