feat: implement udf server in databend

[gitccl]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

This pr merges previous #12168 , #12417 and #12463 .
In this pr, we also change the syntax of create function command to make it similar to snowflake's syntax.

CREATE FUNCTION [IF NOT EXISTS] <udf_name> (<arg_type>, ...) RETURNS <return_type> LANGUAGE <language> HANDLER=<handler> ADDRESS=<udf_server_address>

• Closes Tracking: Implement Databend UDF Server #12177

---

This change is 

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
databend	⬜️ Ignored (Inspect)	Visit Preview		Sep 17, 2023 3:02am

[drmingdrmer]

Reviewed 3 of 71 files at r1, 8 of 57 files at r2, all commit messages.
Reviewable status: 11 of 72 files reviewed, all discussions resolved

[xudong963]

I'll review the PR today

[xudong963]

LGTM, great work @gitccl

Currently, our Python UDF syntax is similar to snowflake:

Snowflake:

create or replace function addone(i int)
returns int
language python
runtime_version = '3.8'
handler = 'addone_py'
as
$$
def addone_py(i):
  return i+1
$$;

Databend:

CREATE FUNCTION split_and_join (VARCHAR, VARCHAR, VARCHAR) RETURNS VARCHAR LANGUAGE python HANDLER = 'split_and_join' ADDRESS = 'http://0.0.0.0:8815';

@BohuTANG @sundy-li Any thoughts?

[xudong963]

Given the security concerns we discussed internally, maybe we still need to add a flag to make it a feature, so we can avoid our cloud attacking. cc @Xuanwo @flaneur2020 @BohuTANG

[xudong963]

cc @soyeric128 After the PR merged, need to add a doc. You can refer the README: https://github.com/datafuselabs/databend/pull/12729/files#diff-bf3534b9c183942327d5e8579cec90a1634bc242312a7cff036d80c035d11d98

[xudong963]

Seems the ee test isn't related to the PR

[BohuTANG]

Given the security concerns we discussed internally, maybe we still need to add a flag to make it a feature, so we can avoid our cloud attacking. cc @Xuanwo @flaneur2020 @BohuTANG

Add a new query config is ok, then we can enable it for some users. @flaneur2020

[gitccl]

It seems that ci is stuck.

[Xuanwo]

It seems that ci is stuck.

Please fix the conflicts first to make CI running, thanks!

[xudong963]

A flaky test cc @TCeason

[TCeason]

A flaky test cc @TCeason

Get

[xudong963]

Databend Query start failure, cause: Code: 1002, Text = TOML parse error at line 41, column 1
553
   |
554
41 | enable_udf_server = true
555
   | ^^^^^^^^^^^^^^^^^
556
unknown field `enable_udf_server`, expected one of `tenant_id`, `cluster_id`, `num_cpus`, `mysql_handler_host`, `mysql_handler_port`, `mysql_handler_tcp_keepalive_timeout_secs`, `max_active_sessions`, `max_server_memory_usage`, `max_memory_limit_enabled`, `clickhouse_handler_host`, `clickhouse_handler_port`, `clickhouse_http_handler_host`, `clickhouse_http_handler_port`, `http_handler_host`, `http_handler_port`, `http_handler_result_timeout_secs`, `flight_sql_handler_host`, `flight_sql_handler_port`, `flight_api_address`, `admin_api_address`, `metric_api_address`, `http_handler_tls_server_cert`, `http_handler_tls_server_key`, `http_handler_tls_server_root_ca_cert`, `flight_sql_tls_server_cert`, `flight_sql_tls_server_key`, `api_tls_server_cert`, `api_tls_server_key`, `api_tls_server_root_ca_cert`, `rpc_tls_server_cert`, `rpc_tls_server_key`, `rpc_tls_query_server_root_ca_cert`, `rpc_tls_query_service_domain_name`, `table_engine_memory_enabled`, `wait_timeout_mills`, `max_query_log_size`, `management_mode`, `jwt_key_file`, `jwt_key_files`, `default_storage_format`, `default_compression`, `users`, `share_endpoint_address`, `share_endpoint_auth_token_file`, `quota`, `internal_enable_sandbox_tenant`, `internal_merge_on_read_mutation`, `table_disk_cache_mb_size`, `table_meta_cache_enabled`, `table_cache_block_meta_count`, `table_memory_cache_mb_size`, `table_disk_cache_root`, `table_cache_snapshot_count`, `table_cache_statistic_count`, `table_cache_segment_count`, `table_cache_bloom_index_meta_count`, `table_cache_bloom_index_filter_count`, `table_cache_bloom_index_data_bytes`, `disable_system_table_load`, `openai_api_key`
557
, source: None
558
TOML parse error at line 41, column 1

cc @gitccl

[gitccl]

Databend Query start failure, cause: Code: 1002, Text = TOML parse error at line 41, column 1
553
   |
554
41 | enable_udf_server = true
555
   | ^^^^^^^^^^^^^^^^^
556
unknown field `enable_udf_server`, expected one of `tenant_id`, `cluster_id`, `num_cpus`, `mysql_handler_host`, `mysql_handler_port`, `mysql_handler_tcp_keepalive_timeout_secs`, `max_active_sessions`, `max_server_memory_usage`, `max_memory_limit_enabled`, `clickhouse_handler_host`, `clickhouse_handler_port`, `clickhouse_http_handler_host`, `clickhouse_http_handler_port`, `http_handler_host`, `http_handler_port`, `http_handler_result_timeout_secs`, `flight_sql_handler_host`, `flight_sql_handler_port`, `flight_api_address`, `admin_api_address`, `metric_api_address`, `http_handler_tls_server_cert`, `http_handler_tls_server_key`, `http_handler_tls_server_root_ca_cert`, `flight_sql_tls_server_cert`, `flight_sql_tls_server_key`, `api_tls_server_cert`, `api_tls_server_key`, `api_tls_server_root_ca_cert`, `rpc_tls_server_cert`, `rpc_tls_server_key`, `rpc_tls_query_server_root_ca_cert`, `rpc_tls_query_service_domain_name`, `table_engine_memory_enabled`, `wait_timeout_mills`, `max_query_log_size`, `management_mode`, `jwt_key_file`, `jwt_key_files`, `default_storage_format`, `default_compression`, `users`, `share_endpoint_address`, `share_endpoint_auth_token_file`, `quota`, `internal_enable_sandbox_tenant`, `internal_merge_on_read_mutation`, `table_disk_cache_mb_size`, `table_meta_cache_enabled`, `table_cache_block_meta_count`, `table_memory_cache_mb_size`, `table_disk_cache_root`, `table_cache_snapshot_count`, `table_cache_statistic_count`, `table_cache_segment_count`, `table_cache_bloom_index_meta_count`, `table_cache_bloom_index_filter_count`, `table_cache_bloom_index_data_bytes`, `disable_system_table_load`, `openai_api_key`
557
, source: None
558
TOML parse error at line 41, column 1

cc @gitccl

This problem is caused by the 1.0.56 version of databend-query loading the latest configuration, but version 1.0.56 does not support the enable_udf_server configuration. Is it expected to use the old version databend-query to load the new version configuration?

[sundy-li]

Is it expected to use the old version databend-query to load the new version configuration?

Currently, it's forbidden. Maybe we should fetch the old config file from git to start the old version process.

[sundy-li]

We already use the old config files in https://github.com/datafuselabs/databend/blob/1d19af8b180b3b88f3e9eba9837d31815c9c9364/tests/compat/test-compat.sh

[gitccl]

We already use the old config files in https://github.com/datafuselabs/databend/blob/1d19af8b180b3b88f3e9eba9837d31815c9c9364/tests/compat/test-compat.sh

test_compat_fuse use this scripts: https://github.com/datafuselabs/databend/blob/1d19af8b180b3b88f3e9eba9837d31815c9c9364/tests/fuse-compat/util.sh