[pick](branch-3.0) #38215 #43281 #43960 #44244
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…when BE uses public vip (apache#43281) If there is a Doris cluster, its FE node can be accessed by the external network, and all its BE nodes can only be accessed by the internal network. This is fine when using Mysql client and JDBC to connect to Doris to execute queries, and the query results will be returned to the client by the Doris FE node. However, using Arrow Flight SQL to connect to Doris cannot execute queries, because the ADBC client needs to connect to the Doris BE node to pull query results, but the Doris BE node is not allowed to be accessed by the external network. In a production environment, it is often inconvenient to expose Doris BE nodes to the external network. However, a reverse proxy (such as nginx) can be added to all Doris BE nodes, and the external client will be randomly routed to a Doris BE node when connecting to nginx. The query results of Arrow Flight SQL will be randomly saved on a Doris BE node. If it is different from the Doris BE node randomly routed by nginx, data forwarding needs to be done inside the Doris BE node. 1. The Ticket returned by Doris FE Arrow Flight Server to ADBC client contains the IP and Brpc Port of the Doris BE node where the query result is located. 2. Doris BE Arrow Flight Server receives a request to pull data. If the IP:BrpcPort in the Ticket is not itself, it pulls the query result Block from the Doris BE node specified by IP:BrpcPort, converts it to Arrow Batch and returns it to ADBC Client; if the IP:BrpcPort in the Ticket is itself, it is the same as before. 1. If the data is not in the current BE node, you can pull the data from other BE nodes asynchronously and cache at least one Block locally in the current BE node, which will reduce the time consumption of serialization, deserialization, and RPC. 1. Create a Doris cluster with 1 FE and 2 BE, and modify `arrow_flight_sql_port` in `fe.conf` and `be.conf`. 2. Root executes `systemctl status nginx` to check whether Nginx is installed. If not, yum install is recommended. 3. `vim /etc/nginx/nginx.conf` adds `underscores_in_headers on;` 4. `touch /etc/nginx/conf.d/arrowflight.conf` creates a file, and `vim /etc/nginx/conf.d/arrowflight.conf` adds: ``` upstream arrowflight { server {BE1_ip}:{BE1_arrow_flight_sql_port}; server {BE2_IP}:{BE2_arrow_flight_sql_port}; } server { listen {nginx port} http2; listen [::]:{nginx port} http2; server_name doris.arrowflight.com; #ssl_certificate /etc/nginx/cert/myCA.pem; #ssl_certificate_key /etc/nginx/cert/myCA.key; location / { grpc_pass grpc://arrowflight; grpc_set_header X-Real-IP $remote_addr; grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for; grpc_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 60s; proxy_send_timeout 60s; #proxy_http_version 1.1; #proxy_set_header Connection ""; } } ``` Where {BE1_ip}:{BE1_arrow_flight_sql_port} is the IP of BE 1 and arrow_flight_sql_port in be.conf, and similarly {BE2_IP}:{BE2_arrow_flight_sql_port}. `{nginx port}` is any available port. 6. Add in be.conf of all BEs: ``` public_access_ip={nginx ip} public_access_port={nginx port} ``` --- 如果存在一个 Doris 集群，它的 FE 节点可以被外部网络访问，它的所有 BE 节点只可以被内网访问。 这在使用 Mysql client 和 JDBC 连接 Doris 执行查询是没问题的，查询结果将由 Doris FE 节点返回给 client。 但使用 Arrow Flight SQL 连接 Doris 无法执行查询，因为 ADBC client 需要连接 Doris BE 节点拉取查询结果，但 Doris BE 节点不允许被外网访问。 生产环境中，很多时候不方便在外网暴露 Doris BE 节点。但可以为所有 Doris BE 节点增加了一层反向代理（比如 nginx），外网的 client 连接 nginx 时会随机路由到一台 Doris BE 节点上。 Arrow Flight SQL 查询结果会随机保存在一台 Doris BE 节点上，如果和 nginx 随机路由的 Doris BE 节点不同，需要在 Doris BE 节点内部做一次数据转发。 1. Doris FE Arrow Flight Server 向 ADBC client 返回的 Ticket 中包含查询结果所在 Doris BE节点的 IP 和 Brpc Port。 2. Doris BE Arrow Flight Server 收到拉取数据请求。如果 Ticket 中的 IP:BrpcPort 不是自己，则从 IP:BrpcPort 指定的 Doris BE 节点拉取查询结果Block，转为 Arrow Batch 后返回 ADBC Client；如果 Ticket 中的 IP:BrpcPort 是自己，则和过去一样。 1. 若数据不在当前 BE 节点，可以异步的从其他 BE 节点拉取数据，并在当前 BE 节点本地缓存至少一个 Block，这将减少序列化、反序列化、RPC 的耗时。 1. 创建一个 1 FE 和 2 BE 的 Doris 集群，修改 `fe.conf` 和 `be.conf` 中的 `arrow_flight_sql_port`。 2. Root 执行 `systemctl status nginx` 查看是否安装 Nginx，若没有则推荐 yum install。 3. `vim /etc/nginx/nginx.conf` 增加 `underscores_in_headers on;` 4. `touch /etc/nginx/conf.d/arrowflight.conf` 创建文件，`vim /etc/nginx/conf.d/arrowflight.conf` 增加： ``` upstream arrowflight { server {BE1_ip}:{BE1_arrow_flight_sql_port}; server {BE2_IP}:{BE2_arrow_flight_sql_port}; } server { listen {nginx port} http2; listen [::]:{nginx port} http2; server_name doris.arrowflight.com; #ssl_certificate /etc/nginx/cert/myCA.pem; #ssl_certificate_key /etc/nginx/cert/myCA.key; location / { grpc_pass grpc://arrowflight; grpc_set_header X-Real-IP $remote_addr; grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for; grpc_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 60s; proxy_send_timeout 60s; #proxy_http_version 1.1; #proxy_set_header Connection ""; } } ``` 其中 {BE1_ip}:{BE1_arrow_flight_sql_port} 是 BE 1 的 IP 和 be.conf 中的 arrow_flight_sql_port，同理 {BE2_IP}:{BE2_arrow_flight_sql_port}。`{nginx port}` 是一个任意可用端口。 6. 在所有 BE 的 be.conf 中增加 ``` public_access_ip={nginx ip} public_access_port={nginx port} ```
…43960) ### What problem does this PR solve? Problem Summary: After query first phase `exec_plan_fragment`, FE will fetches arrow schema to BE, but BE will generate arrow schema when query second stage `ResultSinkLocalState::open`. Therefore, this pr is changed to generate arrow schema in the first phase `ResultSinkLocalState::init`. Fix: ``` rrmsg: Status [errorCode=NOT_FOUND, errorMsg=(172.16.212.191)[NOT_FOUND]FE not found arrow flight schema, maybe query has been canceled], error code: null, error msg: java.lang.RuntimeException: fetch arrow flight schema failed, finstId: 3573efbeb10c44a7-956531d8e15d1630, errmsg: Status [errorCode=NOT_FOUND, errorMsg=(172.16.212.191)[NOT_FOUND]FE not found arrow flight schema, maybe query has been canceled] at org.apache.doris.service.arrowflight.FlightSqlConnectProcessor.fetchArrowFlightSchema(FlightSqlConnectProcessor.java:126) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.service.arrowflight.DorisFlightSqlProducer.executeQueryStatement(DorisFlightSqlProducer.java:229) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.service.arrowflight.DorisFlightSqlProducer.getFlightInfoStatement(DorisFlightSqlProducer.java:260) ~[doris-fe.jar:1.2-SNAPSHOT] ```
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
run buildall |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
pick #38215 #43281 #43960