[Feature](network interface) Support network interface #16617

ZashJie · 2023-02-10T10:04:04Z

Proposed changes

Support network_interface in Apache Doris, the main changes are:

If network_interface is configured in the configuration file, use network_interface configuration

Issue Number: close #xxx

Problem summary

Describe your changes.

Checklist(Required)

Does it affect the original behavior:
- Yes
- No
- I don't know
Has unit tests been added:
- Yes
- No
- No Need
Has document been added or modified:
- Yes
- No
- No Need
Does it need to update dependencies:
- Yes
- No
Are there any changes that cannot be rolled back:
- Yes (If Yes, please explain WHY)
- No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-02-10T10:06:54Z

clang-tidy review says "All clean, LGTM! 👍"

hf200012 · 2023-02-10T10:32:48Z

title style ：[<type>](<scope>) <subject>
https://doris.apache.org/zh-CN/community/how-to-contribute/commit-format-specification
@ZashJie

dataroaring · 2023-02-10T10:55:09Z

be/src/service/backend_options.cpp

-    LOG(INFO) << "priority cidrs in conf: " << config::priority_networks;
-
+// 解析IP为cidr表达方式
+bool BackendOptions::analyze_cidrs(const std::string & ip) {


There is no need to modify this function.

be/src/service/backend_options.cpp

Yukang-Lian · 2023-02-10T14:16:33Z

be/src/service/backend_options.cpp

-    }
-    LOG(INFO) << "priority cidrs in conf: " << config::priority_networks;
-
+// 解析IP为cidr表达方式


please use english comment.

Yukang-Lian

please change your PR title format.

Yukang-Lian · 2023-02-10T14:16:45Z

be/src/service/backend_options.cpp

@@ -101,6 +100,61 @@ bool BackendOptions::analyze_priority_cidrs() {
    return true;
 }

+
+//从network_interface中获取IP


Yukang-Lian · 2023-02-10T14:17:06Z

be/src/service/backend_options.cpp

+    //     }
+    //     _s_priority_cidrs.push_back(cidr);
+    // }
+    // return true;


useless code?

Yukang-Lian · 2023-02-10T14:17:18Z

be/src/service/backend_options.cpp

+    //     _s_priority_cidrs.push_back(cidr);
+    // }
+    // return true;
+}


Yukang-Lian · 2023-02-10T14:18:10Z

conf/be.conf


 # data root path, separate by ';'
 # you can specify the storage medium of each root path, HDD or SSD
 # you can add capacity limit at the end of each root path, separate by ','
 # eg:
-# storage_root_path = /home/disk1/doris.HDD,50;/home/disk2/doris.SSD,1;/home/disk2/doris
+# storage_root_path = /root/git/output/be/storage;/home/disk2/doris.SSD,1;/home/disk2/doris
 # /home/disk1/doris.HDD, capacity limit is 50GB, HDD;
 # /home/disk2/doris.SSD, capacity limit is 1GB, SSD;


modify this file in your environment, but do not commit it.

Yukang-Lian · 2023-02-10T14:18:16Z

conf/fe.conf

@@ -52,7 +52,7 @@ mysql_service_nio_enabled = true
 # If no ip match this rule, will choose one randomly.
 # use CIDR format, e.g. 10.10.10.0/24
 # Default value is empty.
-# priority_networks = 10.10.10.0/24;192.168.0.0/16
+priority_networks = 127.0.0.1/16

 # Advanced configurations 
 # log_roll_size_mb = 1024


github-actions · 2023-02-11T10:31:10Z

clang-tidy review says "All clean, LGTM! 👍"

morningman · 2023-02-13T01:21:27Z

Please amend your PR title and add describe in your PR comment.
And I suggest to wait #14063 merge

dataroaring · 2023-02-15T01:52:21Z

be/src/common/config.h

@@ -49,6 +49,8 @@ CONF_Int32(single_replica_load_brpc_num_threads, "64");
 // If no ip match this rule, will choose one randomly.
 CONF_String(priority_networks, "");

+CONF_Strings(network_interface, "");


String is enough.

dataroaring · 2023-02-15T01:52:47Z

be/src/service/backend_options.cpp

+        return false;
+    }
+    LOG(INFO) << "network name in conf: " << config::network_interface[0] << "\tpriority cidrs in conf: " << config::network_interface[1];
+


we should not print cidrs here, it is confusing.

dataroaring · 2023-02-15T01:53:11Z

be/src/service/backend_options.cpp

+    }
+    LOG(INFO) << "network name in conf: " << config::network_interface[0] << "\tpriority cidrs in conf: " << config::network_interface[1];
+
+    std::vector<std::string> cidr_strs =


variable name is confusing.

github-actions

clang-tidy made some suggestions

github-actions · 2023-03-01T09:20:26Z

be/src/service/backend_options.cpp

+    }
+    LOG(INFO) << "network name in conf: " << config::network_interfaces;
+
+    struct ifaddrs * ifAddrStruct = NULL;


warning: use nullptr [modernize-use-nullptr]

Suggested change

struct ifaddrs * ifAddrStruct = NULL;

struct ifaddrs * ifAddrStruct = nullptr;

github-actions · 2023-03-01T09:20:26Z

be/src/service/backend_options.cpp

+    LOG(INFO) << "network name in conf: " << config::network_interfaces;
+
+    struct ifaddrs * ifAddrStruct = NULL;
+    void * tmpAddrPtr = NULL;


warning: use nullptr [modernize-use-nullptr]

Suggested change

void * tmpAddrPtr = NULL;

void * tmpAddrPtr = nullptr;

github-actions · 2023-03-01T09:20:26Z

be/src/service/backend_options.cpp

+    bool flag = false;
+
+    for (auto& name_str : name_strs) {
+        while (ifAddrStruct!=NULL) {


warning: use nullptr [modernize-use-nullptr]

Suggested change

while (ifAddrStruct!=NULL) {

while (ifAddrStruct!=nullptr) {

dataroaring · 2023-03-02T02:24:10Z

be/src/service/backend_options.cpp

+    if (config::network_interfaces == "") {
+        return true;
+    }
+    LOG(INFO) << "network name in conf: " << config::network_interfaces;


network interfaces in conf:

dataroaring · 2023-03-02T02:25:06Z

be/src/service/backend_options.cpp

+
+    std::vector<std::string> name_strs =
+            strings::Split(config::network_interfaces, PRIORITY_CIDR_SEPARATOR);
+    bool flag = false;


found may be a better name here.

dataroaring · 2023-03-02T02:25:29Z

be/src/service/backend_options.cpp

+
+    getifaddrs(&ifAddrStruct); 
+
+    std::vector<std::string> name_strs =


nic_names is a better var name.

dataroaring · 2023-03-02T02:30:54Z

be/src/service/backend_options.cpp

+
+    for (auto& name_str : name_strs) {
+        while (ifAddrStruct!=NULL) {
+            if (ifAddrStruct->ifa_addr->sa_family==AF_INET) { // check it is IP4


we should put a space around !=, ==, +, - etc. liking while (ifAddrStruct != nullptr)

for (; if_addr_struct != nullptr; if_addr_struct = if_addr_struct->ifa_next) {
...
}

use if_addr_struct instead of ifAddrStruct.

dataroaring · 2023-03-02T02:31:19Z

be/src/service/backend_options.cpp

+            if (ifAddrStruct->ifa_addr->sa_family==AF_INET) { // check it is IP4
+                // is a valid IP4 Address
+                if (name_str == ifAddrStruct->ifa_name) {
+                    tmpAddrPtr=&((struct sockaddr_in *)ifAddrStruct->ifa_addr)->sin_addr;


tmpAddrPtr = &...

dataroaring · 2023-03-02T08:04:04Z

be/src/service/backend_options.cpp

+            } else if (!_s_net_interfaces.empty()) {
+                if (is_in_net_interface(addr_it->get_host_address_v4())) {
+                    _s_localhost = addr_it->get_host_address_v4();
+                }


We should not follow cidrs here.

There is a problem here, e.g. There are 2 nics, eth0 and eth1, we configed eth1 and cidrs, if we tranverse eth0 first then cidr is used. we should let network interface is selected in priority.

So we should handle net_interface before this while.

dataroaring · 2023-03-02T08:04:30Z

be/src/service/backend_options.h

@@ -21,6 +21,8 @@

 #include <string>
 #include <vector>
+#include <ifaddrs.h>
+#include <arpa/inet.h>


put the two .h to .cpp .

dataroaring · 2023-03-02T08:04:54Z

fe/fe-core/src/main/java/org/apache/doris/service/FrontendOptions.java

+                    if (isInNetInterface(addr.getHostAddress())) {
+                        localAddr = addr;
+                        break;
+                    }


same as be.

dataroaring

Please have a look at the comments.

github-actions

clang-tidy made some suggestions

github-actions · 2023-03-03T08:29:44Z

be/src/service/backend_options.cpp

+    }
+    LOG(INFO) << "network interfaces in conf: " << config::network_interfaces;
+
+    struct ifaddrs * if_Addr_Struct = NULL;


warning: use nullptr [modernize-use-nullptr]

Suggested change

struct ifaddrs * if_Addr_Struct = NULL;

struct ifaddrs * if_Addr_Struct = nullptr;

github-actions

clang-tidy made some suggestions

github-actions · 2023-03-05T02:57:12Z

be/src/service/backend_options.cpp

+    }
+    LOG(INFO) << "network interfaces in conf: " << config::network_interfaces;
+
+    struct ifaddrs * if_addr_struct = NULL;


warning: use nullptr [modernize-use-nullptr]

Suggested change

struct ifaddrs * if_addr_struct = NULL;

struct ifaddrs * if_addr_struct = nullptr;

github-actions · 2023-03-05T13:38:39Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring

LGTM

github-actions · 2023-03-06T05:07:52Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-03-06T05:21:00Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-03-06T05:22:40Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-03T00:09:02Z

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!

dataroaring requested changes Feb 10, 2023

View reviewed changes

Yukang-Lian reviewed Feb 10, 2023

View reviewed changes

Yukang-Lian suggested changes Feb 10, 2023

View reviewed changes

ZashJie changed the title ~~ZhangShunjieNet interface~~ [Feature](network interface) Support network interface Feb 13, 2023

dataroaring requested changes Feb 15, 2023

View reviewed changes

Yukang-Lian mentioned this pull request Feb 27, 2023

[Good First Issue] Doris' Future #17176

Open

interface 3.1

e7b03f7

Yukang-Lian added the doris-future label Mar 1, 2023

ZashJie force-pushed the net_interface branch from c18a6f0 to e7b03f7 Compare March 1, 2023 09:17

github-actions bot reviewed Mar 1, 2023

View reviewed changes

dataroaring requested changes Mar 2, 2023

View reviewed changes

Network Interface

4a91dc4

github-actions bot reviewed Mar 3, 2023

View reviewed changes

ZashJie added 2 commits March 3, 2023 16:50

Network InterfaceI

2d7b77a

network_interface_configmd

037688d

github-actions bot added the kind/docs Categorizes issue or PR as related to documentation. label Mar 4, 2023

network_interface_be_style

76e53df

github-actions bot reviewed Mar 5, 2023

View reviewed changes

ZashJie added 2 commits March 5, 2023 14:07

network_interfaces_docs

1a9d1e1

network_interface_modify

696f09a

dataroaring previously approved these changes Mar 5, 2023

View reviewed changes

ZashJie added 3 commits March 6, 2023 01:37

network_interface_rebase

952b25a

network_interface_rebaseipv6

5e5092a

network_interface_3_6_13

e7c3357

ZashJie dismissed dataroaring’s stale review via e7c3357 March 6, 2023 05:04

ZashJie added 2 commits March 6, 2023 13:17

network_interface_beformat

9033644

network_interface_beformat_1

ffa6398

github-actions bot added the Stale label Sep 3, 2023

github-actions bot closed this Sep 4, 2023

	struct ifaddrs * ifAddrStruct = NULL;
	struct ifaddrs * ifAddrStruct = nullptr;


		getifaddrs(&ifAddrStruct);

		std::vector<std::string> name_strs =

	struct ifaddrs * if_Addr_Struct = NULL;
	struct ifaddrs * if_Addr_Struct = nullptr;

[Feature](network interface) Support network interface #16617

[Feature](network interface) Support network interface #16617

Conversation

ZashJie commented Feb 10, 2023 • edited Loading

Proposed changes

Problem summary

Checklist(Required)

Further comments

github-actions bot commented Feb 10, 2023

hf200012 commented Feb 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yukang-Lian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 11, 2023

morningman commented Feb 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Mar 1, 2023

Choose a reason for hiding this comment

github-actions bot Mar 1, 2023

Choose a reason for hiding this comment

github-actions bot Mar 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Mar 3, 2023

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Mar 5, 2023

Choose a reason for hiding this comment

github-actions bot commented Mar 5, 2023

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 6, 2023

github-actions bot commented Mar 6, 2023

github-actions bot commented Mar 6, 2023

github-actions bot commented Sep 3, 2023

ZashJie commented Feb 10, 2023 •

edited

Loading

hf200012 commented Feb 10, 2023 •

edited

Loading