support hertzbeat metrics collector cluster (#1101)

Co-authored-by: Ceilzcx <1758619238@qq.com>
apache · Aug 2, 2023 · 253dd21 · 253dd21
1 parent ea03977
commit 253dd21
Show file tree

Hide file tree

Showing 74 changed files with 9,362 additions and 4,743 deletions.
diff --git a/README.md b/README.md
@@ -30,7 +30,8 @@ An open-source, real-time monitoring system with custom monitoring and agentless
 
 ## 🎡 <font color="green">Introduction</font>
 
-[HertzBeat](https://github.com/dromara/hertzbeat) is an open source, real-time monitoring system with custom monitoring and agentless capabilities. It combines **monitoring, alarm, and notification** features into one platform, and supports monitoring for web service, database, os, middleware, cloud-native, network and more.
+[HertzBeat](https://github.com/dromara/hertzbeat) is an open source, real-time monitoring system with custom monitoring and agentless capabilities.    
+It combines **monitoring, alarm, and notification** features into one platform, and supports monitoring for web service, database, os, middleware, cloud-native, network and more.
 
 ### Features
 
@@ -97,17 +98,17 @@ HertzBeat aims to help developers and teams quickly build their own monitoring s
 
 ### 🍞 Install HertzBeat   
 
-> HertzBeat supports installation through source code, docker or package, cpu support X86/ARM64.   
+> HertzBeat supports installation through source code, docker or package, cpu support x86/arm64.   
 
 ##### 1：Install quickly via docker   
 
 1. Just one command to get started:
 
-```docker run -d -p 1157:1157 --name hertzbeat tancloud/hertzbeat```
+```docker run -d -p 1157:1157 -p 1158:1158 --name hertzbeat tancloud/hertzbeat```
 
 ```or use quay.io (if dockerhub network connect timeout)```
 
-```docker run -d -p 1157:1157 --name hertzbeat quay.io/tancloud/hertzbeat```
+```docker run -d -p 1157:1157 -p 1158:1158 --name hertzbeat quay.io/tancloud/hertzbeat```
 
 2. Access `localhost:1157` to start, default account: `admin/hertzbeat`  
 

diff --git a/README_CN.md b/README_CN.md
@@ -94,17 +94,17 @@
 - 如果您是想将HertzBeat部署到内网环境搭建监控系统，请参考下面的部署文档进行操作。  
 
 ### 🍞 HertzBeat安装
-> HertzBeat支持通过源码安装启动，Docker容器运行和安装包方式安装部署，CPU架构支持X86/ARM64。
+> HertzBeat支持通过源码安装启动，Docker容器运行和安装包方式安装部署，CPU架构支持x86/arm64。
 
 ##### 方式一：Docker方式快速安装  
 
 1. `docker` 环境仅需一条命令即可开始     
 
-```docker run -d -p 1157:1157 --name hertzbeat tancloud/hertzbeat```
+```docker run -d -p 1157:1157 -p 1158:1158 --name hertzbeat tancloud/hertzbeat```
 
 ```或者使用 quay.io (若 dockerhub 网络链接超时)```
 
-```docker run -d -p 1157:1157 --name hertzbeat quay.io/tancloud/hertzbeat```
+```docker run -d -p 1157:1157 -p 1158:1158 --name hertzbeat quay.io/tancloud/hertzbeat```
 
 2. 浏览器访问 `localhost:1157` 即可开始，默认账号密码 `admin/hertzbeat`
 

diff --git a/collector/README.md b/collector/README.md
@@ -5,32 +5,52 @@ Collect Metrics Data from peer-to-peer machine by common protocols.
 
 support protocols：http/https, jdbc, ssh, telnet, jmx, ping, prometheus...
 
-#### Monitor Type   
-
-* OS
-  * Linux
-  * Windows
-  * Ubuntu
-  * CentOs
-* Database
-  * Mysql
-  * Oracle
-  * PostgreSQL
-* Middleware   
-  * Kafka
-  * Zookeeper
-  * RocketMq
-  * Etcd  
-* CloudNative  
-  * Docker  
-  * Kubernetes  
-  * Istio  
-* Service  
-  * Tomcat
-  * Jetty
-  * Http
-  * Ping
-  * Port
-
-....
+## Supported
 
+> We define all monitoring collection types such as `mysql`, `jvm`, and `k8s` as `YML` monitoring templates, allowing users to import them to support corresponding types of monitoring.
+> Welcome everyone to contribute your customized general monitoring type YML template during use.
+
+
+- [Website](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-website.yml), [Port Telnet](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-port.yml),
+  [Http Api](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-api.yml), [Ping Connect](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-ping.yml),
+  [Jvm](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-jvm.yml), [SiteMap](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-fullsite.yml),
+  [Ssl Certificate](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-ssl_cert.yml), [SpringBoot2](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-springboot2.yml),
+  [FTP Server](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-ftp.yml), [SpringBoot3](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-springboot3.yml)
+- [Mysql](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-mysql.yml), [PostgreSQL](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-postgresql.yml),
+  [MariaDB](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-mariadb.yml), [Redis](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-redis.yml),
+  [ElasticSearch](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-elasticsearch.yml), [SqlServer](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-sqlserver.yml),
+  [Oracle](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-oracle.yml), [MongoDB](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-mongodb.yml),
+  [DM](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-dm.yml), [OpenGauss](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-opengauss.yml),
+  [ClickHouse](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-clickhouse.yml), [IoTDB](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-iotdb.yml),
+  [Redis Cluster](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-redis_cluster.yml), [Redis Sentinel](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-redis_sentinel.yml)
+- [Linux](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-linux.yml), [Ubuntu](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-ubuntu.yml),
+  [CentOS](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-centos.yml), [Windows](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-windows.yml),
+  [EulerOS](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-euleros.yml), [Fedora CoreOS](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-coreos.yml),
+  [OpenSUSE](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-opensuse.yml), [Rocky Linux](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-rockylinux.yml),
+  [Red Hat](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-redhat.yml), [FreeBSD](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-freebsd.yml),
+  [AlmaLinux](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-almalinux.yml), [Debian Linux](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-debian.yml)
+- [Tomcat](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-tomcat.yml), [Nacos](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-nacos.yml),
+  [Zookeeper](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-zookeeper.yml), [RabbitMQ](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-rabbitmq.yml),
+  [Flink](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-flink.yml), [Kafka](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-kafka.yml),
+  [ShenYu](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-shenyu.yml), [DynamicTp](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-dynamic_tp.yml),
+  [Jetty](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-jetty.yml), [ActiveMQ](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-activemq.yml)
+- [Kubernetes](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-kubernetes.yml), [Docker](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-docker.yml)
+- [CiscoSwitch](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-cisco_switch.yml), [HpeSwitch](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-hpe_switch.yml),
+  [HuaweiSwitch](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-huawei_switch.yml), [TpLinkSwitch](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-tplink_switch.yml),
+  [H3cSwitch](https://raw.githubusercontent.com/dromara/hertzbeat/master/manager/src/main/resources/define/app-h3c_switch.yml)
+- And More Your Custom Template.
+- Notified Support `Discord` `Slack` `Telegram` `Email` `DingDing` `WeChat` `FeiShu` `Webhook` `SMS`.
+
+## Build HertzBeat Collector Install Package  
+
+1. Execute command in root
+
+```mvn clean install```
+
+2. Execute command in collector module
+
+```shell
+cd collector
+
+mvn clean package -Pcluster
+```
diff --git a/collector/pom.xml b/collector/pom.xml
@@ -211,6 +211,7 @@
                             <include>*.yml</include>
                             <include>*.properties</include>
                             <include>banner.txt</include>
+                            <include>META-INF/**</include>
                         </includes>
                     </resource>
                 </resources>

diff --git a/collector/src/main/java/org/dromara/hertzbeat/collector/dispatch/DispatchProperties.java b/collector/src/main/java/org/dromara/hertzbeat/collector/dispatch/DispatchProperties.java
@@ -67,11 +67,24 @@ public void setExport(ExportProperties export) {
      * 入口可以时etcd信息,http请求,消息中间件消息请求
      */
     public static class EntranceProperties {
+
+        /**
+         * netty server client config
+         */
+        private NettyProperties netty;
         /**
          * etcd配置信息
          */
         private EtcdProperties etcd;
-
+
+        public NettyProperties getNetty() {
+            return netty;
+        }
+
+        public void setNetty(NettyProperties netty) {
+            this.netty = netty;
+        }
+
         public EtcdProperties getEtcd() {
             return etcd;
         }
@@ -86,7 +99,7 @@ public static class EtcdProperties {
              * Whether etcd scheduling is started
              * etcd调度是否启动
              */
-            private boolean enabled = true;
+            private boolean enabled = false;
 
             /**
              * etcd's connection endpoint url
@@ -194,6 +207,62 @@ public void setJobDir(String jobDir) {
                 this.jobDir = jobDir;
             }
         }
+
+        public static class NettyProperties {
+
+            /**
+             * whether netty scheduling is started
+             */
+            private boolean enabled = false;
+
+            /**
+             * this collector unique identity
+             * default is the host name
+             */
+            private String identity;
+
+            /**
+             * connect cluster master ip
+             */
+            private String managerIp;
+
+            /**
+             * connect cluster master port
+             */
+            private int managerPort = 1158;
+
+            public boolean isEnabled() {
+                return enabled;
+            }
+
+            public void setEnabled(boolean enabled) {
+                this.enabled = enabled;
+            }
+
+            public String getIdentity() {
+                return identity;
+            }
+
+            public void setIdentity(String identity) {
+                this.identity = identity;
+            }
+
+            public String getManagerIp() {
+                return managerIp;
+            }
+
+            public void setManagerIp(String managerIp) {
+                this.managerIp = managerIp;
+            }
+
+            public int getManagerPort() {
+                return managerPort;
+            }
+
+            public void setManagerPort(int managerPort) {
+                this.managerPort = managerPort;
+            }
+        }
     }
 
     /**

diff --git a/collector/src/main/java/org/dromara/hertzbeat/collector/dispatch/MetricsCollect.java b/collector/src/main/java/org/dromara/hertzbeat/collector/dispatch/MetricsCollect.java
@@ -139,7 +139,6 @@ public void run() {
             response.setCode(CollectRep.Code.FAIL);
             response.setMsg("not support " + app + ", "
                     + metrics.getName() + ", " + metrics.getProtocol());
-            return;
         } else {
             try {
                 abstractCollect.collect(response, monitorId, app, metrics);