-
Notifications
You must be signed in to change notification settings - Fork 9.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ETCD unusual long request when proxying from the grpc gateway #14065
Comments
Actually here are some additional info. |
Please collect metrics (e.g. |
Gday @ptabor As Requested, this is collected metrics, I still cant see what is the issue from the metrics (sigh....) I really appreciate any helps Update: I have tried to downgrade etcd to 3.4.16 and it still happens, and the metrics file is based on 3.4 |
What happened?
during setting up 3 node etcd cluster, it is found that if there are more than 5 apisix nodes exists, the etcd may takes longer than 20 seconds to process the request.
![image](https://user-images.githubusercontent.com/66858976/169774048-8031920a-4636-4fb4-bfa3-af37eb29eb8c.png)
As picture indicated below
What did you expect to happen?
Usually with in this amount of data and connections, it usually wont be wait more than 1 second to process.
How can we reproduce it (as minimally and precisely as possible)?
setting up 3 node etcd cluster, and 6 apisix nodes
try to modify a route, and it produces.
Anything else we need to know?
No response
Etcd version (please run commands below)
Etcd configuration (command line flags or environment variables)
paste your configuration here
flags:
--cert-file=/root/etcd-cert/server.crt
--key-file=/root/etcd-cert/server.key
--peer-auto-tls --metrics 'extensive'
--enable-pprof
--log-level 'debug'
ENV_PARAMS:
ETCD_INITIAL_CLUSTER="node1=https://node1:2380,node2=https://node2:2380,node3=https://node3:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_NAME="node1"
ETCD_DATA_DIR="/data/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://node1:2380"
ETCD_LISTEN_CLIENT_URLS="https://node1:2379,https://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://node1:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://node1:2379"
Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)
Relevant log output
No response
The text was updated successfully, but these errors were encountered: