Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gnmic 0.35.1 - failed to create a gRPC client for target XXX: context deadline exceeded #367

Closed
tiago-amado opened this issue Jan 31, 2024 · 10 comments
Closed

gnmic 0.35.1 - failed to create a gRPC client for target XXX: context deadline exceeded #367

tiago-amado opened this issue Jan 31, 2024 · 10 comments

Comments

@tiago-amado
Copy link

tiago-amado commented Jan 31, 2024 •
edited
Loading

Description

  1. gnmictls setup with 0.34.3 working fine for SROS
  2. After upgrade to 0.35.1 its returning an error:
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4:57400 -u admin -p admin --tls-ca ca.pem capabilities
target "172.20.20.4:57400", capabilities request failed: failed to create a gRPC client for target "172.20.20.4:57400" : 172.20.20.4:57400: context deadline exceeded
Error: one or more requests failed
  1. Downgrade to the previous release solves the error.

Outputs

    • Ok
[root@AF6 CLAB_CA]# gnmic version
version : 0.34.3
commit : d461fa9
   date : 2023-11-14T02:13:11Z
gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net/
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4 -u admin -p admin --tls-ca ca.pem capabilities
gNMI version: 0.8.0
supported models:
  - nokia-conf, Nokia, 23.10.R2
  - nokia-state, Nokia, 23.10.R2
  - nokia-li-state, Nokia, 23.10.R2
supported encodings:
  - JSON
  - BYTES
  - PROTO
  - JSON_IETF

[root@AF6 CLAB_CA]#
  1. Upgrade NOK
[root@AF6 ~]#
[root@AF6 ~]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)"
gnmic 0.35.1 is available. Changing from version 0.34.3.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.35.1/gnmic_0.35.1_linux_x86_64.tar.gz
Preparing to install gnmic 0.35.1 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.35.1
commit : 91f4205
   date : 2024-01-25T21:10:47Z
gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net/
[root@AF6 ~]# 
[root@AF6 ~]# 
[root@AF6 ~]# gnmic version
version : 0.35.1
commit : 91f4205
   date : 2024-01-25T21:10:47Z
gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net/
[root@AF6 ~]# 
[root@AF6 ~]# 
[root@AF6 ~]# cd /home/tiago/clab_projects/CLAB_CA/
[root@AF6 CLAB_CA]# ls -al
total 20
drwxr-xr-x  2 root root   66 Jan 31 12:31 .
drwxrwxrwx 15 root root 4096 Jan 31 12:14 ..
-rw-r--r--  1 root root 1675 Jan 31 12:15 ca.key
-rw-r--r--  1 root root 1367 Jan 31 12:15 ca.pem
-rw-r--r--  1 root root 1675 Jan 31 12:31 cert.key
-rw-r--r--  1 root root 1367 Jan 31 12:31 cert.pem
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4:57400 -u admin -p admin --tls-ca ca.pem capabilities
target "172.20.20.4:57400", capabilities request failed: failed to create a gRPC client for target "172.20.20.4:57400" : 172.20.20.4:57400: context deadline exceeded
Error: one or more requests failed
[root@AF6 CLAB_CA]#
  1. Downgrade Ok
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)" -- -v 0.34.3
gnmic 0.34.3 is available. Changing from version 0.35.1.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.34.3/gnmic_0.34.3_linux_x86_64.tar.gz
Preparing to install gnmic 0.34.3 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.34.3
commit : d461fa9
   date : 2023-11-14T02:13:11Z
gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net/
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# pwd
/home/tiago/clab_projects/CLAB_CA
[root@AF6 CLAB_CA]# ls -al
total 20
drwxr-xr-x  2 root root   66 Jan 31 12:31 .
drwxrwxrwx 15 root root 4096 Jan 31 12:14 ..
-rw-r--r--  1 root root 1675 Jan 31 12:15 ca.key
-rw-r--r--  1 root root 1367 Jan 31 12:15 ca.pem
-rw-r--r--  1 root root 1675 Jan 31 12:31 cert.key
-rw-r--r--  1 root root 1367 Jan 31 12:31 cert.pem
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4:57400 -u admin -p admin --tls-ca ca.pem capabilities
gNMI version: 0.8.0
supported models:
  - nokia-conf, Nokia, 23.10.R2
  - nokia-state, Nokia, 23.10.R2
  - nokia-li-state, Nokia, 23.10.R2
supported encodings:
  - JSON
  - BYTES
  - PROTO
  - JSON_IETF

[root@AF6 CLAB_CA]#
@karimra
Copy link
Collaborator

karimra commented Jan 31, 2024

Can you add the --debug flag and share the log output in the failed case ?

@tiago-amado
Copy link
Author

debug.txt

@karimra
Copy link
Collaborator

karimra commented Feb 1, 2024

This is due to a change in go-grpc v1.60 grpc/grpc-go#6776 that enforces this section of RFC7540.
go-grpc started filtering out forbidden cipher suites set in the TLS profile. The remaining ones don't overlap with the ones supported by SROS => auth handshake fails.

I will set the cipher suites manually so that go-grpc doesn't filter them out.
Thanks for reporting this.

@karimra karimra mentioned this issue Feb 5, 2024
Merged
@karimra
Copy link
Collaborator

karimra commented Feb 6, 2024

@tiago-amado v0.35.2 should advertise the same cipher suites as before go-grpc v1.60, please give it a try and let me know

@tiago-amado
Copy link
Author

tiago-amado commented Feb 8, 2024 •
edited
Loading

Hi @karimra , I've tested and its still failing.

OK with v0.34.3

[root@AF6 CLAB_CA]# gnmic version
version : 0.34.3
 commit : d461fa9
   date : 2023-11-14T02:13:11Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
[root@AF6 CLAB_CA]# 

[root@AF6 CLAB_CA]# gnmic -a clab-cert01-sr  -u admin -p admin --tls-ca ca.pem capabilities
gNMI version: 0.8.0
supported models:
  - nokia-conf, Nokia, 23.10.R2
  - nokia-state, Nokia, 23.10.R2
  - nokia-li-state, Nokia, 23.10.R2
supported encodings:
  - JSON
  - BYTES
  - PROTO
  - JSON_IETF

[root@AF6 CLAB_CA]#

NOK after upgrade to v0.35.2

[root@AF6 CLAB_CA]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)" -- -v 0.35.2
gnmic 0.35.2 is available. Changing from version 0.34.3.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.35.2/gnmic_0.35.2_linux_x86_64.tar.gz
Preparing to install gnmic 0.35.2 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.35.2
 commit : bd70a53
   date : 2024-02-06T00:46:16Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
[root@AF6 CLAB_CA]# 

[root@AF6 CLAB_CA]# gnmic -a clab-cert01-sr  -u admin -p admin --tls-ca ca.pem capabilities
target "clab-cert01-sr", capabilities request failed: failed to create a gRPC client for target "clab-cert01-sr" : clab-cert01-sr:57400: context deadline exceeded
Error: one or more requests failed
[root@AF6 CLAB_CA]#

OK after downgrade to v0.33.0

[root@AF6 ~]# cd /home/tiago/clab_projects/CLAB_CA/
[root@AF6 CLAB_CA]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)" -- -v 0.33.0
gnmic 0.33.0 is available. Changing from version 0.35.2.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.33.0/gnmic_0.33.0_linux_x86_64.tar.gz
Preparing to install gnmic 0.33.0 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.33.0
 commit : d8f931e
   date : 2023-10-08T17:08:01Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a clab-cert01-sr  -u admin -p admin --tls-ca ca.pem capabilities
gNMI version: 0.8.0
supported models:
  - nokia-conf, Nokia, 23.10.R2
  - nokia-state, Nokia, 23.10.R2
  - nokia-li-state, Nokia, 23.10.R2
supported encodings:
  - JSON
  - BYTES
  - PROTO
  - JSON_IETF

[root@AF6 CLAB_CA]#

@tiago-amado
Copy link
Author

gnmic_v0.35.2_debug.txt

@karimra
Copy link
Collaborator

karimra commented Feb 8, 2024

Can you share which cipher suites are configured on your SROS?

@tiago-amado
Copy link
Author

This one: tls-rsa-with3des-ede-cbc-sha
Attached the SROS config:
gnmic_v0.35.2_SR-config.txt

@tiago-amado
Copy link
Author

Hi @karimra.
I've tested with the latest 0.36.2 and its working now! many thanks!

OK with v0.36.2

after upgrade its working

[root@AF6 CLAB_CA]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)"  
gnmic 0.36.2 is available. Changing from version 0.35.2.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.36.2/gnmic_0.36.2_linux_x86_64.tar.gz
Preparing to install gnmic 0.36.2 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.36.2
 commit : a7844a6d
   date : 2024-03-05T20:10:26Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4:57400 -u admin -p admin --tls-ca ca.pem capabilities
gNMI version: 0.8.0
supported models:
  - nokia-conf, Nokia, 23.10.R2
  - nokia-state, Nokia, 23.10.R2
  - nokia-li-state, Nokia, 23.10.R2
supported encodings:
  - JSON
  - BYTES
  - PROTO
  - JSON_IETF

[root@AF6 CLAB_CA]#

NOK with v0.35.2

I've downgrade to v0.35.2 to retest and it is still failing. I've added more ciphers on SROS and did not solve.

[root@AF6 CLAB_CA]# bash -c "$(curl -sL https://get-gnmic.openconfig.net/)"  -- -v 0.35.2
gnmic 0.35.2 is available. Changing from version 0.36.2.
Downloading https://github.com/openconfig/gnmic/releases/download/v0.35.2/gnmic_0.35.2_linux_x86_64.tar.gz
Preparing to install gnmic 0.35.2 into /usr/local/bin
gnmic installed into /usr/local/bin/gnmic
version : 0.35.2
 commit : bd70a53
   date : 2024-02-06T00:46:16Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# 
[root@AF6 CLAB_CA]# gnmic -a 172.20.20.4:57400 -u admin -p admin --tls-ca ca.pem capabilities
target "172.20.20.4:57400", capabilities request failed: failed to create a gRPC client for target "172.20.20.4:57400" : 172.20.20.4:57400: context deadline exceeded
Error: one or more requests failed
[root@AF6 CLAB_CA]#

@karimra
Copy link
Collaborator

karimra commented Mar 9, 2024

Thanks, I think I messed up some imports in v0.35.2.

@karimra karimra closed this as completed Mar 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@karimra @tiago-amado