Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error when adding more nodes #93

Closed
TitanFighter opened this issue Aug 2, 2022 · 1 comment
Closed

Error when adding more nodes #93

TitanFighter opened this issue Aug 2, 2022 · 1 comment

Comments

@TitanFighter
Copy link

TitanFighter commented Aug 2, 2022 •
edited
Loading

Hi.

Initially cluster was created using v0.5.6.

Now Im trying to add 4 new worker nodes (from 3 to 7) using v0.5.7.

What did I do:

  1. I ran touch /etc/ready && echo "true" > /etc/ready according to this issue.
  2. changed instance_count from 3 to 7 in this block
worker_node_pools:
- name: cpx31
  instance_type: cpx31
  instance_count: 7
  1. Ran docker run --rm -it -v ${PWD}:/cluster -v ${HOME}/.ssh:/tmp/.ssh vitobotta/hetzner-k3s:v0.5.7 create-cluster --config-file /cluster/hetzner-k3s_config.yaml

and I got the next:

Unable to find image 'vitobotta/hetzner-k3s:v0.5.7' locally
v0.5.7: Pulling from vitobotta/hetzner-k3s
2408cc74d12b: Pull complete 
fdbd9a0f0a99: Pull complete 
4af624e0dc7d: Pull complete 
c013a669105b: Pull complete 
6f2847677b49: Pull complete 
19e3c5c17578: Pull complete 
909d0b061efd: Pull complete 
28eb14bf1870: Pull complete 
602358fb47f0: Pull complete 
ce4629e9f94c: Pull complete 
Digest: sha256:f533f7e9c6511fb357209fd18293cf4ab1dfe7276b1d66d8a99eab65359e6842
Status: Downloaded newer image for vitobotta/hetzner-k3s:v0.5.7
Enter passphrase for /root/.ssh/id_rsa: 

Placement group us-east already exists, skipping.


Firewall already exists, skipping.


Private network already exists, skipping.


SSH key already exists, skipping.


Placement group us-east-cpx31 already exists, skipping.

Server us-east-cpx21-master3 already exists, skipping.

Server us-east-cpx21-master2 already exists, skipping.

Creating server us-east-cpx31-pool-cpx31-worker4...
Creating server us-east-cpx31-pool-cpx31-worker6...
Server us-east-cpx31-pool-cpx31-worker3 already exists, skipping.

Server us-east-cpx21-master1 already exists, skipping.

Creating server us-east-cpx31-pool-cpx31-worker7...
Server us-east-cpx31-pool-cpx31-worker1 already exists, skipping.

Server us-east-cpx31-pool-cpx31-worker2 already exists, skipping.

Creating server us-east-cpx31-pool-cpx31-worker5...
...server us-east-cpx31-pool-cpx31-worker6 created.

...server us-east-cpx31-pool-cpx31-worker7 created.

Error creating server us-east-cpx31-pool-cpx31-worker4. Response details below:

#<HTTP::Response/1.1 403 Forbidden {"Date"=>"Tue, 02 Aug 2022 14:01:59 GMT", "Content-Type"=>"application/json", "Content-Length"=>"210", "Connection"=>"close", "ratelimit-limit"=>"3600", "ratelimit-remaining"=>"3591", "ratelimit-reset"=>"1659448924", "x-correlation-id"=>"ed813851-38a7-4723-b4d8-095864611a12", "Strict-Transport-Security"=>"max-age=15724800; includeSubDomains", "Access-Control-Allow-Origin"=>"*", "Access-Control-Allow-Credentials"=>"true"}>
Error creating server us-east-cpx31-pool-cpx31-worker5. Response details below:

#<HTTP::Response/1.1 403 Forbidden {"Date"=>"Tue, 02 Aug 2022 14:02:00 GMT", "Content-Type"=>"application/json", "Content-Length"=>"210", "Connection"=>"close", "ratelimit-limit"=>"3600", "ratelimit-remaining"=>"3591", "ratelimit-reset"=>"1659448925", "x-correlation-id"=>"78c46884-cddd-4897-b65b-2abcecd4b0ea", "Strict-Transport-Security"=>"max-age=15724800; includeSubDomains", "Access-Control-Allow-Origin"=>"*", "Access-Control-Allow-Credentials"=>"true"}>
Waiting for server us-east-cpx21-master2 to be up...
Waiting for server us-east-cpx21-master3 to be up...
Waiting for server us-east-cpx21-master1 to be up...
#<Thread:0x00007f547291e3d0 /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/k3s/cluster.rb:583 run> terminated with exception (report_on_exception is true):
#<Thread:0x00007f547291dcc8 /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/k3s/cluster.rb:583 run> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:56:in `block in wait_for_ssh': undefined method `[]' for nil:NilClass (NoMethodError)

      server_name = server['name']
                          ^^^^^^^^
        from /usr/local/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `block in catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:123:in `timeout'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:55:in `wait_for_ssh'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/k3s/cluster.rb:583:in `block (2 levels) in wait_for_servers'
/usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:56:in `block in wait_for_ssh': undefined method `[]' for nil:NilClass (NoMethodError)

      server_name = server['name']
                          ^^^^^^^^
        from /usr/local/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `block in catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:123:in `timeout'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:55:in `wait_for_ssh'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/k3s/cluster.rb:583:in `block (2 levels) in wait_for_servers'
Waiting for server us-east-cpx31-pool-cpx31-worker3 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker1 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker2 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
...server us-east-cpx21-master2 is now up.
...server us-east-cpx21-master3 is now up.
...server us-east-cpx21-master1 is now up.
...server us-east-cpx31-pool-cpx31-worker1 is now up.
...server us-east-cpx31-pool-cpx31-worker2 is now up.
...server us-east-cpx31-pool-cpx31-worker3 is now up.
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
...server us-east-cpx31-pool-cpx31-worker6 is now up.
...server us-east-cpx31-pool-cpx31-worker7 is now up.
/usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:56:in `block in wait_for_ssh': undefined method `[]' for nil:NilClass (NoMethodError)

      server_name = server['name']
                          ^^^^^^^^
        from /usr/local/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `block in catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:123:in `timeout'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:55:in `wait_for_ssh'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/k3s/cluster.rb:583:in `block (2 levels) in wait_for_servers'

The 2nd attempt gave this:

Placement group us-east already exists, skipping.


Firewall already exists, skipping.


Private network already exists, skipping.


SSH key already exists, skipping.


Placement group us-east-cpx31 already exists, skipping.

Creating server us-east-cpx31-pool-cpx31-worker4...
Server us-east-cpx21-master3 already exists, skipping.

Server us-east-cpx31-pool-cpx31-worker7 already exists, skipping.

Server us-east-cpx21-master2 already exists, skipping.

Creating server us-east-cpx31-pool-cpx31-worker5...
Server us-east-cpx31-pool-cpx31-worker1 already exists, skipping.

Server us-east-cpx31-pool-cpx31-worker6 already exists, skipping.

Server us-east-cpx21-master1 already exists, skipping.

Server us-east-cpx31-pool-cpx31-worker3 already exists, skipping.

Error creating server us-east-cpx31-pool-cpx31-worker4. Response details below:

#<HTTP::Response/1.1 403 Forbidden {"Date"=>"Tue, 02 Aug 2022 14:04:47 GMT", "Content-Type"=>"application/json", "Content-Length"=>"202", "Connection"=>"close", "ratelimit-limit"=>"3600", "ratelimit-remaining"=>"3586", "ratelimit-reset"=>"1659449101", "x-correlation-id"=>"07547d2b-0b4e-4f98-8b23-fd1e3e053ea1", "Strict-Transport-Security"=>"max-age=15724800; includeSubDomains", "Access-Control-Allow-Origin"=>"*", "Access-Control-Allow-Credentials"=>"true"}>
Error creating server us-east-cpx31-pool-cpx31-worker5. Response details below:

#<HTTP::Response/1.1 403 Forbidden {"Date"=>"Tue, 02 Aug 2022 14:04:47 GMT", "Content-Type"=>"application/json", "Content-Length"=>"202", "Connection"=>"close", "ratelimit-limit"=>"3600", "ratelimit-remaining"=>"3585", "ratelimit-reset"=>"1659449102", "x-correlation-id"=>"047dc2d9-f136-4a29-a2c7-8289b80247b4", "Strict-Transport-Security"=>"max-age=15724800; includeSubDomains", "Access-Control-Allow-Origin"=>"*", "Access-Control-Allow-Credentials"=>"true"}>
Server us-east-cpx31-pool-cpx31-worker2 already exists, skipping.

Waiting for server us-east-cpx21-master3 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker7 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker1 to be up...
#<Thread:0x00007f107cd9c3b0 /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/k3s/cluster.rb:583 run> terminated with exception (report_on_exception is true):
#<Thread:0x00007f107cd9c568 /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/k3s/cluster.rb:583 run> terminated with exception (report_on_exception is true):
Waiting for server us-east-cpx21-master2 to be up...
/usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:56:in `block in wait_for_ssh': undefined method `[]' for nil:NilClass (NoMethodError)

      server_name = server['name']
                          ^^^^^^^^
        from /usr/local/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `block in catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:123:in `timeout'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:55:in `wait_for_ssh'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/k3s/cluster.rb:583:in `block (2 levels) in wait_for_servers'
/usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:56:in `block in wait_for_ssh': undefined method `[]' for nil:NilClass (NoMethodError)

      server_name = server['name']
                          ^^^^^^^^
        from /usr/local/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `block in catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:123:in `timeout'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:55:in `wait_for_ssh'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/k3s/cluster.rb:583:in `block (2 levels) in wait_for_servers'
Waiting for server us-east-cpx21-master1 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker3 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker6 to be up...
Waiting for server us-east-cpx31-pool-cpx31-worker2 to be up...
...server us-east-cpx31-pool-cpx31-worker7 is now up.
...server us-east-cpx21-master3 is now up.
...server us-east-cpx31-pool-cpx31-worker6 is now up.
...server us-east-cpx21-master2 is now up.
...server us-east-cpx21-master1 is now up.
...server us-east-cpx31-pool-cpx31-worker1 is now up.
...server us-east-cpx31-pool-cpx31-worker3 is now up.
/usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:56:in `block in wait_for_ssh': undefined method `[]' for nil:NilClass (NoMethodError)

      server_name = server['name']
                          ^^^^^^^^
        from /usr/local/lib/ruby/3.1.0/timeout.rb:107:in `block in timeout'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `block in catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:36:in `catch'
        from /usr/local/lib/ruby/3.1.0/timeout.rb:123:in `timeout'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/utils.rb:55:in `wait_for_ssh'
        from /usr/local/bundle/gems/hetzner-k3s-0.5.7/lib/hetzner/k3s/cluster.rb:583:in `block (2 levels) in wait_for_servers'

In Hetzner's console I see that worker4 and worker5 have not been created (6 and 7 have been created).

upgrade-cluster gives "No value provided for required options '--new-k3s-version'"

Config:

# https://github.com/vitobotta/hetzner-k3s
---
hetzner_token: xyz
cluster_name: us-east
kubeconfig_path: "/cluster/kubeconfig"
k3s_version: v1.23.3+k3s1
public_ssh_key_path: "~/.ssh/id_rsa.pub"
private_ssh_key_path: "~/.ssh/id_rsa"
ssh_allowed_networks:
  - 0.0.0.0/0
verify_host_key: false
location: ash
schedule_workloads_on_masters: false
masters:
  instance_type: cpx21
  instance_count: 3
worker_node_pools:
- name: cpx31
  instance_type: cpx31
  instance_count: 7
enable_encryption: true
kube_scheduler_args:
- bind-address=0.0.0.0
kube_controller_manager_args:
- bind-address=0.0.0.0
kubelet_args:
- authentication-token-webhook=true
- authorization-mode=Webhook

Any ideas how to add more nodes?

Thanks.
@TitanFighter
Copy link
Author

TitanFighter commented Aug 2, 2022 •
edited
Loading

It looks like this is due to rate limit

#<HTTP::Response/1.1 403 Forbidden {"Date"=>"Tue, 02 Aug 2022 15:00:32 GMT", "Content-Type"=>"application/json", "Content-Length"=>"202", "Connection"=>"close", "ratelimit-limit"=>"3600", "ratelimit-remaining"=>"3591", "ratelimit-reset"=>"1659452441", "x-correlation-id"=>"baa1560d-868c-46ca-9923-7a59c7a3c892", "Strict-Transport-Security"=>"max-age=15724800; includeSubDomains", "Access-Control-Allow-Origin"=>"*", "Access-Control-Allow-Credentials"=>"true"}>

Limit of servers was reached - in Hetzner Console the button "Add server" was deactivated.
Solution? - Click "Person" icon in the right top corner -> Limits -> Request limit increase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@TitanFighter