Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SSH reliability settings #317

Merged
merged 15 commits into from
Aug 19, 2024
Merged

SSH reliability settings #317

merged 15 commits into from
Aug 19, 2024

Conversation

ab77
Copy link
Contributor

@ab77 ab77 commented Aug 5, 2024 •
edited
Loading

tl;dr docker-compose uses so many SSH connections (using DOCKER_HOST=ssh://...) that SSHd starts throttling it and breaking CI runs

Thank you for the hint:

   MaxStartups
           Specifies the maximum number of concurrent
           unauthenticated connections to the SSH daemon.
           Additional connections will be dropped until
           authentication succeeds or the LoginGraceTime expires for
           a connection.  The default is 10:30:100.

           Alternatively, random early drop can be enabled by
           specifying the three colon separated values
           start:rate:full (e.g. "10:30:60").  sshd(8) will refuse
           connection attempts with a probability of rate/100 (30%)
           if there are currently start (10) unauthenticated
           connections.  The probability increases linearly and all
           connection attempts are refused if the number of
           unauthenticated connections reaches full (60).

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 5, 2024 •
edited
Loading

Website deployed to CF Pages, 👀 preview link https://f03c5f25.open-balena.pages.dev

@flowzone-app flowzone-app bot enabled auto-merge August 5, 2024 16:06
@ab77 ab77 had a problem deploying to compose-private-pki August 5, 2024 16:06 — with GitHub Actions Failure
@ab77 ab77 temporarily deployed to balena-public-pki August 5, 2024 16:06 — with GitHub Actions Inactive
@ab77 ab77 had a problem deploying to compose-private-pki August 5, 2024 16:28 — with GitHub Actions Error
@ab77 ab77 temporarily deployed to balena-public-pki August 5, 2024 16:28 — with GitHub Actions Inactive
@ab77 ab77 temporarily deployed to compose-private-pki August 5, 2024 16:38 — with GitHub Actions Inactive
@ab77 ab77 temporarily deployed to balena-public-pki August 5, 2024 16:38 — with GitHub Actions Inactive
@ab77 ab77 requested a review from a team August 5, 2024 16:57
@ab77 ab77 marked this pull request as draft August 5, 2024 17:25
auto-merge was automatically disabled August 5, 2024 17:25

Pull request was converted to draft

@ab77 ab77 removed the request for review from a team August 5, 2024 17:25
@ab77 ab77 had a problem deploying to compose-private-pki August 5, 2024 17:28 — with GitHub Actions Failure
@ab77 ab77 temporarily deployed to balena-public-pki August 5, 2024 17:28 — with GitHub Actions Inactive
@ab77 ab77 temporarily deployed to compose-private-pki August 5, 2024 20:46 — with GitHub Actions Inactive
@ab77 ab77 temporarily deployed to balena-public-pki August 5, 2024 20:46 — with GitHub Actions Inactive
@ab77 ab77 temporarily deployed to compose-private-pki August 6, 2024 15:18 — with GitHub Actions Inactive
@ab77 ab77 temporarily deployed to balena-public-pki August 6, 2024 15:18 — with GitHub Actions Inactive
@ab77 ab77 had a problem deploying to compose-private-pki August 7, 2024 03:23 — with GitHub Actions Error
@ab77 ab77 had a problem deploying to balena-public-pki August 7, 2024 03:23 — with GitHub Actions Error
@ab77 ab77 had a problem deploying to compose-private-pki August 7, 2024 03:30 — with GitHub Actions Error
@ab77 ab77 had a problem deploying to balena-public-pki August 7, 2024 03:30 — with GitHub Actions Error
@ab77 ab77 had a problem deploying to compose-private-pki August 7, 2024 03:41 — with GitHub Actions Failure
@ab77 ab77 had a problem deploying to balena-public-pki August 7, 2024 03:41 — with GitHub Actions Failure
@ab77 ab77 had a problem deploying to compose-private-pki August 9, 2024 21:35 — with GitHub Actions Error
@ab77 ab77 had a problem deploying to balena-public-pki August 9, 2024 21:35 — with GitHub Actions Error
@ab77 ab77 had a problem deploying to compose-private-pki August 12, 2024 15:06 — with GitHub Actions Error
@ab77 ab77 temporarily deployed to balena-public-pki August 19, 2024 16:34 — with GitHub Actions Inactive
@ab77 ab77 requested a review from a team August 19, 2024 17:07
@ab77 ab77 marked this pull request as ready for review August 19, 2024 17:07
ab77 added 14 commits August 19, 2024 10:07
ab77
ab77 commented Aug 19, 2024
View reviewed changes
.github/workflows/tests.yml
@@ -839,6 +846,10 @@ jobs:
aws ec2 wait instance-running --instance-ids "${instance_id}"
with_backoff aws ec2 wait instance-status-ok --instance-ids "${instance_id}"

private_ip="$(aws ec2 describe-instances --instance-id "${instance_id}" \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we may want to use this in the future

ab77
ab77 commented Aug 19, 2024
View reviewed changes
.github/workflows/tests.yml
# https://forums.docker.com/t/docker-compose-through-ssh-failing-and-referring-to-docker-example-com/115165/18
- path: /etc/ssh/sshd_config.d/00-cloud-init
content: |
MaxStartups 100:0:100
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the actual fix

@flowzone-app flowzone-app bot enabled auto-merge August 19, 2024 17:11
@ab77 ab77 temporarily deployed to compose-private-pki August 19, 2024 17:11 — with GitHub Actions Inactive
@ab77 ab77 temporarily deployed to balena-public-pki August 19, 2024 17:11 — with GitHub Actions Inactive
@ab77
SSH reliability settings
539adad 
* enable SSHD multiplexing, KeepAlive and relax throttling

Thank you for the hint:
https://forums.docker.com/t/docker-compose-through-ssh-failing-and-referring-to-docker-example-com/115165/18?u=ab77

       MaxStartups
               Specifies the maximum number of concurrent
               unauthenticated connections to the SSH daemon.
               Additional connections will be dropped until
               authentication succeeds or the LoginGraceTime expires for
               a connection.  The default is 10:30:100.

               Alternatively, random early drop can be enabled by
               specifying the three colon separated values
               start:rate:full (e.g. "10:30:60").  sshd(8) will refuse
               connection attempts with a probability of rate/100 (30%)
               if there are currently start (10) unauthenticated
               connections.  The probability increases linearly and all
               connection attempts are refused if the number of
               unauthenticated connections reaches full (60).

change-type: patch
@ab77 ab77 temporarily deployed to compose-private-pki August 19, 2024 17:30 — with GitHub Actions Inactive
@ab77 ab77 temporarily deployed to balena-public-pki August 19, 2024 18:04 — with GitHub Actions Inactive
@flowzone-app flowzone-app bot merged commit 366249a into master Aug 19, 2024
50 checks passed
@flowzone-app flowzone-app bot deleted the ab77/patch branch August 19, 2024 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@klutchell klutchell klutchell approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ab77 @klutchell