Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: resolve issue with integration tests where lack of disk space caused k3s issues #2804

Merged

Conversation

aaron-prindle
Copy link
Collaborator

@aaron-prindle aaron-prindle commented Oct 17, 2023

fixes #2803

PR here adds a step to our integration test Github Action workflow to free up additional space to resolve issue where tests were failing to do not having enough disk space. See below:

Ongoing Kaniko CI/CD integration tests failures are related to disk pressure which is making it so the registry pod is not coming up properly (being evicted). Snippet of k3s status during failing run:

log link:
https://github.com/GoogleContainerTools/kaniko/actions/runs/6554012241/job/17800290036?pr=2804#step:6:127

log error snippet

...
       local-path-provisioner-957fdf8bc-h2b8x                1/1     Running     0          22m
            coredns-77ccd57875-9rd7w                              1/1     Running     0          22m
            helm-install-traefik-crd-fgmlk                        0/1     Completed   0          22m
            helm-install-local-registry-8zsth                     0/1     Completed   0          22m
            helm-install-traefik-wlmmz                            0/1     Completed   1          22m
            traefik-64f55bb67d-wv4b5                              1/1     Running     0          21m
            metrics-server-648b5df564-srtt4                       1/1     Running     0          22m
            local-registry-docker-registry-56d58d5d6-6dncc        0/1     Error       0          21m
            local-registry-docker-registry-56d58d5d6-sh4ls        0/1     Evicted     0          18s
            local-registry-docker-registry-56d58d5d6-tphm2        0/1     Pending     0          18s
            svclb-local-registry-docker-registry-c7ea5916-zggj6   0/1     Evicted     0          6s
            svclb-traefik-050d8b73-crcsf                          0/2     Evicted     0          0s

@aaron-prindle aaron-prindle marked this pull request as ready for review October 17, 2023 21:45
@aaron-prindle aaron-prindle force-pushed the fix-int-tests branch 2 times, most recently from 48608d8 to 154ebd7 Compare October 18, 2023 00:00
@aaron-prindle aaron-prindle changed the title debug: DO NOT MERGE - attempting to fix integration tests fix: resolve issue with integration tests where lack of disk space caused k3s issues Oct 18, 2023
Copy link
Collaborator

@JeromeJu JeromeJu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aaron-prindle aaron-prindle merged commit 458789f into GoogleContainerTools:main Oct 18, 2023
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Kaniko integration tests failing - localhost:5000 inaccessible -> k3s is evicting registry pod
2 participants