Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI (Buildkite): use a TMPDIR that is backed by real storage #42995

Merged
merged 1 commit into from
Nov 10, 2021
Merged

CI (Buildkite): use a TMPDIR that is backed by real storage #42995

merged 1 commit into from
Nov 10, 2021

Conversation

DilumAluthge
Copy link
Member

No description provided.

@DilumAluthge DilumAluthge added ci Continuous integration backport 1.6 Change should be backported to release-1.6 backport 1.7 labels Nov 8, 2021
@DilumAluthge DilumAluthge requested a review from a team as a code owner November 8, 2021 00:23
@DilumAluthge DilumAluthge removed the request for review from a team November 8, 2021 00:24
@DilumAluthge
Copy link
Member Author

DilumAluthge commented Nov 8, 2021 •
edited
Loading

@staticfloat I am seeing the following test failure on several Buildkite jobs.

file                               (13) |         failed at 2021-11-08T00:42:22.921
Test Failed at /cache/build/amdci4-5/julialang/julia-master/julia-e6d4b33a81/share/julia/test/file.jl:113
  Expression: dirname(t) == d
   Evaluated: "/cache/build/amdci4-5/julialang/julia-master/tmp" == "/cache/build/amdci4-5/julialang/julia-master/tmp/jl_PnXa94"

Keno
Keno reviewed Nov 8, 2021
View reviewed changes
.buildkite/pipelines/main/platforms/tester_linux.yml
@@ -61,6 +61,11 @@ steps:
unset JULIA_DEPOT_PATH
unset JULIA_PKG_SERVER

# Make sure that temp files and temp directories are created in a location that is
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this required?

Copy link
Member Author

@DilumAluthge DilumAluthge Nov 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inside the Buildkite sandbox, the /tmp directory lives on a tempfs that is backed by 1GB of RAM.

On certain Buildkite jobs (e.g. when we run the Pkg test suite with the Pkg server disabled), we end up cloning a lot of Git repos, and as a result we end up running out of space in the tempfs.

While /tmp lives in the tempfs, the build directory (the current working directory to which the Julia repo has been cloned) is actually backed by a real filesystem. So we want to tell all programs to create temp files and temp directories inside our current working directory.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Increase the size of the tempfs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could do that. I know that @staticfloat and @maleadt have expressed concerns about that in the past. We have multiple Buildkite agents on each machine, and we don't want to use up too much RAM.

This was referenced Nov 8, 2021
Merged
Merged
@DilumAluthge DilumAluthge mentioned this pull request Nov 9, 2021
Merged
@DilumAluthge DilumAluthge added the merge me PR is reviewed. Merge when all tests are passing label Nov 9, 2021
@DilumAluthge DilumAluthge merged commit f317d57 into master Nov 10, 2021
@DilumAluthge DilumAluthge deleted the dpa/bk-tmpdir branch November 10, 2021 03:45
@DilumAluthge DilumAluthge removed the merge me PR is reviewed. Merge when all tests are passing label Nov 10, 2021
KristofferC pushed a commit that referenced this pull request Nov 10, 2021
@DilumAluthge @KristofferC
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
9e11179 
(cherry picked from commit f317d57)
KristofferC pushed a commit that referenced this pull request Nov 10, 2021
@DilumAluthge @KristofferC
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
7000a62 
(cherry picked from commit f317d57)
KristofferC pushed a commit that referenced this pull request Nov 10, 2021
@DilumAluthge @KristofferC
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
0012a6a 
(cherry picked from commit f317d57)
KristofferC pushed a commit that referenced this pull request Nov 11, 2021
@DilumAluthge @KristofferC
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
c0dfb80 
(cherry picked from commit f317d57)
KristofferC pushed a commit that referenced this pull request Nov 12, 2021
@DilumAluthge @KristofferC
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
593922a 
(cherry picked from commit f317d57)
@KristofferC KristofferC removed backport 1.6 Change should be backported to release-1.6 backport 1.7 labels Nov 13, 2021
daviehh pushed a commit to daviehh/julia that referenced this pull request Nov 16, 2021
@DilumAluthge @daviehh
CI (Buildkite): use a TMPDIR that is backed by real storage (JuliaL…
fc55ebf 
…ang#42995)

(cherry picked from commit f317d57)
LilithHafner pushed a commit to LilithHafner/julia that referenced this pull request Feb 22, 2022
@DilumAluthge @LilithHafner
CI (Buildkite): use a TMPDIR that is backed by real storage (JuliaL…
064ff76 
…ang#42995)
LilithHafner pushed a commit to LilithHafner/julia that referenced this pull request Mar 8, 2022
@DilumAluthge @LilithHafner
CI (Buildkite): use a TMPDIR that is backed by real storage (JuliaL…
692669a 
…ang#42995)
staticfloat pushed a commit that referenced this pull request Dec 23, 2022
@DilumAluthge @staticfloat
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
079b1f3 
(cherry picked from commit f317d57)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Keno Keno Keno left review comments

@staticfloat staticfloat Awaiting requested review from staticfloat

Assignees
No one assigned
Labels
ci Continuous integration
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@DilumAluthge @Keno @KristofferC