CI (Buildkite): move the `sanitizers` builders (e.g. `asan`) to the experimental pipeline #41606

DilumAluthge · 2021-07-15T20:07:14Z

If you look at the last nine builds on master (https://buildkite.com/julialang/julia-master/builds?branch=master), the asan builder is failing in three of them.

This PR moves it to the experimental pipeline.

So it will still run, but:

The build badge in the README won't be red.
We won't get notified in Slack every time it fails.
It will only run on pushes to master and PRs to master. It won't run on pushes to release-* or PRs to release-*.

tkf · 2021-07-15T20:39:00Z

One failure (https://buildkite.com/julialang/julia-master/builds/1975#60f9f268-e332-4ed7-b184-a3747c6f60e9/1072) is the same as #41571 (comment). Since this happens after the build is successful, we can simply turn off only the check script if it were the only issue.

Two failures are due to

Generating REPL precompile statements... 13/31# Received cancellation signal, interrupting

https://buildkite.com/julialang/julia-master/builds/1929#318bb162-2b49-4817-b304-8c959b0b620e/192-1070

https://buildkite.com/julialang/julia-master/builds/1933#7f406849-c4cb-4504-bf87-791a03cede43/180-1058

But I'm not sure where # Received cancellation signal, interrupting is coming from. Is this something built into buildkite?

tkf

It's unfortunate but disabling this until we have a solution (or moving to the experimental pipeline) sounds good to me.

I don't have a good idea about how to debug generate_precompile.jl running in the CI though. Maybe it'd be better if we dump the debug output in CI on failure:

julia/contrib/generate_precompile.jl

Lines 229 to 231 in 1c92e0d

    
           function generate_precompile_statements() 
        
               start_time = time_ns() 
        
               debug_output = devnull # or stdout

DilumAluthge · 2021-07-16T05:18:38Z

Let's do the experimental pipeline, instead of disabling it completely.

I'll push those changes to this branch.

DilumAluthge · 2021-07-17T06:37:09Z

Alright, I've moved it to the experimental pipeline instead of disabling it completely.

…xperimental pipeline

DilumAluthge · 2021-07-17T07:49:19Z

@staticfloat Any idea what's going on with the failing asan job? Looks like I broke something when I moved it to the experimental pipeline.

Downloading Julia 1.6
Source URL: https://julialang-s3.julialang.org/bin/linux/x64/1.6/julia-1.6-latest-linux-x86_64.tar.gz
Error: The plugin julia pre-command hook exited with status 60

maleadt · 2021-07-19T16:56:40Z

But I'm not sure where # Received cancellation signal, interrupting is coming from. Is this something built into buildkite?

That happens when a job times out.

staticfloat · 2021-07-20T00:19:51Z

It also happens if we have build cancellation enabled, so if you push to the same branch while it's running.

staticfloat · 2021-07-20T00:20:19Z

I will also note that the experimental pipeline is reporting statuses to this PR; we probably don't want that.

DilumAluthge · 2021-07-20T00:22:47Z

I will also note that the experimental pipeline is reporting statuses to this PR; we probably don't want that.

Ah yes I forgot to comment out the lines that post the status; I'll do that now.

staticfloat · 2021-07-20T00:24:20Z

Looks like I broke something when I moved it to the experimental pipeline.

The curl process is failing with error code 60; that means some kind of HTTPS error. My guess is that something is broken about the rootfs image it's using; what are the differences between the v1 and v1.1 versions?

DilumAluthge · 2021-07-20T00:29:38Z

Looks like I broke something when I moved it to the experimental pipeline.

The curl process is failing with error code 60; that means some kind of HTTPS error. My guess is that something is broken about the rootfs image it's using; what are the differences between the v1 and v1.1 versions?

So the differences are:

v1.1 includes bash and less (CI: Add more Debian packages to the llvm-passes rootfs image #41544)
I built v1.1 inside Docker.

DilumAluthge · 2021-07-20T05:59:04Z

I think the problem was that I built v1.1 inside Docker.

I've built v2.0 of the llvm_passes rootfs using GitHub Actions CI on the https://github.com/JuliaCI/rootfs-images repo. Let's see if that fixes things.

Since the asan builder won't post a commit status on this PR, follow along here: https://buildkite.com/julialang/julia-master-experimental

DilumAluthge · 2021-07-20T06:44:02Z

I've built v2.0 of the llvm_passes rootfs using GitHub Actions CI on the https://github.com/JuliaCI/rootfs-images repo. Let's see if that fixes things.

It did in fact fix things!

https://buildkite.com/julialang/julia-master-experimental/builds/137

DilumAluthge · 2021-07-20T07:15:19Z

Since asan is now passing (https://buildkite.com/julialang/julia-master-experimental/builds/137), this PR is now ready to merge once CI is green.

…xperimental pipeline (#41606) (cherry picked from commit c95448f)

DilumAluthge requested review from tkf and staticfloat July 15, 2021 20:07

DilumAluthge requested a review from a team as a code owner July 15, 2021 20:07

DilumAluthge removed the request for review from a team July 15, 2021 20:07

DilumAluthge added the ci Continuous integration label Jul 15, 2021

DilumAluthge marked this pull request as draft July 15, 2021 20:30

tkf approved these changes Jul 15, 2021

View reviewed changes

DilumAluthge added backport 1.6 Change should be backported to release-1.6 backport 1.7 labels Jul 16, 2021

DilumAluthge changed the title ~~CI (Buildkite): disable the sanitizers builders (e.g. asan)~~ CI (Buildkite): move the sanitizers builders (e.g. asan) to the experimental pipeline Jul 17, 2021

DilumAluthge marked this pull request as ready for review July 17, 2021 06:33

CI (Buildkite): move the sanitizers builders (e.g. asan) to the e…

b4685aa

…xperimental pipeline

DilumAluthge force-pushed the dpa/disable-asan-buildkite branch from f542e4b to b4685aa Compare July 17, 2021 07:47

This was referenced Jul 19, 2021

Backports for julia-1.6.3 #41554

Merged

release-1.7: Backports for 1.7-RC1/1.7-beta4 #41499

Merged

Turn off the asan commit status

9b5a4bc

Use v2.0 of the llvm_passes rootfs image

6b4bce3

DilumAluthge mentioned this pull request Jul 20, 2021

CI: delete the .buildkite/rootfs_images directory #41648

Merged

DilumAluthge added the merge me PR is reviewed. Merge when all tests are passing label Jul 20, 2021

DilumAluthge mentioned this pull request Jul 20, 2021

CI: add the sanitizers pipelines (e.g. ASAN) to Buildkite #41530

Merged

DilumAluthge merged commit c95448f into master Jul 20, 2021

DilumAluthge deleted the dpa/disable-asan-buildkite branch July 20, 2021 09:58

DilumAluthge removed the merge me PR is reviewed. Merge when all tests are passing label Jul 20, 2021

KristofferC pushed a commit that referenced this pull request Jul 26, 2021

CI (Buildkite): move the sanitizers builders (e.g. asan) to the e…

1ddf238

…xperimental pipeline (#41606) (cherry picked from commit c95448f)

KristofferC pushed a commit that referenced this pull request Jul 26, 2021

CI (Buildkite): move the sanitizers builders (e.g. asan) to the e…

8d49537

…xperimental pipeline (#41606) (cherry picked from commit c95448f)

KristofferC removed the backport 1.7 label Aug 3, 2021

KristofferC mentioned this pull request Aug 25, 2021

release-1.7: Backports for 1.7-RC1 #41781

Merged

63 tasks

KristofferC pushed a commit that referenced this pull request Aug 31, 2021

CI (Buildkite): move the sanitizers builders (e.g. asan) to the e…

f6804d6

…xperimental pipeline (#41606) (cherry picked from commit c95448f)

KristofferC pushed a commit that referenced this pull request Sep 3, 2021

CI (Buildkite): move the sanitizers builders (e.g. asan) to the e…

15cb1fd

…xperimental pipeline (#41606) (cherry picked from commit c95448f)

KristofferC removed the backport 1.6 Change should be backported to release-1.6 label Sep 7, 2021

staticfloat pushed a commit that referenced this pull request Dec 23, 2022

CI (Buildkite): move the sanitizers builders (e.g. asan) to the e…

bde55be

…xperimental pipeline (#41606) (cherry picked from commit c95448f)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI (Buildkite): move the `sanitizers` builders (e.g. `asan`) to the experimental pipeline #41606

CI (Buildkite): move the `sanitizers` builders (e.g. `asan`) to the experimental pipeline #41606

DilumAluthge commented Jul 15, 2021 •

edited

Loading

tkf commented Jul 15, 2021

tkf left a comment

DilumAluthge commented Jul 16, 2021 •

edited

Loading

DilumAluthge commented Jul 17, 2021 •

edited

Loading

DilumAluthge commented Jul 17, 2021 •

edited

Loading

maleadt commented Jul 19, 2021

staticfloat commented Jul 20, 2021

staticfloat commented Jul 20, 2021

DilumAluthge commented Jul 20, 2021

staticfloat commented Jul 20, 2021

DilumAluthge commented Jul 20, 2021 •

edited

Loading

DilumAluthge commented Jul 20, 2021 •

edited

Loading

DilumAluthge commented Jul 20, 2021

DilumAluthge commented Jul 20, 2021

	function generate_precompile_statements()
	start_time = time_ns()
	debug_output = devnull # or stdout

CI (Buildkite): move the sanitizers builders (e.g. asan) to the experimental pipeline #41606

CI (Buildkite): move the sanitizers builders (e.g. asan) to the experimental pipeline #41606

Conversation

DilumAluthge commented Jul 15, 2021 • edited Loading

tkf commented Jul 15, 2021

tkf left a comment

Choose a reason for hiding this comment

DilumAluthge commented Jul 16, 2021 • edited Loading

DilumAluthge commented Jul 17, 2021 • edited Loading

DilumAluthge commented Jul 17, 2021 • edited Loading

maleadt commented Jul 19, 2021

staticfloat commented Jul 20, 2021

staticfloat commented Jul 20, 2021

DilumAluthge commented Jul 20, 2021

staticfloat commented Jul 20, 2021

DilumAluthge commented Jul 20, 2021 • edited Loading

DilumAluthge commented Jul 20, 2021 • edited Loading

DilumAluthge commented Jul 20, 2021

DilumAluthge commented Jul 20, 2021

CI (Buildkite): move the `sanitizers` builders (e.g. `asan`) to the experimental pipeline #41606

CI (Buildkite): move the `sanitizers` builders (e.g. `asan`) to the experimental pipeline #41606

DilumAluthge commented Jul 15, 2021 •

edited

Loading

DilumAluthge commented Jul 16, 2021 •

edited

Loading

DilumAluthge commented Jul 17, 2021 •

edited

Loading

DilumAluthge commented Jul 17, 2021 •

edited

Loading

DilumAluthge commented Jul 20, 2021 •

edited

Loading

DilumAluthge commented Jul 20, 2021 •

edited

Loading