Setup buildkite as an alternative to AZP #4771

vchuravy · 2022-04-16T21:12:28Z

TODO:

Nail down artifact upload story
- Using buildkite upload is neat since we get something in the UI
- Ideally we should upload directly to our own buckets, but need to make sure not to leak keys
  - Should be good citizens and not cause tremendous traffic to BK buckets
  - Can we generate ephermal keys? We could inject those from the generator script
Test jll_init and register steps
CCACHE/distcc
Limit concurreny of registration
@giordano hook up remarks from binary verifier to BK annotations (maybe a custom logger?)
Party like it's the 90s
Double check that ccache is actually working

.buildkite/generator.jl

vchuravy · 2022-04-16T22:20:52Z

Ah great we can also do logs extraction to create warnings https://buildkite.com/docs/agent/v3/cli-annotate

giordano · 2022-04-17T19:01:53Z

Some comments about what we currently do in Azure Pipelines. In terms of race conditions:

the JLL repo init job is not safe: in some cases we can build multiple versions of the same package (when they're split in different directories, e.g. CUDA, OpenBLAS or LLVM)
the build and package registration jobs are safe, however in some cases registration may cause merge conflicts in the registry (but we have relatively little control over that). Note that currently we do registrations in sequence (Don't allow parallel registration #208)

Note that buildkite is currently running, and failing, also on master: https://buildkite.com/julialang/yggdrasil/builds/4#d85857dd-92f6-43cd-82b5-2827d84e5602. It'd be nice to disable it until this is working

vchuravy · 2022-04-18T07:56:35Z

It'd be nice to disable it until this is working

Done.

.buildkite/utils.jl

.buildkite/generator.jl

giordano

First round of comments.

Reminder that skipping build isn't implemented at the moment.

.buildkite/generator.jl

.buildkite/pipeline.yml

vchuravy · 2022-04-19T21:21:47Z

So the current failure is real and I don't know why and how.
ERROR: LoadError: failed process: Process(git merge-base --fork-point remotes/origin/master HEAD, ProcessExited(1)) [1]

It seems to occur always when master is ahead of this feature branch, but git does not give a useful error.

giordano · 2022-04-19T21:23:07Z

I guess that's because this doesn't run on the merge commit (which is my gripe with buildkite) 🙃

DilumAluthge · 2022-04-19T21:25:07Z

When I have some free time, I want to get https://github.com/JuliaCI/pull-request-merge-commit-buildkite-plugin working. Then you would just stick that plugin at the top of your list of plugins, and it would automatically take care of checking out the merge commit.

Don't use it yet though, it doesn't work.

.buildkite/generator.jl

vchuravy · 2022-04-19T21:36:36Z

I guess that's because this doesn't run on the merge commit (which is my gripe with buildkite) 🙃

It's particularly weird since I can't reproduce locally

DilumAluthge · 2022-04-20T01:09:34Z

Alright, the minimum functionality of https://github.com/JuliaCI/merge-commit-buildkite-plugin is now there.

Basically, you just add this to the top of your plugin list:

- JuliaCI/merge-commit: ~

So e.g.

steps:
  - plugins:
    - JuliaCI/merge-commit: ~

You'll want to make sure you do this for every step.

Note: eventually, you'll want to replace JuliaCI/merge-commit: ~ with JuliaCI/merge-commit#v1: ~, but that'll have to wait until we first make a tag on the https://github.com/JuliaCI/merge-commit-buildkite-plugin repo.

.buildkite/pipeline.yml

.buildkite/0_webui.yml

.buildkite/generator.jl

DilumAluthge · 2022-04-20T07:33:59Z

You should run the generator inside Sandbox.jl.

vchuravy · 2022-04-20T08:05:43Z

You should run the generator inside Sandbox.jl.

How can I tell if I am? The cp error just turned out to be a user error

.buildkite/generator.jl

DilumAluthge · 2022-04-21T02:29:33Z

It looks like we are now checking out the merge commit correctly!

.buildkite/utils.jl

vchuravy · 2022-04-25T16:51:00Z

.buildkite/utils.jl

+        :plugins => plugins(),
+        :timeout_in_minutes => 30,
+        :concurrency => 1,
+        :concurrency_group => "yggdrasil/register",


Maybe do project name here

vchuravy · 2022-04-25T16:51:13Z

.buildkite/utils.jl

+        :label => "register -- $NAME",
+        :agents => agent(),
+        :plugins => plugins(),
+        :timeout_in_minutes => 30,


Look at time out

.buildkite/utils.jl

vchuravy · 2023-01-07T16:02:08Z

@giordano sadly NUMA has no warnings, do you have suggestions for a package that is fast and does emit some warnings?

giordano · 2023-01-07T18:10:10Z

GR has some hundreds of warnings and it shouldn't take too long: https://dev.azure.com/JuliaPackaging/Yggdrasil/_build/results?buildId=24439&view=results

vchuravy · 2023-01-14T20:28:43Z

From Elliot:

Generator job that analyzes the diff, and launches pipelines based on that.

Builder jobs that do the actual building. We want these to be skipped if they’re building the same thing as was built previously, so we’ll use coppermind (with cryptic, to store the S3 access key) to skip the actual build step if a treehash match is found

Registration job that downloads the binaries from buildkite artifacts again (since coppermind’s caching on S3 is an implementation detail, not the true storage location), then uploads them to GH releases

.buildkite/NOTES.md

.buildkite/path_processors/per-project

.buildkite/pipeline.yml

.buildkite/generator.jl

vchuravy · 2023-01-15T15:21:46Z

.buildkite/utils.jl

+            "inputs" => [
+                PROJECT,
+                ".ci/",
+                # ?meta.json


@staticfloat in AZP we hash in the meta.json Not sure how to do that here

vchuravy · 2023-01-16T13:13:29Z

So the last steps are:

Use a token for jlbuild instead one of mine
Improve Add buildkite logger BinaryBuilder.jl#1255 (but we can do that separatly and I can revert the BB bump here)

giordano

Mentioned in the CI call, but repeating it also here: we should implement also here the [skip build] keyword, which doesn't skip the CI job completely, only the build steps:

Yggdrasil/azure-pipelines.yml

Lines 32 to 49 in 8e7a9c4

    
                 # Keyword to be used in the commit message to skip a rebuild 
        
                 SKIP_BUILD_COOKIE="[skip build]" 
        
                 # This variable will tell us whether we want to skip the build 
        
                 export SKIP_BUILD="false" 
        
                 if [[ $(Build.Reason) == "PullRequest" ]]; then 
        
                     # If we're on a PR though, we look at the entire branch at once 
        
                     TARGET_BRANCH="remotes/origin/$(System.PullRequest.TargetBranch)" 
        
                     COMPARE_AGAINST=$(git merge-base --fork-point ${TARGET_BRANCH} HEAD) 
        
                     git fetch origin "refs/pull/$(System.PullRequest.PullRequestNumber)/head:refs/remotes/origin/pr/$(System.PullRequest.PullRequestNumber)" 
        
                     if [[ "$(git show -s --format=%B origin/pr/$(System.PullRequest.PullRequestNumber))" == *"${SKIP_BUILD_COOKIE}"* ]]; then 
        
                         SKIP_BUILD="true" 
        
                     fi 
        
                 else 
        
                     if [[ "$(git show -s --format=%B)" == *"${SKIP_BUILD_COOKIE}"* ]]; then 
        
                         SKIP_BUILD="true" 
        
                     fi 
        
                 fi

And this will be used during registration to re-use the artifacts from the previous build:

Yggdrasil/.ci/register_package.jl

Line 96 in 8e7a9c4

skip_build = get(ENV, "SKIP_BUILD", "false") == "true"

giordano · 2023-01-16T13:16:19Z

H/HelloWorldC/build_tarballs.jl

@@ -3,6 +3,7 @@ using BinaryBuilder
 name = "HelloWorldC"
 version = v"1.2.1"

+


Bump the version.

giordano · 2023-01-16T13:19:18Z

.ci/register_package.jl

+function download_cached_binaries(download_dir)
+    NAME = ENV["NAME"]
+    PROJECT = ENV["PROJECT"]
+    artifacts = "$(PROJECT)/products/$(NAME)*.tar.gz"


How does globbing work in the Cmd below?

giordano · 2023-01-16T17:53:29Z

Looking at JuliaBinaryWrappers/HelloWorldC_jll.jl@bd36c4c I believe you aren't exporting the environment variable

YGGDRASIL=true

are you? That's currently set in https://github.com/JuliaPackaging/Yggdrasil/blob/8e7a9c4f1761fcb71c8d2e33688d36ff6fb9f9d2/.ci/azp_agent/.env, so you may have missed it.

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

giordano reviewed Apr 16, 2022

View reviewed changes

.buildkite/generator.jl Outdated Show resolved Hide resolved

vchuravy force-pushed the bk/initial branch 3 times, most recently from 81e570d to 9c10f73 Compare April 17, 2022 11:35

giordano added the meta 🌳 Issues and pull requested related to Yggdrasil itself label Apr 18, 2022

DilumAluthge added the ci continuous integration label Apr 18, 2022

DilumAluthge reviewed Apr 18, 2022

View reviewed changes

.buildkite/utils.jl Show resolved Hide resolved

vchuravy force-pushed the bk/initial branch from 94ff54b to e4b8ee7 Compare April 18, 2022 09:17

vchuravy requested review from giordano and staticfloat April 18, 2022 09:24

vchuravy commented Apr 18, 2022

View reviewed changes

.buildkite/generator.jl Outdated Show resolved Hide resolved

giordano reviewed Apr 19, 2022

View reviewed changes

.buildkite/generator.jl Outdated Show resolved Hide resolved

DilumAluthge reviewed Apr 20, 2022

View reviewed changes

.buildkite/pipeline.yml Show resolved Hide resolved

DilumAluthge reviewed Apr 20, 2022

View reviewed changes

.buildkite/0_webui.yml Show resolved Hide resolved

vchuravy commented Apr 20, 2022

View reviewed changes

.buildkite/generator.jl Outdated Show resolved Hide resolved

vchuravy force-pushed the bk/initial branch from 1c2b5ee to e096798 Compare April 20, 2022 08:20

giordano reviewed Apr 20, 2022

View reviewed changes

.buildkite/generator.jl Outdated Show resolved Hide resolved

giordano reviewed Apr 20, 2022

View reviewed changes

.buildkite/generator.jl Outdated Show resolved Hide resolved

vchuravy commented Apr 21, 2022

View reviewed changes

.buildkite/utils.jl Outdated Show resolved Hide resolved

vchuravy commented Apr 21, 2022

View reviewed changes

.buildkite/utils.jl Outdated Show resolved Hide resolved

vchuravy commented Apr 25, 2022

View reviewed changes

.buildkite/utils.jl Outdated Show resolved Hide resolved

giordano mentioned this pull request Nov 10, 2022

Run only a subset of tests on PRs that only modify the manual JuliaCI/julia-buildkite#243

Open

vchuravy force-pushed the bk/initial branch from 73f9245 to b1efda2 Compare January 7, 2023 15:45

giordano reviewed Jan 14, 2023

View reviewed changes

.buildkite/NOTES.md Outdated Show resolved Hide resolved

.buildkite/path_processors/per-project Outdated Show resolved Hide resolved

.buildkite/pipeline.yml Show resolved Hide resolved

.buildkite/generator.jl Outdated Show resolved Hide resolved

vchuravy commented Jan 15, 2023

View reviewed changes

vchuravy force-pushed the bk/initial branch from f6c9303 to 944ede2 Compare January 16, 2023 13:11

vchuravy marked this pull request as ready for review January 16, 2023 13:12

vchuravy requested review from DilumAluthge and giordano January 16, 2023 13:13

giordano reviewed Jan 16, 2023

View reviewed changes

Setup buildkite as an alternative to AZP

f35e918

Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com>

vchuravy force-pushed the bk/initial branch from 16910dc to f35e918 Compare January 17, 2023 13:58

vchuravy merged commit a1b9ad5 into master Jan 17, 2023

vchuravy deleted the bk/initial branch January 17, 2023 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup buildkite as an alternative to AZP #4771

Setup buildkite as an alternative to AZP #4771

vchuravy commented Apr 16, 2022 •

edited

Loading

vchuravy commented Apr 16, 2022

giordano commented Apr 17, 2022

vchuravy commented Apr 18, 2022

giordano left a comment

vchuravy commented Apr 19, 2022

giordano commented Apr 19, 2022

DilumAluthge commented Apr 19, 2022

vchuravy commented Apr 19, 2022

DilumAluthge commented Apr 20, 2022 •

edited

Loading

DilumAluthge commented Apr 20, 2022

vchuravy commented Apr 20, 2022

DilumAluthge commented Apr 21, 2022 •

edited

Loading

vchuravy Apr 25, 2022

vchuravy Apr 25, 2022

vchuravy commented Jan 7, 2023

giordano commented Jan 7, 2023

vchuravy commented Jan 14, 2023

vchuravy Jan 15, 2023

vchuravy commented Jan 16, 2023

giordano left a comment

giordano Jan 16, 2023

giordano Jan 16, 2023

giordano commented Jan 16, 2023

	# Keyword to be used in the commit message to skip a rebuild
	SKIP_BUILD_COOKIE="[skip build]"
	# This variable will tell us whether we want to skip the build
	export SKIP_BUILD="false"

	if [[ $(Build.Reason) == "PullRequest" ]]; then
	# If we're on a PR though, we look at the entire branch at once
	TARGET_BRANCH="remotes/origin/$(System.PullRequest.TargetBranch)"
	COMPARE_AGAINST=$(git merge-base --fork-point ${TARGET_BRANCH} HEAD)
	git fetch origin "refs/pull/$(System.PullRequest.PullRequestNumber)/head:refs/remotes/origin/pr/$(System.PullRequest.PullRequestNumber)"
	if [[ "$(git show -s --format=%B origin/pr/$(System.PullRequest.PullRequestNumber))" == "${SKIP_BUILD_COOKIE}" ]]; then
	SKIP_BUILD="true"
	fi
	else
	if [[ "$(git show -s --format=%B)" == "${SKIP_BUILD_COOKIE}" ]]; then
	SKIP_BUILD="true"
	fi
	fi

		@@ -3,6 +3,7 @@ using BinaryBuilder
		name = "HelloWorldC"
		version = v"1.2.1"

Setup buildkite as an alternative to AZP #4771

Setup buildkite as an alternative to AZP #4771

Conversation

vchuravy commented Apr 16, 2022 • edited Loading

TODO:

vchuravy commented Apr 16, 2022

giordano commented Apr 17, 2022

vchuravy commented Apr 18, 2022

giordano left a comment

Choose a reason for hiding this comment

vchuravy commented Apr 19, 2022

giordano commented Apr 19, 2022

DilumAluthge commented Apr 19, 2022

vchuravy commented Apr 19, 2022

DilumAluthge commented Apr 20, 2022 • edited Loading

DilumAluthge commented Apr 20, 2022

vchuravy commented Apr 20, 2022

DilumAluthge commented Apr 21, 2022 • edited Loading

vchuravy Apr 25, 2022

Choose a reason for hiding this comment

vchuravy Apr 25, 2022

Choose a reason for hiding this comment

vchuravy commented Jan 7, 2023

giordano commented Jan 7, 2023

vchuravy commented Jan 14, 2023

vchuravy Jan 15, 2023

Choose a reason for hiding this comment

vchuravy commented Jan 16, 2023

giordano left a comment

Choose a reason for hiding this comment

giordano Jan 16, 2023

Choose a reason for hiding this comment

giordano Jan 16, 2023

Choose a reason for hiding this comment

giordano commented Jan 16, 2023

vchuravy commented Apr 16, 2022 •

edited

Loading

DilumAluthge commented Apr 20, 2022 •

edited

Loading

DilumAluthge commented Apr 21, 2022 •

edited

Loading