Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zombie tests build pipeline rework #2159

Merged
merged 101 commits into from
Jul 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
41af97e
add usrmerge to zombie-builder build step
qdrvm-ci Jun 16, 2024
c10fb9a
new zombie_builder.Dockerfile
qdrvm-ci Jun 16, 2024
d5fd4ac
testing ci for zombie-builder
qdrvm-ci Jun 16, 2024
4b0c825
Replace minideb with the official Rust image.
qdrvm-ci Jun 16, 2024
eb73a91
testing images for zombie_tester
qdrvm-ci Jun 16, 2024
1ee01eb
test pkg dependencies for zombie-builder
qdrvm-ci Jun 16, 2024
d3e6ee2
kagome-dev minideb-testing Docker
qdrvm-ci Jun 16, 2024
17d70ee
add libraries for zombie_builder docker image
qdrvm-ci Jun 16, 2024
fdde097
enable cache on zombie-builder step
qdrvm-ci Jun 17, 2024
e73ea02
zombie builder image group update
qdrvm-ci Jun 17, 2024
294b0b4
change repo for zombie builder, separate builder steps
qdrvm-ci Jun 18, 2024
da8da1c
add working directory
qdrvm-ci Jun 18, 2024
928d9db
update image path for zombie-builder
qdrvm-ci Jun 18, 2024
03fd1ab
building_binary step for zombie builder
qdrvm-ci Jun 18, 2024
147b86c
github inputs test
qdrvm-ci Jun 19, 2024
1a5b8f7
test sccache
qdrvm-ci Jun 20, 2024
0a9ba33
Exit code disabled
qdrvm-ci Jun 20, 2024
ab18767
check sccache working
qdrvm-ci Jun 20, 2024
de9500b
Revert "check sccache working"
qdrvm-ci Jun 20, 2024
aaaad40
make dirs when CI zombie-builder starts
qdrvm-ci Jun 20, 2024
52f5a0b
show build result for polkadot_binary
qdrvm-ci Jun 21, 2024
a40bc37
change runner for zombie-builder
qdrvm-ci Jun 21, 2024
a8226c0
zombie builder, change paths
qdrvm-ci Jun 21, 2024
7a06a4a
build apt package pipeline
qdrvm-ci Jun 22, 2024
5f40ded
docker-builder - add debug info
qdrvm-ci Jun 22, 2024
d8ab7e7
zombie-builder ci architecture fix
qdrvm-ci Jun 22, 2024
5cfe4f5
New kagome dev minideb builder
qdrvm-ci Jun 22, 2024
f168614
temporarily ci changes (autotests)
qdrvm-ci Jun 22, 2024
765c3b5
enable kagome-builder-deb workflow
qdrvm-ci Jun 22, 2024
95d09ef
zombie_builder and zombie_tester updated
qdrvm-ci Jun 22, 2024
caffab9
zombie_builder syntax fixes
qdrvm-ci Jun 22, 2024
c294c78
zombie builder - build logic variable added
qdrvm-ci Jun 23, 2024
50a047b
save zombie-builder env to file and restore it
qdrvm-ci Jun 23, 2024
8c3e908
New build kagome Makefile for kagome-dev
qdrvm-ci Jun 23, 2024
67d11e7
Zombie-Tests kagome build, release by default
qdrvm-ci Jun 23, 2024
ecfce8a
new build kagome pipeline - activate cache
qdrvm-ci Jun 23, 2024
ae589ca
enable show info when build deb packet
qdrvm-ci Jun 23, 2024
cb1df90
update dependencies for packet
qdrvm-ci Jun 23, 2024
3fd35c6
test update
qdrvm-ci Jun 23, 2024
a1d72e0
Check package version exists in build steps
qdrvm-ci Jun 24, 2024
a797b26
show package version in build step
qdrvm-ci Jun 24, 2024
e1ab5dc
add date to version apt package name
qdrvm-ci Jun 24, 2024
0c64d8f
Test zombie-tests job
qdrvm-ci Jun 24, 2024
712be49
update run zombienet test
qdrvm-ci Jun 24, 2024
41b48dd
update Google Auth for zombie-tests ci
qdrvm-ci Jun 24, 2024
985697c
Run zombie-tests on actions-runner-controller
qdrvm-ci Jun 24, 2024
6385490
wasmedge get SHA script
qdrvm-ci Jun 24, 2024
4f68bc2
Runtime cache build packet - first iteration
qdrvm-ci Jun 24, 2024
46d7138
ci runtime build fix
qdrvm-ci Jun 24, 2024
8090838
Kagome runtime ci update
qdrvm-ci Jun 24, 2024
ff582d2
zombie test -> ubuntu latest
qdrvm-ci Jun 24, 2024
d65b095
Revert "zombie test -> ubuntu latest"
qdrvm-ci Jun 24, 2024
c8052be
Merge branch 'master' into zombie_builder_rework
qdrvm-ci Jun 24, 2024
ba1e523
add hunter cache to safe.directory
qdrvm-ci Jun 24, 2024
86b9ba6
CI change cache using config
qdrvm-ci Jun 24, 2024
177e57c
Enable swap on Ubuntu zombie-tests
qdrvm-ci Jun 24, 2024
7ed3426
change cache logic in kagome build step
qdrvm-ci Jun 25, 2024
6e94c98
Reduce kagome validators in 0001 test
kamilsa Jun 25, 2024
98c344a
Merge remote-tracking branch 'origin/fix/reduce-kagome-vals' into zom…
qdrvm-ci Jun 25, 2024
d824811
change swap size - 4G (for ubuntu runners)
qdrvm-ci Jun 25, 2024
37154e0
update cache save settings - build kagome step
qdrvm-ci Jun 25, 2024
7c22a6e
Fix zndsl
kamilsa Jun 25, 2024
2f298b8
zombie-tests runs on actions-runner-controller
qdrvm-ci Jul 5, 2024
5d66fac
disable swap
qdrvm-ci Jul 5, 2024
c349a39
disable broken zombie tests
qdrvm-ci Jul 5, 2024
0460253
zombie test - kagome build - google cache
qdrvm-ci Jul 8, 2024
53ab22a
disable broken zombie-tests
qdrvm-ci Jul 8, 2024
5c02cb4
Merge branch 'ci/working_zombie_tests' into ci/builld_pipeline_rework
qdrvm-ci Jul 8, 2024
fdf34ee
zombie tests - disable always save cache, change cache action
qdrvm-ci Jul 8, 2024
4b35b80
return github action version
qdrvm-ci Jul 8, 2024
ff26b6b
change gcp github action version
qdrvm-ci Jul 8, 2024
e93c413
testing actions step
qdrvm-ci Jul 8, 2024
57f4359
return github cache save
qdrvm-ci Jul 9, 2024
0fa54d1
select working zombie tests
qdrvm-ci Jul 9, 2024
e8df89d
runtime cache build update
Jul 20, 2024
ee87dbf
Rebuild runtime option in CI
Jul 21, 2024
78b23a8
update zombienet simlinks for tester image
Jul 21, 2024
5d167c0
Zombie-Tests runs-on -> ubuntu-latest
Jul 21, 2024
917b89c
Build runtime -> actions-runner-controller
Jul 21, 2024
cbcd448
Clean old files, clean Zombie tests Makefile, change version numeric
Jul 21, 2024
60a8e13
Zombienet Makefile new line fix
Jul 21, 2024
2df8391
Clear Kagome docker Build makefile
Jul 21, 2024
c0a0892
use default kagome_dev version in runtime_cache ci build
Jul 21, 2024
de145d1
runtime cache version ci fix
Jul 21, 2024
2787f14
Add kagome-crates dependency
Harrm Jul 23, 2024
0ba14e2
Switch to tag
Harrm Jul 23, 2024
52a8151
Update main build pipeline (zombie tests new pipeline)
qdrvm-ci Jul 23, 2024
d47d457
fix ci errors in test.yaml
qdrvm-ci Jul 23, 2024
b37996c
Build type for Zombie tests added
qdrvm-ci Jul 23, 2024
e3b940b
code cleanup
qdrvm-ci Jul 23, 2024
3a35273
Zombie tests in main pipeline -> actions-runner-controller
qdrvm-ci Jul 23, 2024
95c0f3a
Merge remote-tracking branch 'origin/feature/kagome-crates' into ci/b…
qdrvm-ci Jul 24, 2024
1686c92
change mount dirs for docker build
qdrvm-ci Jul 24, 2024
6e16355
cleaning cache and default build typo for main build workflow
qdrvm-ci Jul 24, 2024
dccbd34
disable external project
qdrvm-ci Jul 24, 2024
fa28047
add CI env to workflows
qdrvm-ci Jul 28, 2024
b113682
-Werror param support for CI building
qdrvm-ci Jul 28, 2024
6f7208d
add check -Werror compiler support
qdrvm-ci Jul 28, 2024
4501df7
Change the order of applying -Werror
qdrvm-ci Jul 28, 2024
718ff88
add_flag cmake function update
qdrvm-ci Jul 29, 2024
395dbf4
Merge branch 'master' into ci/builld_pipeline_rework
zerg-su Jul 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .github/workflows/action-validator_darwin_arm64
Binary file not shown.
22 changes: 22 additions & 0 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#
# Copyright Quadrivium LLC
# All Rights Reserved
# SPDX-License-Identifier: Apache-2.0
#

name: Manual Workflow Dispatch

on:
workflow_dispatch:

jobs:
example-job:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Print template message
run: echo "This is a template for testing workflows in branches."

102 changes: 102 additions & 0 deletions .github/workflows/kagome-builder-deb.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
#
# Copyright Quadrivium LLC
# All Rights Reserved
# SPDX-License-Identifier: Apache-2.0
#

name: Kagome Builder | Debian (Bookworm)

on:
# schedule:
# - cron: '0 2 * * 0' # Every Sunday at 2:00 AM
workflow_dispatch:
inputs:
kagome_image_rebuild:
description: 'Rebuild Kagome Builder Minideb image'
required: false
default: 'false'
type: 'boolean'
runtime_cache_rebuild:
description: 'Rebuild Kagome Runtime Cache'
required: false
default: 'false'
type: 'boolean'

env:
DOCKER_REGISTRY_PATH: ${{ secrets.GCP_REGISTRY }}/${{ secrets.GCP_PROJECT }}/kagome-dev/

jobs:
build_polkadot_builder:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name : "Authenticate with Google Cloud"
uses: 'google-github-actions/auth@v2'
with:
credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}

- name: "Set up Cloud SDK"
uses: 'google-github-actions/setup-gcloud@v2'

- name: "Configure Docker for GCR"
run: |
gcloud auth configure-docker --quiet
gcloud auth configure-docker ${{ secrets.GCP_REGISTRY }} --quiet

- name: "Check if image tag exists"
id: check-tag
working-directory: ./housekeeping/docker/kagome-dev
run: |
IMAGE_TAG=$(grep '^MINIDEB_IMAGE' Makefile | awk -F '[@:]' '{print $3}' | cut -c 1-7)_rust-$(grep '^RUST_VERSION' Makefile | sed 's/.*?= //')
KAGOME_BUILDER_DEB_IMAGE=${DOCKER_REGISTRY_PATH}kagome_builder_deb:${IMAGE_TAG}
TAG_EXISTS=$(docker manifest inspect ${KAGOME_BUILDER_DEB_IMAGE} > /dev/null 2>&1 && echo "true" || echo "false")
echo "exists=$TAG_EXISTS" >> $GITHUB_ENV
echo "kagome_builder_deb=KAGOME_BUILDER_DEB_IMAGE" >> GITHUB_OUTPUT

- name: "Build Polkadot Builder"
working-directory: ./housekeeping/docker/kagome-dev
if: env.exists == 'false' || github.event.inputs.kagome_image_rebuild == 'true'
run: make kagome_builder_deb \
DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}

- name: "Push Polkadot Builder"
working-directory: ./housekeeping/docker/kagome-dev
if: env.exists == 'false' || github.event.inputs.kagome_image_rebuild == 'true'
run: make kagome_builder_deb_push \
DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}

build_kagome_runtime:
runs-on: [ actions-runner-controller ]
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name : "Authenticate with Google Cloud"
uses: 'google-github-actions/auth@v2'
with:
credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}

- name: "Set up Cloud SDK"
uses: 'google-github-actions/setup-gcloud@v2'

- name: "Configure Docker for GCR"
run: |
gcloud auth configure-docker --quiet
gcloud auth configure-docker ${{ secrets.GCP_REGISTRY }} --quiet

- name: "Build Runtime Cache"
working-directory: ./housekeeping/docker/kagome-dev
if: github.event.inputs.runtime_cache_rebuild == 'true'
run: make runtime_cache \
DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}

- name: "Push Runtime Cache"
working-directory: ./housekeeping/docker/kagome-dev
if: github.event.inputs.runtime_cache_rebuild == 'true'
run: make upload_apt_package_runtime \
DOCKER_REGISTRY_PATH=${DOCKER_REGISTRY_PATH}

6 changes: 6 additions & 0 deletions .github/workflows/telegram_notify.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
#
# Copyright Quadrivium LLC
# All Rights Reserved
# SPDX-License-Identifier: Apache-2.0
#

name: Telegram Notification

on:
Expand Down
Loading
Loading