Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run bridges zombienet tests on CI #2439

Merged
merged 93 commits into from
Jan 12, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
c6f8e61
try to build bridges + zombienet image
svyatonik Nov 22, 2023
1e369ee
fix
svyatonik Nov 22, 2023
4c06584
Merge branch 'master' into sv-bridges-zombienet-tests
svyatonik Nov 22, 2023
58e9981
another try
svyatonik Nov 23, 2023
da2bd72
typo
svyatonik Nov 23, 2023
baf85c9
add bridges.yml
svyatonik Nov 23, 2023
9c921c8
pass ZOMBIENET_IMAGE
svyatonik Nov 23, 2023
5e5c72b
typo
svyatonik Nov 23, 2023
5ccf7fe
use full image name
svyatonik Nov 23, 2023
dd7b631
fixes
svyatonik Nov 23, 2023
9f617dc
Merge branch 'master' into sv-bridges-zombienet-tests
svyatonik Nov 23, 2023
c2327a2
Merge branch 'master' into sv-bridges-zombienet-tests
svyatonik Nov 24, 2023
97b7666
dbg
svyatonik Nov 24, 2023
638924c
dbg
svyatonik Nov 24, 2023
8653c05
dbg++
svyatonik Nov 24, 2023
0ebc852
dbg++++
svyatonik Nov 24, 2023
81c9ec6
try with other zombienet?
svyatonik Nov 24, 2023
065a69e
install ps
svyatonik Nov 27, 2023
4afa6a8
fix
svyatonik Nov 27, 2023
fcc4216
yarn global add @polkadot/api-cli
svyatonik Nov 27, 2023
7172580
flush
svyatonik Dec 18, 2023
7393a97
flush
svyatonik Dec 18, 2023
54497f0
Merge branch 'master' into sv-bridges-zombienet-tests
svyatonik Dec 18, 2023
e0382ea
fix paths for CI
svyatonik Dec 18, 2023
8e20872
updated costs + constants
svyatonik Dec 18, 2023
b050f1a
npm install for generate_hex_encoded_call
svyatonik Dec 18, 2023
3230786
cc
svyatonik Dec 18, 2023
d107297
pushd -> cd
svyatonik Dec 19, 2023
3022c82
fix path
svyatonik Dec 19, 2023
f19668c
try to chear zombienet
svyatonik Dec 19, 2023
83d2177
retry failure
svyatonik Dec 19, 2023
ab99318
faster
svyatonik Dec 19, 2023
c562cd9
add test runner logs to artifacts
svyatonik Dec 19, 2023
f2cb940
fix path
svyatonik Dec 19, 2023
8a5ace0
fix path
svyatonik Dec 19, 2023
8a95a9e
one more try
svyatonik Dec 19, 2023
8c4a7b2
try paths again
svyatonik Dec 19, 2023
d79a43b
++
svyatonik Dec 20, 2023
1156753
try dummy file???
svyatonik Dec 20, 2023
cb127c9
continue debugging
svyatonik Dec 20, 2023
46b13f7
another try
svyatonik Dec 20, 2023
1f0e267
kill -9
svyatonik Dec 20, 2023
14ac859
temporary comment kill on exit
svyatonik Dec 20, 2023
b0c8cf6
ls
svyatonik Dec 20, 2023
a984032
try to cat zn logs from test runner
svyatonik Dec 21, 2023
accd8fc
try to cat logs
svyatonik Dec 22, 2023
fd32ee7
upd
svyatonik Dec 22, 2023
8e3b76d
try more
svyatonik Dec 22, 2023
67d0f16
more
svyatonik Dec 22, 2023
adad841
???!!!
svyatonik Dec 22, 2023
39ca333
tmp comment
svyatonik Dec 25, 2023
6a93477
try++
svyatonik Dec 25, 2023
82f8e64
logs?
svyatonik Dec 25, 2023
3a71ba6
fix scripts
svyatonik Dec 25, 2023
c3ebf00
xxx
svyatonik Dec 25, 2023
e02121c
too long lines
svyatonik Dec 25, 2023
960558f
why it works now?
svyatonik Dec 25, 2023
92d3b7f
Revert "why it works now?"
svyatonik Dec 26, 2023
49d3090
trigger CI again
svyatonik Dec 26, 2023
88d4c00
remove test files
svyatonik Dec 26, 2023
e9b058d
Merge branch 'master' into sv-bridges-zombienet-tests
svyatonik Dec 26, 2023
1e2b3b6
-
svyatonik Dec 26, 2023
ead14d2
attempt
svyatonik Dec 26, 2023
f581afa
fix
svyatonik Dec 26, 2023
911de8f
fix
svyatonik Dec 26, 2023
34d894f
++
svyatonik Dec 26, 2023
2d2a616
up @polkadot/api and @polkadot/util package version to latest
svyatonik Dec 28, 2023
0a25a83
remove unneeded file
svyatonik Dec 28, 2023
d61b5bc
remove unneeded file
svyatonik Dec 28, 2023
228bc1b
uncomment test changes to publish.yml
svyatonik Dec 28, 2023
cbe832d
remove unneeded files
svyatonik Dec 28, 2023
7cb9189
revert commented lines from zombienet.yml
svyatonik Dec 28, 2023
c1743e8
revert unneeded change
svyatonik Dec 29, 2023
1ce62ec
revert unneeded logs
svyatonik Dec 29, 2023
ab42c1d
revert on_exit changes
svyatonik Dec 29, 2023
523c0a4
--zombienet flag
svyatonik Dec 29, 2023
96111d0
some strange change
svyatonik Dec 29, 2023
558a9ee
customizable relay path
svyatonik Dec 29, 2023
c12391c
fix paths
svyatonik Jan 11, 2024
db93f6b
some fixes + polishing
svyatonik Jan 11, 2024
99e0519
print /tmp
svyatonik Jan 11, 2024
811afb3
also play with after_script
svyatonik Jan 11, 2024
43c7c04
revert commented code
svyatonik Jan 11, 2024
6df5381
Merge branch 'master' into sv-bridges-zombienet-tests
svyatonik Jan 11, 2024
d57e183
also removed unneeded comments
svyatonik Jan 11, 2024
3aafccb
Revert "revert commented code"
svyatonik Jan 11, 2024
0791aae
Merge branch 'master' into sv-bridges-zombienet-tests
svyatonik Jan 11, 2024
370b4ce
trigger CI
svyatonik Jan 11, 2024
86906f2
try to use regular binaries now?
svyatonik Jan 12, 2024
c4ba86f
ok, found the bad guy
svyatonik Jan 12, 2024
9b36e5c
leave comment + rename job
svyatonik Jan 12, 2024
327ad7c
relay_path -> relayer_path
svyatonik Jan 12, 2024
78bbc70
removed copying of obsolete logs
svyatonik Jan 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .gitlab/pipeline/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -372,3 +372,19 @@ build-subkey-linux:
# after_script: [""]
# tags:
# - osx

# bridges

# we need some non-binary artifacts in our bridges+zombienet image
prepare-bridges-zombienet-artifacts:
stage: build
extends:
- .docker-env
- .common-refs
- .collect-artifacts
before_script:
serban300 marked this conversation as resolved.
Show resolved Hide resolved
- mkdir -p ./artifacts/bridges/
script:
- cp -r bridges/zombienet ./artifacts/bridges
- cp -r cumulus/scripts ./artifacts/bridges/cumulus-scripts
- cp -r cumulus/zombienet/brige-hubs ./artifacts/bridges/cumulus-brige-hubs
18 changes: 18 additions & 0 deletions .gitlab/pipeline/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ publish-rustdoc:
--build-arg VCS_REF="${CI_COMMIT_SHA}"
--build-arg BUILD_DATE="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
--build-arg IMAGE_NAME="${IMAGE_NAME}"
--build-arg ZOMBIENET_IMAGE="${ZOMBIENET_IMAGE}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved the image definition to .zombienet-ref in this pr https://github.com/paritytech/polkadot-sdk/pull/2396/files#diff-037ea159eb0a7cb0ac23b851e66bee30fb838ee8d0d99fa331a1ba65283d37f7L33, so here you will not have the env var defined. I think you can extend from that ref but we should check.

  extends:
    - .zombienet-refs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems a bit too specific to our use case. I would try to make this more generic. Maybe something like
adding ${EXTRA_BUILDAH_ARGS} here and defining EXTRA_BUILDAH_ARGS = --build-arg ZOMBIENET_IMAGE=... in build-push-image-bridges-zombienet-tests if possible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to keep meaningful naming here - we could wrap everything in the EXTRA_BUILDAH_ARGS, making it hard to find the origins. But I could change if there's a collective agreement re that

--tag "$IMAGE_NAME:${DOCKER_IMAGES_VERSION}"
--file ${DOCKERFILE} .
- echo "$PARITYPR_PASS" |
Expand Down Expand Up @@ -164,3 +165,20 @@ build-push-image-substrate-pr:
variables:
DOCKERFILE: "docker/dockerfiles/substrate_injected.Dockerfile"
IMAGE_NAME: "docker.io/paritypr/substrate"

build-push-image-bridges-zombienet-tests:
stage: publish
extends:
- .kubernetes-env
- .common-refs
- .build-push-image
needs:
- job: build-linux-stable
artifacts: true
- job: build-linux-stable-cumulus
artifacts: true
- job: prepare-bridges-zombienet-artifacts
artifacts: true
variables:
DOCKERFILE: "docker/dockerfiles/bridges_zombienet_tests_injected.Dockerfile"
IMAGE_NAME: "docker.io/paritypr/bridges-zombienet-tests"
52 changes: 52 additions & 0 deletions docker/dockerfiles/bridges_zombienet_tests_injected.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# this image is built on top of existing Zombienet image
ARG ZOMBIENET_IMAGE
# this image uses substrate-relay image built elsewhere
ARG SUBSTRATE_RELAY_IMAGE=paritytech/substrate-relay:v2023-11-07-rococo-westend-initial-relayer

# metadata
ARG VCS_REF
ARG BUILD_DATE
ARG IMAGE_NAME

# we need `substrate-relay` binary, built elsewhere
FROM ${SUBSTRATE_RELAY_IMAGE} as relay-builder

# the base image is the zombienet image - we are planning to run zombienet tests using native
# provider here
FROM ${ZOMBIENET_IMAGE}

LABEL io.parity.image.authors="devops-team@parity.io" \
io.parity.image.vendor="Parity Technologies" \
io.parity.image.title="${IMAGE_NAME}" \
io.parity.image.description="Bridges Zombienet tests." \
io.parity.image.source="https://github.com/paritytech/polkadot-sdk/blob/${VCS_REF}/docker/dockerfiles/bridges_zombienet_tests_injected.Dockerfile" \
io.parity.image.revision="${VCS_REF}" \
io.parity.image.created="${BUILD_DATE}" \
io.parity.image.documentation="https://github.com/paritytech/polkadot-sdk/bridges/zombienet"

# show backtraces
ENV RUST_BACKTRACE 1
USER root

# add polkadot binary to the docker image
COPY ./artifacts/polkadot /usr/local/bin/
COPY ./artifacts/polkadot-execute-worker /usr/local/bin/
COPY ./artifacts/polkadot-prepare-worker /usr/local/bin/
# add polkadot-parachain binary to the docker image
COPY ./artifacts/polkadot-parachain /usr/local/bin
# copy substrate-relay to the docker image
COPY --from=relay /home/user/substrate-relay /usr/local/bin/
# finally - we need bridges zombienet runner and tests
RUN mkdir -p /home/nonroot/bridges
COPY ./artifacts/bridges /home/nonroot/bridges

# check if executable works in this container
USER nonroot
RUN /usr/local/bin/polkadot --version
RUN /usr/local/bin/polkadot-parachain --version
RUN /usr/local/bin/substrate-relay --version

# https://polkadot.js.org/apps/?rpc=ws://127.0.0.1:{PORT}#/explorer
EXPOSE 9942 9910 8943 9945 9010 8945

ENTRYPOINT ["/bin/bash", "-c", "/home/nonroot/bridges/zombienet/run-tests.sh"]