Switch self-hosted runners to ubicloud github runners #1877

saraswatpuneet · 2024-02-13T18:13:10Z

Goal

The goal of this PR is to switch the self hosted runners in github actions (except for benchmarks as they would run on self-hosted) to ubicloud-standard-4, code coverage is an exception as it requires higher verticals hence using ubicloud-standard-8 only for code coverage jobs

Release Workflow Test: https://github.com/LibertyDSNP/frequency/actions/runs/8900764362

Tests

Run benchmarks
Check e2e tests
Check rust tests

Checklist

.github/workflows/release.yml

codecov · 2024-02-15T16:30:58Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.13%. Comparing base (10c144c) to head (6d2915e).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1877   +/-   ##
=======================================
  Coverage   83.13%   83.13%           
=======================================
  Files          56       56           
  Lines        4530     4530           
=======================================
  Hits         3766     3766           
  Misses        764      764

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

.github/workflows/merge-pr.yml

github-actions · 2024-02-15T18:57:19Z

⏳ Running benchmarks and calculating weights. DO NOT MERGE! A new commit will be added upon completion...

github-actions · 2024-02-15T19:34:58Z

✅ Finished running benchmarks. Updated weights have been committed to this PR branch in commit bab3a68.

This reverts commit bab3a68.

saraswatpuneet · 2024-02-16T19:33:45Z

.cargo/config.toml

@@ -41,3 +41,6 @@ rustflags = [
 	"-Dclippy::unwrap_used",
 	"-Dclippy::expect_used",
 ]
+
+[net]
+git-fetch-with-cli = true


the random network issues on ubicloud warrants this check

https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

saraswatpuneet · 2024-02-21T16:18:50Z

.github/workflows/verify-pr-commit.yml

@@ -371,7 +371,8 @@ jobs:
          RUST_LOG=debug SRTOOL_TAG="1.70.0" srtool build \
            --build-opts="'--features on-chain-release-build,no-metadata-docs,${{matrix.features}}'" \
            --profile=${{matrix.build-profile}} \
-            --package=${{matrix.package}}
+            --package=${{matrix.package}} \
+            --root


needed for permissions

furkansahin · 2024-02-22T12:41:00Z

Hey @saraswatpuneet ,
I am a Ubicloud developer. It's exciting to see that you consider us for your runners.

I can see that you had a memory issue with our standard-8 runners. I would like to inform you that we found a misalignment in between our runners and github's in terms of swap space configurations. We have updated our runners to be in line with Github's as you can see here ubicloud/ubicloud#1244.

I can also see that you had a networking related issue. We found out that the memory related issues can manifest themselves as network drop as well. You may want to give a try to our latest runners and see if the issues persist :)

Is there any other specific problem I can help you and we can fix in ubicloud? Please just let me know if we can help anyway.
Best!

saraswatpuneet · 2024-02-26T15:36:19Z

@furkansahin we still get some intermittent issues (seems to be network but not sure), please check this job

https://github.com/LibertyDSNP/frequency/actions/runs/8051209010/job/21988632422

furkansahin · 2024-02-27T17:31:02Z

Hey @saraswatpuneet ,
I have cloned your repository and will debug this further. I should have more to share with you in the next 2 business days. A couple of questions that would help me to pinpoint the issues would be;

Did you have this kind of issue before?
I can see that you are switching from self hosted, is your self hosted instance located in a cloud provider? Also, what would be the location (US, europe?)

Best

saraswatpuneet · 2024-02-27T17:39:03Z

hey @furkansahin , thank you for taking a look, we havent faced this issue before, either on self hosted or github provider runner. this is new to us too. Following a re-run it passed, so its not reproducible everytime.

Regarding where we are hosting our runner, @wilwade can add details to it below.

wilwade · 2024-02-28T13:40:40Z

@furkansahin

It currently uses US based AWS runners. They are c7i.4xlarge and c7a.4xlarge.

furkansahin · 2024-02-29T15:01:29Z

Hey @saraswatpuneet ,
I can see what the problem is and reproduce it. However, since it fails randomly, it is difficult to catch the bug on the fly and reason about it. I have tried bunch of changes here in my fork https://github.com/furkansahin/frequency/actions/workflows/verify-pr-commit.yml. The problem seems like the srtool build is pulling bunch of repositories, the biggest being [https://github.com/paritytech/polkadot-sdk](https://github.com/paritytech/polkadot-sdk). During fetch, the connection is dropped, sometimes. This is not expected and I believe it has something to do with the location of our runners being in Europe.
In your self hosted runners, do you use caching? I believe the amount of pull in your self hosted runner may not be as high as it is in ubicloud, is that assumption correct?
I can see that you have added git-fetch-with-cli = true to overcome the networking issue but it did not help. Is that correct? Or, do you think it actually solved something but this issue is separate?
I am at the Devworld Conference in Amsterdam at the moment, so, unfortunately this is all I could find so far.
Best,
Furkan

furkansahin · 2024-03-04T15:11:39Z

Hey @saraswatpuneet , @wilwade
I have looked into this more and here is my analysis and a solution proposal for you.
Analysis: Looking into the networking logs in detail (tcpdump + wireshark), it is correct that when cargo fetches the repositories, there are some packet drops in between. While some packet drops are totally acceptable and harmless, in this instance, they cause random connection disruption. This mostly happens when the global traffic is high (US daytime). You would not see that in your self hosted instances or regular github runners because the repositories sit closer to those. In Ubicloud's scenario, our runners are provisioned either in Germany of Finland.
We are continuously improving our networking stack and I will look further into this problem how we can setup our networking in a way these issues would not happen. In the meantime, if you would give us a try, I found a way that makes the Verify Build Runtime jobs to pass.

Solution proposal: Unfortunately, this is a band aid proposal. I realised git-fetch-with-cli = true is not really necessary, but instead, you can use;

[net]
retry = 5

This way, cargo will retry if the fetch fails up to 5 times. In fact, in my last 27 Verify Build Runtime for X jobs, it had to try 3 times for only 1 of the jobs. You can see the latest one here https://github.com/furkansahin/frequency/actions/runs/8140985585.

I understand the inconvenience this issue may cause and would completely understand any decisions you make regarding this PR. However, I really hope you will consider giving us the opportunity to demonstrate our commitment to the overall customer experience.

Best,
Furkan

This reverts commit 0b3ce58.

wilwade · 2024-05-01T12:23:13Z

.cargo/config.toml

@@ -41,3 +41,6 @@ rustflags = [
 	"-Dclippy::unwrap_used",
 	"-Dclippy::expect_used",
 ]
+
+[net]
+retry = 5


Needed due to the location of the ubicloud runners

wilwade · 2024-05-01T12:23:21Z

.github/workflows/common/reset-self-hosted-runner/action.yml

wilwade · 2024-05-01T12:37:25Z

.github/workflows/release.yml

+permissions:
+  contents: write


This allows us to get rid of the special GHA_GIT_COMMIT token

nit: I think behind the scenes it generates a token with those permissions, unless we want to control the lifetime of a token

This is correct. There is an option to control it such that the permission is only on specific steps instead the entire workflow, but I figured this is the best 1:1 swap for now.

wilwade · 2024-05-01T12:39:11Z

.github/workflows/merge-pr.yml

@@ -12,7 +12,6 @@ jobs:
  changes:
    name: Determine Changed Files
    runs-on: ubuntu-22.04
-    container: ghcr.io/libertydsnp/frequency/ci-base-image:latest


You'll see a lot of places where I removed the use of the ci-base-image. It really is only needed for things that use rust.

wilwade · 2024-05-01T12:39:41Z

.github/workflows/rococo.yml

@@ -17,7 +17,7 @@ env:
 jobs:
  run-e2e:
    name: Run E2E Tests
-    runs-on: [self-hosted, Linux, X64, build, v2]
+    runs-on: ubuntu-22.04


Some things didn't need to be on ubicloud as they don't need the speed.

wilwade · 2024-05-01T12:40:18Z

pallets/schemas/src/tests/other_tests.rs

@@ -490,7 +490,7 @@ fn create_schema_v3_requires_valid_schema_size() {
 				expected: (Error::<Test>::LessThanMinSchemaModelBytes, 3),
 			},
 			TestCase {
-				input: r#"{"id": "long", "title": "I am a very very very long schema", "properties": "just way too long to live a long life", "description": "Just a never ending stream of bytes that goes on for a minute too long"}"#,
+				input: r#"{"id": "long", "title": "I am a very very very very long schema", "properties": "just way too long to live a long life", "description": "Just a never ending stream of bytes that goes on for a minute too long"}"#,


Triggering rust code changes

wilwade · 2024-05-01T13:49:32Z

.github/workflows/verify-pr-commit.yml

      - name: Install Subwasm
        run: |
-          cargo install --locked --git https://github.com/chevdor/subwasm --tag v0.19.1 --force
+          cargo install --locked --git https://github.com/chevdor/subwasm --tag v0.20.0 --force


Highlighting version bump in subwasm

wilwade · 2024-05-01T13:50:01Z

.github/workflows/verify-pr-commit.yml

+          package: ${{matrix.package}}
+          chain: ${{matrix.chain}}
+          runtime_dir: ${{ matrix.runtime-dir }}
+          tag: "1.77.0"


Highlighting version bump in srtool

This reverts commit 1c5a338.

mattheworris

Read through code changes
Read through comment thread

🚢 it!

JoeCap08055

Read through changes and comments... if we're satisfied with the solutions from ubicloud for the network drops and memory issues,

saraswatpuneet

@wilwade thank you for taking this ahead, great work and looks good to go
🚢

wilwade reviewed Feb 14, 2024

View reviewed changes

.github/workflows/release.yml Outdated Show resolved Hide resolved

saraswatpuneet changed the title ~~runs-on: ubicloud~~ Switch self-hosted runners to ubicloud github runners Feb 14, 2024

saraswatpuneet marked this pull request as ready for review February 15, 2024 17:56

saraswatpuneet requested a review from demisx as a code owner February 15, 2024 17:56

saraswatpuneet commented Feb 15, 2024

View reviewed changes

.github/workflows/merge-pr.yml Show resolved Hide resolved

demisx pushed a commit that referenced this pull request Feb 15, 2024

Update weights for PR #1877

bab3a68

github-actions bot added metadata-changed Metadata has changed since the latest full release metadata-version-not-incremented Metadata has changed since the latest full release, but the version has not been incremented labels Feb 15, 2024

saraswatpuneet added a commit that referenced this pull request Feb 15, 2024

Revert "Update weights for PR #1877"

37f02ad

This reverts commit bab3a68.

github-actions bot removed metadata-changed Metadata has changed since the latest full release metadata-version-not-incremented Metadata has changed since the latest full release, but the version has not been incremented labels Feb 15, 2024

saraswatpuneet commented Feb 16, 2024

View reviewed changes

saraswatpuneet commented Feb 21, 2024

View reviewed changes

saraswatpuneet marked this pull request as draft February 26, 2024 16:19

saraswatpuneet added 6 commits April 23, 2024 15:09

runs-on: ubicloud

be8050c

runs-on: ubicloud-standard-2

5d017dc

address feedback and fix code cov action

8d751fb

ubicloud-standard-4

0ad82f8

try root user

2f0f723

try use permission on/build

a8c63ca

wilwade added 3 commits April 30, 2024 08:21

Setting GHA permissions to write to create release and release branch

0a11459

Remove custom base image where not needed

8a746a5

Add arch back in

1dd8296

wilwade temporarily deployed to github-pages April 30, 2024 20:29 — with GitHub Actions Inactive

Revert "Testing workflow"

d2916bc

This reverts commit 0b3ce58.

wilwade marked this pull request as ready for review April 30, 2024 22:23

wilwade reviewed May 1, 2024

View reviewed changes

.github/workflows/common/reset-self-hosted-runner/action.yml Outdated

Copy link

Collaborator

wilwade May 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused

wilwade added 3 commits May 1, 2024 08:25

Remove unneeded comment

0f8af34

Remove unneeded ci image use

16e79cc

Switch around CI power levels

1c5a338

wilwade reviewed May 1, 2024

View reviewed changes

16 is the max

93dd3ba

wilwade reviewed May 1, 2024

View reviewed changes

Revert "Switch around CI power levels"

6d2915e

This reverts commit 1c5a338.

wilwade requested review from mattheworris, aramikm, JoeCap08055, shannonwells and enddynayn May 1, 2024 13:54

mattheworris approved these changes May 1, 2024

View reviewed changes

JoeCap08055 approved these changes May 1, 2024

View reviewed changes

saraswatpuneet commented May 1, 2024

View reviewed changes

wilwade approved these changes May 1, 2024

View reviewed changes

wilwade merged commit d984c8b into main May 1, 2024
31 checks passed

wilwade deleted the ubicloud branch May 1, 2024 16:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch self-hosted runners to ubicloud github runners #1877

Switch self-hosted runners to ubicloud github runners #1877

saraswatpuneet commented Feb 13, 2024 •

edited by wilwade

Loading

codecov bot commented Feb 15, 2024 •

edited

Loading

github-actions bot commented Feb 15, 2024

github-actions bot commented Feb 15, 2024

saraswatpuneet Feb 16, 2024

saraswatpuneet Feb 21, 2024

furkansahin commented Feb 22, 2024

saraswatpuneet commented Feb 26, 2024 •

edited

Loading

furkansahin commented Feb 27, 2024

saraswatpuneet commented Feb 27, 2024 •

edited

Loading

wilwade commented Feb 28, 2024

furkansahin commented Feb 29, 2024

furkansahin commented Mar 4, 2024

wilwade May 1, 2024

wilwade May 1, 2024

wilwade May 1, 2024

saraswatpuneet May 1, 2024

wilwade May 1, 2024

wilwade May 1, 2024

wilwade May 1, 2024

wilwade May 1, 2024

wilwade May 1, 2024

wilwade May 1, 2024

mattheworris left a comment

JoeCap08055 left a comment

saraswatpuneet left a comment •

edited

Loading

Switch self-hosted runners to ubicloud github runners #1877

Switch self-hosted runners to ubicloud github runners #1877

Conversation

saraswatpuneet commented Feb 13, 2024 • edited by wilwade Loading

Goal

Checklist

codecov bot commented Feb 15, 2024 • edited Loading

Codecov Report

github-actions bot commented Feb 15, 2024

github-actions bot commented Feb 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

furkansahin commented Feb 22, 2024

saraswatpuneet commented Feb 26, 2024 • edited Loading

furkansahin commented Feb 27, 2024

saraswatpuneet commented Feb 27, 2024 • edited Loading

wilwade commented Feb 28, 2024

furkansahin commented Feb 29, 2024

furkansahin commented Mar 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattheworris left a comment

Choose a reason for hiding this comment

JoeCap08055 left a comment

Choose a reason for hiding this comment

saraswatpuneet left a comment • edited Loading

Choose a reason for hiding this comment

saraswatpuneet commented Feb 13, 2024 •

edited by wilwade

Loading

codecov bot commented Feb 15, 2024 •

edited

Loading

saraswatpuneet commented Feb 26, 2024 •

edited

Loading

saraswatpuneet commented Feb 27, 2024 •

edited

Loading

saraswatpuneet left a comment •

edited

Loading