ci: workaround for the cache bug on Linux #11568

ordian · 2020-03-13T21:41:51Z

bjorn3 · 2020-03-13T22:09:25Z

.github/workflows/build-test.yml

@@ -23,6 +23,10 @@ jobs:
        uses:                   actions/checkout@v2
        with:
          fetch-depth:          50
+      # https://github.com/actions/cache/issues/133
+      - name:                   Fixup the owner of ~/.cargo/
+        if:                     matrix.platform == 'ubuntu-16.04'


You will probably also want to do this on macOS.

It doesn't seem to be a problem for macOS.
Github uses a third-party service for macOS runners: https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#cloud-hosts-for-github-hosted-runners as opposed to Standard_DS2_v2 VMs on Asure for Linux and Windows.

Example job on macOS: https://github.com/openethereum/openethereum/runs/506808782.

bjorn3 · 2020-03-14T16:58:28Z

Cache cargo registry still has the permission problem, while Cache cargo index does work fine. I noticed that I have the same problem with my project. registry and ~/.cargo/bin don't work while index and target dir work fine: https://github.com/bjorn3/rustc_codegen_cranelift/runs/507914716

ordian · 2020-03-14T17:04:00Z

Cache cargo registry still has the permission problem, while Cache cargo index does work fine. I noticed that I have the same problem with my project. registry and ~/.cargo/bin don't work while index and target dir work fine: https://github.com/bjorn3/rustc_codegen_cranelift/runs/507914716

Wasn't that the case already, it means that this fix doesn't actually work, sadly.

bjorn3 · 2020-03-14T17:14:40Z

lrwxrwxrwx 1 runner docker 22 Mar 13 15:33 /home/runner/.cargo -> /usr/share/rust/.cargo

~/.cargo is a symlink for /usr/share/rust/.cargo, so the chown only changed the owner of the symlink. Appending a / to the path (sudo chown -R $(whoami):$(id -ng) ~/.cargo/) does fix it without requiring a cache invalidation: https://github.com/bjorn3/rustc_codegen_cranelift/runs/508021777?check_suite_focus=true

ordian · 2020-03-14T17:18:41Z

@bjorn3 wow thank you for looking into this ❤️

bjorn3 · 2020-03-16T11:32:17Z

.github/workflows/build-test.yml

        shell:                  bash
-        run:                    git submodule update --init --recursive
+        run:                    bash sudo chown -R $(whoami):$(id -ng) ~/.cargo/


This will try to run sudo as shell script I believe. Also windows doesn't have chown, whoami and id. The permission system on Windows works quite different.

I expected running bash on Windows would use the Windows Subsystem for Linux 2, but maybe I'm wrong.

bash on windows is just a recompilation of bash for windows I believe. You have to use wsl to run commands on Windows Subsystem for Linux. I assume it is not enabled on gh actions though.

dvdplm · 2020-03-16T15:11:31Z

Cargo.lock

@@ -3556,9 +3556,9 @@ dependencies = [

 [[package]]
 name = "parity-util-mem"
-version = "0.5.2"
+version = "0.5.1"


Why the downgrade?

It's not a downgrade compared to master, just reverted unrelated to this PR change (see REVERTME commit).

ordian · 2020-03-16T17:04:08Z

.github/workflows/build-test.yml

+          $acl = Get-Acl "C:\Users\runneradmin\.cargo\"
+          $accessRule = New-Object System.Security.AccessControl.FileSystemAccessRule("everyone","FullControl","Allow")
+          $acl.SetAccessRule($accessRule)
+          $acl | Set-Acl "C:\Users\runneradmin\.cargo\"      


#11572 was filed for Windows, but it's fine to do here as well

bjorn3 · 2020-03-16T17:08:04Z

.github/workflows/build-test.yml

+        if:                     matrix.platform == 'windows-latest'
+        shell:                  powershell
+        run:                    |
+          $registry-path = "C:\Users\runneradmin\.cargo\registry"


Unexpected token '-path' in expression or statement.

Suggested change

$registry-path = "C:\Users\runneradmin\.cargo\registry"

$registry_path = "C:\Users\runneradmin\.cargo\registry"

I assume _ is allowed.

bjorn3 · 2020-03-16T17:21:01Z

./index/git.luolix.top-1ecc6299db9ec823/.git/objects/pack/pack-d3f48734b45eac2d457e12f98a6936cf31500e2b.idx: Can't unlink already-existing object
./index/git.luolix.top-1ecc6299db9ec823/.git/objects/pack/pack-d3f48734b45eac2d457e12f98a6936cf31500e2b.pack: Can't unlink already-existing object

https://github.com/actions/virtual-environments/blob/7584c7b8793620167c46cd587f35c46520b5f26e/images/win/scripts/Installers/Install-Rust.ps1#L10

bjorn3 · 2020-03-16T17:36:11Z

.github/workflows/build-test.yml

+        if:                     matrix.platform == 'windows-latest'
+        shell:                  powershell
+        run:                    |
+          $registryPath = "C:\Rust\.cargo\registry"


Maybe use

Suggested change

$registryPath = "C:\Rust\.cargo\registry"

$registryPath = "$CARGO_HOME\registry"

to be more resillient against changes to the path? (Not sure if it is correct powershell syntax)

ordian · 2020-03-16T17:42:34Z

I'm not even sure this is a permission issue on Windows, and the current attempt doesn't seem to fix the problem: https://github.com/openethereum/openethereum/runs/511742121#step:6:7.

So let's try to tackle it in a separate PR?

.github/workflows/build-test.yml

ordian added 2 commits March 13, 2020 22:41

ci: workaround for the cache bug

1ad6248

ci: use the workaround only for linux

769399e

ordian force-pushed the ao-github-actions-cache-workaround branch from 92edff4 to 769399e Compare March 13, 2020 21:53

cleanup

aab4641

ordian added the A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. label Mar 13, 2020

REVERTME: invalidate cache

0079bd1

bjorn3 reviewed Mar 13, 2020

View reviewed changes

test I

21c72e2

ordian added 4 commits March 14, 2020 18:19

test II

9c6383f

simplify checkout

dcb3199

try bash on windows

aa9b3fa

fix checkout

4032f69

bjorn3 reviewed Mar 16, 2020

View reviewed changes

ordian added 2 commits March 16, 2020 12:33

sudo bash?

ec2302c

ok, I don't have time for this sh*t

98c3d3c

ordian changed the title ~~ci: workaround for the cache bug~~ ci: workaround for the cache bug on Linux Mar 16, 2020

revert Cargo.lock change

e530855

ordian added A0-pleasereview 🤓 Pull request needs code review. M1-ci 🙉 Continuous integration. and removed A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. labels Mar 16, 2020

ordian requested review from General-Beck and denisgranha March 16, 2020 12:13

ordian mentioned this pull request Mar 16, 2020

CI: find a workaround for the caching issue on Windows #11572

Open

dvdplm reviewed Mar 16, 2020

View reviewed changes

General-Beck added 2 commits March 16, 2020 19:50

Test rules for .cargo in windows-latest

61d78bc

update rules

f1c6086

ordian commented Mar 16, 2020

View reviewed changes

bjorn3 reviewed Mar 16, 2020

View reviewed changes

Get Acl

f88f15b

new path to windows cargo home

09f01cb

https://github.com/actions/virtual-environments/blob/7584c7b8793620167c46cd587f35c46520b5f26e/images/win/scripts/Installers/Install-Rust.ps1#L10

bjorn3 reviewed Mar 16, 2020

View reviewed changes

Set-ExecutionPolicy

697a3f0

bjorn3 mentioned this pull request Mar 16, 2020

Fix cargo registry caching for builds rust-lang/rust-analyzer#3613

Merged

Test

5039503

General-Beck approved these changes Mar 16, 2020

View reviewed changes

denisgranha approved these changes Mar 17, 2020

View reviewed changes

.github/workflows/build-test.yml Show resolved Hide resolved

.github/workflows/build-test.yml Show resolved Hide resolved

ordian added 3 commits March 20, 2020 21:31

another experiment

7b94612

another experiment II

4d0589d

revert temp changes

3317d6f

ordian merged commit 78916be into master Mar 21, 2020

ordian deleted the ao-github-actions-cache-workaround branch March 21, 2020 14:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: workaround for the cache bug on Linux #11568

ci: workaround for the cache bug on Linux #11568

ordian commented Mar 13, 2020

bjorn3 Mar 13, 2020

ordian Mar 13, 2020

ordian Mar 13, 2020

bjorn3 commented Mar 14, 2020

ordian commented Mar 14, 2020

bjorn3 commented Mar 14, 2020 •

edited

Loading

ordian commented Mar 14, 2020

bjorn3 Mar 16, 2020

ordian Mar 16, 2020

bjorn3 Mar 16, 2020

dvdplm Mar 16, 2020

ordian Mar 16, 2020

ordian Mar 16, 2020

bjorn3 Mar 16, 2020

bjorn3 commented Mar 16, 2020

bjorn3 Mar 16, 2020

ordian commented Mar 16, 2020

	$registry-path = "C:\Users\runneradmin\.cargo\registry"
	$registry_path = "C:\Users\runneradmin\.cargo\registry"

	$registryPath = "C:\Rust\.cargo\registry"
	$registryPath = "$CARGO_HOME\registry"

ci: workaround for the cache bug on Linux #11568

ci: workaround for the cache bug on Linux #11568

Conversation

ordian commented Mar 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bjorn3 commented Mar 14, 2020

ordian commented Mar 14, 2020

bjorn3 commented Mar 14, 2020 • edited Loading

ordian commented Mar 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bjorn3 commented Mar 16, 2020

Choose a reason for hiding this comment

ordian commented Mar 16, 2020

bjorn3 commented Mar 14, 2020 •

edited

Loading