Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow updating an existing cache #505

Open
davidsbond opened this issue Jun 25, 2020 · 19 comments
Open

Allow updating an existing cache #505

davidsbond opened this issue Jun 25, 2020 · 19 comments
Labels
cache enhancement New feature or request

Comments

@davidsbond
Copy link

davidsbond commented Jun 25, 2020

GitHub actions supports caching across builds which is super powerful. However, once a cache has been saved it can no longer be modified. While this is fine for most scenarios where you need the cache there are a few times when you would like to update it.

I created this PR on the actual cache action to allow updating a cache when providing it in configuration:

actions/cache#353

I created it after reading the related issue:

actions/cache#342

However, it seems like something either needs to change in this package or server-side on the actions cache.

My use-case for wanting to update a cache is when using GitHub actions in golang projects. Go supports a test cache which allows tests to be skipped if the code they use has not changed, which can significantly improve test times (especially in large projects). Each time I run my tests, I'd like to maintain a single cache for tests, that way, when a run completes I can update the cache and all my workflows share a single test cache

There are some extra use-cases described on the issue above.

@davidsbond davidsbond added the enhancement New feature or request label Jun 25, 2020
@smorimoto
Copy link
Contributor

I would also like to support this proposal.

@half2me
Copy link

half2me commented Sep 4, 2021

I need this too

@eLarocque
Copy link

same

@CAMOBAP
Copy link

CAMOBAP commented Oct 27, 2022

@Blissful89
Copy link

#poke

@hyperxpro
Copy link

+1

3 similar comments
@stephanrotolante
Copy link

+1

@DenizUgur
Copy link

+1

@emrebayramc
Copy link

+1

@YegorMedvedev
Copy link

In the meantime, here's a workaround that I use:

- name: Get cache
  uses: actions/cache@v3
  with:
    path: ... # a list of cache paths
    key: ${{ runner.os }}-my-cache-${{ github.head_ref }}-${{ hashFiles(...for example yarn.lock...) }}

- name: Do something to change a cache
  run: ...

- name: Enforce cache update for a local NX cache
  run: |
    STATE_CACHE_KEY="${{ runner.os }}-my-cache-${{ github.head_ref }}-${{ hashFiles(...) }}"
    echo "STATE_CACHE_KEY=${STATE_CACHE_KEY}" >> $GITHUB_ENV

@emrebayramc
Copy link

+1

4 similar comments
@wind57
Copy link

wind57 commented Jun 9, 2023

+1

@arpanrec
Copy link

arpanrec commented Jul 8, 2023

+1

@bexsoft
Copy link

bexsoft commented Jul 12, 2023

+1

@nnsay
Copy link

nnsay commented Jul 13, 2023

+1

@dsame
Copy link

dsame commented Jul 25, 2023

Updating the cache is supposed to be done with restore-keys

This input allows to restore the cache even if it is not fully matched. And to generate the new key to update the cache do one of:

  1. include the changing file into ${{ hashFiles(...) }}
  2. set output of the previous step and include it into the key

@isaacbowen
Copy link

@dsame This is exactly what I was missing. Thank you for sharing! Super super super helpful. ❤️

@pauldraper
Copy link

@dsame's workaround is the best available currently, though it does have problems.

Github hard-limits repos to 10GB of caches. So creating lots of caches can push other less-frequent-but-helpful caches to deletion. Whereas if you could update/replace an existing cache, that wouldn't happen.

wongsyrone added a commit to wongsyrone/wongsyrone-scale-build-ntfs3 that referenced this issue Sep 27, 2023
CI: split pkg build and update w/ iso

ntfs3 patches from paragon group (unupstream)

openseachest: use upstream pull script

install pkgs

single-branch

add artifacts

ignore error for make all step to get log artifacts

china timezone

linux: merge linux 6.1

update grub2 to latest

remove grub2 single job

 * this is leftover of previous bash-based build script

add iso and update chksum files

use latest pip packages

 * pin jsonschema due to schema exception

CI: use git from Ubuntu PPA

git: depth 1

prepend sudo to cmdline

show info after build

CI: 8G root; 10G swap + 7G mem = 17G

update smartmontools to 7.3

CI: reclaim disk space

truenas/zfs@165d115

CI: do not run in the background

CI: setup-python github action uses this dir

CI: only use setup-python for scale-build

do NOT set PYTHON env

CI: add comments and echoes

disable packages with '-dbg' by removing them from manifest

scst: back to official after cleanup dbg pkgs

copy manifest from update dir

use python 3.11

do NOT show kernel config changes

CI: exclude ccache dir

CI: use sudo for commands on GH

CI: exclude scale-build repo hash from cache key calculation
 also takes conf manifest into consideration

CI: set permission for tmp

CI: cache rootfs and iso chroot

output flag on whether rootfs and iso chroot cache is usable to /tmp

add subpkg to build grub-efi-amd64-signed
when we are using customized grub

CI: always upload cache when success build

CI: use unversioned cache

delete old one if we can upload new(update existing cache is not
possible) actions/toolkit#505
wongsyrone added a commit to wongsyrone/wongsyrone-scale-build-ntfs3 that referenced this issue Sep 28, 2023
CI: split pkg build and update w/ iso

ntfs3 patches from paragon group (unupstream)

openseachest: use upstream pull script

install pkgs

single-branch

add artifacts

ignore error for make all step to get log artifacts

china timezone

linux: merge linux 6.1

update grub2 to latest

remove grub2 single job

 * this is leftover of previous bash-based build script

add iso and update chksum files

use latest pip packages

 * pin jsonschema due to schema exception

CI: use git from Ubuntu PPA

git: depth 1

prepend sudo to cmdline

show info after build

CI: 8G root; 10G swap + 7G mem = 17G

update smartmontools to 7.3

CI: reclaim disk space

truenas/zfs@165d115

CI: do not run in the background

CI: setup-python github action uses this dir

CI: only use setup-python for scale-build

do NOT set PYTHON env

CI: add comments and echoes

disable packages with '-dbg' by removing them from manifest

scst: back to official after cleanup dbg pkgs

copy manifest from update dir

use python 3.11

do NOT show kernel config changes

CI: exclude ccache dir

CI: use sudo for commands on GH

CI: exclude scale-build repo hash from cache key calculation
 also takes conf manifest into consideration

CI: set permission for tmp

CI: cache rootfs and iso chroot

output flag on whether rootfs and iso chroot cache is usable to /tmp

add subpkg to build grub-efi-amd64-signed
when we are using customized grub

CI: always upload cache when success build

CI: use unversioned cache

delete old one if we can upload new(update existing cache is not
possible) actions/toolkit#505
wongsyrone added a commit to wongsyrone/wongsyrone-scale-build-ntfs3 that referenced this issue Sep 28, 2023
CI: split pkg build and update w/ iso

ntfs3 patches from paragon group (unupstream)

openseachest: use upstream pull script

install pkgs

single-branch

add artifacts

ignore error for make all step to get log artifacts

china timezone

linux: merge linux 6.1

update grub2 to latest

remove grub2 single job

 * this is leftover of previous bash-based build script

add iso and update chksum files

use latest pip packages

 * pin jsonschema due to schema exception

CI: use git from Ubuntu PPA

git: depth 1

prepend sudo to cmdline

show info after build

CI: 8G root; 10G swap + 7G mem = 17G

update smartmontools to 7.3

CI: reclaim disk space

truenas/zfs@165d115

CI: do not run in the background

CI: setup-python github action uses this dir

CI: only use setup-python for scale-build

do NOT set PYTHON env

CI: add comments and echoes

disable packages with '-dbg' by removing them from manifest

scst: back to official after cleanup dbg pkgs

copy manifest from update dir

use python 3.11

do NOT show kernel config changes

CI: exclude ccache dir

CI: use sudo for commands on GH

CI: exclude scale-build repo hash from cache key calculation
 also takes conf manifest into consideration

CI: set permission for tmp

CI: cache rootfs and iso chroot

output flag on whether rootfs and iso chroot cache is usable to /tmp

add subpkg to build grub-efi-amd64-signed
when we are using customized grub

CI: always upload cache when success build

CI: use unversioned cache

delete old one if we can upload new(update existing cache is not
possible) actions/toolkit#505
@lcswillems
Copy link

lcswillems commented Jun 22, 2024

On Gitlab or a lot of other different CI/CD tools, this is built-in 😭 Such a pain not to have this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cache enhancement New feature or request
Projects
None yet
Development

No branches or pull requests