Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache CI downloads #599

Closed
gibfahn opened this issue Jan 25, 2017 · 8 comments
Closed

Cache CI downloads #599

gibfahn opened this issue Jan 25, 2017 · 8 comments

Comments

@gibfahn
Copy link
Member

gibfahn commented Jan 25, 2017

There has been some discussion of setting up a cache on one of the CI machines to store things we download all the time, I couldn't find an existing issue covering it.

There are a couple of things that I know about:

  1. We need a place to put packages so we don't have to manually download them every time we setup a machine (definitely useful for AIX, not sure about other platforms).

  2. deps: upgrade zlib to 1.2.11 node#10980 (comment) (by @bnoordhuis)

The fips buildbot failed to download https://openssl.org/source/openssl-fips-2.0.9.tar.gz.

@nodejs/build Can we cache it somewhere? It's a point of failure and not very good netizenship to download it with every CI run.

@jbergstroem
Copy link
Member

@gibfahn you mean like setting up a squid/proxy/similar? In order for it to be effective I'm thinking we would want to do that per-continent. The arm setup at @rvagg's seems like a good first setup; second could be at DO us east?

@gibfahn
Copy link
Member Author

gibfahn commented Jan 25, 2017

cc/ @mhdawson

@bnoordhuis
Copy link
Member

@jbergstroem In the case of openssl-fips-2.0.9.tar.gz, it could be hosted on local disk or on the local network. That tarball was released 2 years ago and hasn't been the current one for 1.5 years.

@jbergstroem
Copy link
Member

@bnoordhuis ok; similar to how we treat ICU I guess.

@mhdawson
Copy link
Member

The prior discusion/agreement was that we could host them on the main ci machine, we just never made progress on doing that.

@bnoordhuis
Copy link
Member

Backref to nodejs/node#13219 which was about the fips tarball download failing.

@joaocgreis
Copy link
Member

While there's no better solution, this is what the ChakraCore Linux jobs use now:

Bash function
function cache_download {
  if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
    echo "Missing argument"
    return 1
  fi
  URL=$1
  EXPECTED_SHA=$2
  DEST=$3

  mkdir -p "$HOME/cache_files"
  CACHE_FILE="$HOME/cache_files/$EXPECTED_SHA"

  CACHE_SHA="X"
  if [ -f "$CACHE_FILE" ]; then
    CACHE_SHA=`shasum -a 256 "$CACHE_FILE" | cut '-d ' -f1`
  fi

  if [ "X${CACHE_SHA}" != "X${EXPECTED_SHA}" ]; then
    rm "$CACHE_FILE" || true
    curl -L "$URL" -o "$CACHE_FILE"
    DOWNLOADED_SHA=`shasum -a 256 "$CACHE_FILE" | cut '-d ' -f1`
    if [ "X${DOWNLOADED_SHA}" != "X${EXPECTED_SHA}" ]; then
      echo Downloaded SHA256 mismatch.
      rm -v "$CACHE_FILE"
      return 1
    fi
  fi

  cp -fv "$CACHE_FILE" "$DEST"
}

@Trott
Copy link
Member

Trott commented Mar 22, 2019

Closing due to long period of inactivity. Feel free to re-open if this is a thing. I'm just trying to close stuff that has been ignored for sufficiently long that it seems likely it's not something we're going to get to.

@Trott Trott closed this as completed Mar 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants