Shorten CI build times

[jpellizzari]

Builds on Circle take ~25 minutes, even for documentation changes. Some ideas from the Scope meeting:

• Only run integration tests when back-end stuff has changed
• Convert to yarn for FE dependency management: Upgrade to npm v4.0.5 or switch to using yarn #2090

[bboreham]

I did some analysis of build times:

Action	time
Start container	00:13
MACHINE
Restore cache	00:13
DEPENDENCIES
$ pip install --upgrade requests	00:28
This line takes a long time because CircleCI piggy-backs a Ruby install inside it. This in turn is because the .ruby-version file added for Netlify builds says 2.3.0 and CircleCI doesn't have that version pre-installed.
$ sudo apt-get update && sudo apt-get install jq pv && sudo chmod a+wr --recursive /usr/local/go/pkg && sudo chown ubuntu:ubuntu "$HOME/.bashrc.backup" (curl https://sdk.cloud.google.com | bash) && (bin/setup-circleci-secrets "$SECRET_PASSWORD") && make deps && mkdir -p $(dirname $SRCDIR) && cp -r $(pwd)/ $SRCDIR	00:33
We don't need to apt-get update; doesn't matter if we have a slightly out-of-date jq.
$ cd $SRCDIR/client; ../tools/rebuild-image weaveworks/scope-ui-build . Dockerfile package.json webpack.production.config.js .eslintrc .babelrc && touch $SRCDIR/.scope_ui_build.uptodate	00:42
This line is pulling the two build containers out of cache.
$ cd $SRCDIR/backend; ../tools/rebuild-image weaveworks/scope-backend-build . Dockerfile build.sh && touch $SRCDIR/.scope_backend_build.uptodate	00:27
$ sudo apt-get update && sudo apt-get install python-pip && sudo pip install awscli	00:10
Seems kinda dumb to install pip here when we used it several lines earlier
TEST
$ cd $SRCDIR; make RM= lint	01:41
$ cd $SRCDIR; COVERDIR=./coverage make RM= CODECGEN_UID=23 tests	03:37
Note unit tests are spread across the CircleCI lanes.
$ cd $SRCDIR; make RM= client-test static	03:31
Most of the time here is in make static running webpack (twice)
$ cd $SRCDIR; make RM= client-lint static	00:26
$ cd $SRCDIR; rm -f prog/scope; if [ "$CIRCLE_NODE_INDEX" = "0" ]; then GOARCH=arm make GO_BUILD_INSTALL_DEPS= RM= prog/scope; else GOOS=darwin make GO_BUILD_INSTALL_DEPS= RM= prog/scope; fi	00:48
$ cd $SRCDIR; rm -f prog/scope; make RM=	00:39
$ cd $SRCDIR/extras; ./build_on_circle.sh	00:59
$ (cd $SRCDIR/integration; ./gce.sh setup && eval $(./gce.sh hosts); ./setup.sh)	02:58
Setting up GCE hosts could be done in parallel with running tests; this has been set up for the Weave Net build
$ (cd $SRCDIR/integration; eval $(./gce.sh hosts); ./run_all.sh)	02:38
$ (cd $SRCDIR/integration; ./gce.sh destroy)	01:16
Could destroy in the background also; we don't care if it completes as there is a garbage-collector running elsewhere.
DEPLOYMENT
Collect artifacts	00:19

Excluding everything under 10 seconds (total 64s):

Action	time
Starting the build	00:06
Enable SSH	00:04
Restore source cache	00:04
Checkout using deploy key:	00:07
Configure the build	00:07
$ curl -sSL https://s3.amazonaws.com/circle-downloads/install-circleci-docker.sh | bash -s -- 1.10.0	00:02
Exporting env vars from circle.yml	00:02
Exporting env vars from project settings	00:02
$ (cd $SRCDIR/integration; ./gce.sh make_template)00:06
Save cache	00:04
$ (cd $SRCDIR; ./tools/cover/gather_coverage.sh ./coverage $SRCDIR/coverage)	00:09
$ (goveralls -repotoken $COVERALLS_REPO_TOKEN -coverprofile=$SRCDIR/profile.cov -service=circleci
$ (cd $SRCDIR; cp scope.tar $CIRCLE_ARTIFACTS)	00:02
$ ( docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS && docker tag weaveworks/scope:latest ${DOCKER_ORGANIZATION:-$DOCKER_USER}/scope:${CIRCLE_BRANCH////-} && docker push ${DOCKER_ORGANIZATION:-$DOCKER_USER}/scope:${CIRCLE_BRANCH////-} )	00:02
Collect test metadata	00:04

[bboreham]

Would it be OK to do less webpack? Maybe reserve the highest level for builds of master?

[bboreham]

The lint script executes each tool on each file individually; I think it would go a lot faster if they were grouped together.

i.e. instead of:

go tool vet prog/app.go
go tool vet prog/main.go
go tool vet prog/main_test.go
go tool vet prog/probe.go

do:

go tool vet prog/app.go prog/main.go prog/main_test.go prog/probe.go

[jpellizzari]

@bboreham We can try to make webpack run only once, but I think its best to have it run on all branches to ensure we catch issues that break the build. We also don't need to do a build in order to run tests, if that is what is happening.

[bboreham]

webpack runs twice because we have build and build-external.

I've separated the build from the tests as part of #2574.

[bboreham]

In another context we use yarn to keep a cache of webpack info in memory to speed up builds.

[jpellizzari]

@bboreham FYI yarn downloads/manages/caches external JS dependencies.