Skip to content

Commit

Permalink
go-get-kubernetes.sh: automate Kubernetes dependency handling
Browse files Browse the repository at this point in the history
This script handles the necessary "replace" statements and determines
which packages need to be updated in lockstep.
  • Loading branch information
pohly committed Oct 7, 2019
1 parent 194289a commit c1078a6
Show file tree
Hide file tree
Showing 2 changed files with 124 additions and 0 deletions.
20 changes: 20 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,23 @@ verify that it's content is up-to-date.
The `vendor` directory is optional. It is still present in projects
because it avoids downloading sources during CI builds. If this is no
longer deemed necessary, then a project can also remove the directory.

When using packages that are part of the Kubernetes source code, the
commands above are not enough because the [lack of semantic
versioning](https://github.com/kubernetes/kubernetes/issues/72638)
prevents `go mod` from finding newer releases. Importing directly from
`kubernetes/kubernetes` also needs `replace` statements to override
the fake `v0.0.0` versions
(https://github.com/kubernetes/kubernetes/issues/79384). The
`go-get-kubernetes.sh` script can be used to update all packages in
lockstep to a different Kubernetes version. It takes a single version
number like "1.16.0".

Conversion of a repository that uses `dep` to `go mod` can be done with:

GO111MODULE=on go mod init
release-tools/go-get-kubernetes.sh <current Kubernetes version from Gopkg.toml>
GO111MODULE=on go mod tidy
GO111MODULE=on go mod vendor
git rm -f Gopkg.toml Gopkg.lock
git add go.mod go.sum vendor
104 changes: 104 additions & 0 deletions go-get-kubernetes.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
#!/usr/bin/env bash

# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# This script can be used while converting a repo from "dep" to "go mod"
# by calling it after "go mod init" or to update the Kubernetes packages
# in a repo that has already been converted. Only packages that are
# part of kubernetes/kubernetes and thus part of a Kubernetes release
# are modified. Other k8.io packages (like k8s.io/klog, k8s.io/utils)
# need to be updated separately.

set -o pipefail

cmd=$0

function help () {
echo "$cmd <kubernetes version = x.y.z> - update all components from kubernetes/kubernetes to that version"
}

if [ $# -ne 1 ]; then
help
exit 1
fi
case "$1" in -h|--help|help) help; exit 0;; esac

die () {
echo >&2 "$@"
exit 1
}

k8s="$1"

# If the repo imports k8s.io/kubernetes (directly or indirectly), then
# "go mod" will try to find "v0.0.0" versions because
# k8s.io/kubernetes has those in it's go.mod file
# (https://github.com/kubernetes/kubernetes/blob/2bd9643cee5b3b3a5ecbd3af49d09018f0773c77/go.mod#L146-L157).
# (https://github.com/kubernetes/kubernetes/issues/79384).
#
# We need to replicate the replace statements to override those fake
# versions also in our go.mod file (idea and some code from
# https://github.com/kubernetes/kubernetes/issues/79384#issuecomment-521493597).
mods=$( (set -x; curl --silent --show-error --fail "https://raw.githubusercontent.com/kubernetes/kubernetes/v${k8s}/go.mod") |
sed -n 's|.*k8s.io/\(.*\) => ./staging/src/k8s.io/.*|k8s.io/\1|p'
) || die "failed to determine Kubernetes staging modules"
for mod in $mods; do
# The presence of a potentially incomplete go.mod file affects this command,
# so move elsewhere.
modinfo=$(set -x; cd /; env GO111MODULE=on go mod download -json "$mod@kubernetes-${k8s}") ||
die "failed to determine version of $mod: $modinfo"
v=$(echo "$modinfo" | sed -n 's|.*"Version": "\(.*\)".*|\1|p')
(set -x; env GO111MODULE=on go mod edit "-replace=$mod=$mod@$v") || die "'go mod edit' failed"
done

packages=

# Beware that we have to work with packages, not modules (i.e. no -m
# flag), because some modules trigger a "no Go code except tests"
# error. Getting their packages works.
if ! packages=$( (set -x; env GO111MODULE=on go list all) | grep ^k8s.io/ | sed -e 's; *;;'); then
cat >&2 <<EOF
Warning: "GO111MODULE=on go list all" failed, trying individual packages instead.
EOF
if ! packages=$( (set -x; env GO111MODULE=on go list -f '{{ join .Deps "\n" }}' ./...) | grep ^k8s.io/); then
cat >&2 <<EOF
ERROR: could not obtain package list, both of these commands failed:
GO111MODULE=on go list all
GO111MODULE=on go list -f '{{ join .Deps "\n" }}' ./pkg/...
EOF
exit 1
fi
fi

deps=
for package in $packages; do
# Some k8s.io packages do not come from Kubernetes staging and
# thus have different versioning (or none at all...). We need to
# skip those. We know what packages are from staging because we
# now have "replace" statements for them in go.mod.
#
# shellcheck disable=SC2001
module=$(echo "$package" | sed -e 's;k8s.io/\([^/]*\)/.*;k8s.io/\1;')
if grep -q -w "$module *=>" go.mod; then
deps="$deps $(echo "$package" | sed -e "s;\$;@kubernetes-$k8s;" -e 's;^k8s.io/kubernetes\(/.*\)@kubernetes-;k8s.io/kubernetes\1@v;')"
fi
done

# shellcheck disable=SC2086
(set -x; env GO111MODULE=on go get $deps 2>&1) || die "go get failed"
echo "SUCCESS"

0 comments on commit c1078a6

Please sign in to comment.