-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Release kubebuilder and k8s bundles separately #686
Comments
cc: @droot @DirectXMan12 |
/assign @DirectXMan12 So we (Cluster API) have a transitive dependency problem due to kube-builder deps. Kube builder forces us to take a specific version of kubernetes dependencies that we ideally would like to have abstracted. This causes a bunch of issues when trying to vendor in other Types whose versions may be newer. This is a pretty big problem for us and would love for it to be fixed. |
@timothysc this issue is mostly about testenv bundles, not about dependencies. |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/lifecycle frozen /help |
@DirectXMan12: Please ensure the request meets the requirements listed here. If this request no longer meets these requirements, the label can be removed In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
It was requested here a well: #1446. So +1 👍 for this request. |
The remaining things to do here are to rev the release process and maybe add a little helper to the kb binary to download different deps. We already release a standalone copy of the deps, so we just need to to tweak the main process. |
/good-first-issue |
@DirectXMan12: Please ensure the request meets the requirements listed here. If this request no longer meets these requirements, the label can be removed In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
For what I'd look for in the download deps: list from GCS bucket's XML response, convert to version when possible, falling back to straight names in case we change the format, download to location specified in the |
don't do this automatically -- have a separate command. Do warn on |
Hi @DirectXMan12, Shows that KB has bin shipping the bins of So, I understand that for we are able to remove the bins from the kb build we need:
.PHONY: testbin
testbin:
./test-setup.sh `script to setup the bins (valid for mac and linux)`set -eu
# */
# To use envtest is required etcd, kube-apiserver and kubetcl binaries in the testbin directory.
# This sript will perform this setup for linux or mac os x envs.
# */
K8S_VER=v1.18.2
ETCD_VER=v3.4.3
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m | sed 's/x86_64/amd64/')
ETCD_EXT="tar.gz"
rm -rf testbin
mkdir -p testbin
# install etcd bin
# the extension for linux env is not equals for mac os x
if [ $OS == "darwin" ]; then
ETCD_EXT="zip"
fi
[[ -x testbin/etcd ]] || curl -L https://storage.googleapis.com/etcd/${ETCD_VER}/etcd-${ETCD_VER}-${OS}-${ARCH}.${ETCD_EXT} | tar zx -C testbin --strip-components=1 etcd-${ETCD_VER}-${OS}-${ARCH}/etcd
# install kube-apiserver and kubetcl bin
if [ $OS == "darwin" ]
then
# kubernetes do not provide the kubernetes-server for darwin, so to have the kube-apiserver is rquired to build it locally
# if the project is already cloned locally do nothing
if [ ! -d $GOPATH/src/k8s.io/kubernetes ]; then
git clone https://github.com/kubernetes/kubernetes $GOPATH/src/k8s.io/kubernetes --depth=1 -b v1.18.2
fi
# if the kube-apiserve is alredy built just copy
if [ ! -f $GOPATH/src/k8s.io/kubernetes/_output/local/bin/darwin/amd64/kube-apiserver ]; then
DIR=$(pwd)
cd $GOPATH/src/k8s.io/kubernetes
# Build for linux first otherwise it won't work for darwin - :(
export KUBE_BUILD_PLATFORMS="linux/amd64"
make WHAT=cmd/kube-apiserver
export KUBE_BUILD_PLATFORMS="darwin/amd64"
make WHAT=cmd/kube-apiserver
cd ${DIR}
fi
cp $GOPATH/src/k8s.io/kubernetes/_output/local/bin/darwin/amd64/kube-apiserver testbin/
# now let's get the kubectl
curl -LO https://storage.googleapis.com/kubernetes-release/release/${K8S_VER}/bin/darwin/amd64/kubectl
chmod +x kubectl
mv kubectl testbin/
else
[[ -x testbin/kube-apiserver && -x testbin/kubectl ]] || curl -L https://dl.k8s.io/${K8S_VER}/kubernetes-server-${OS}-${ARCH}.tar.gz | tar zx -C testbin --strip-components=3 kubernetes/server/bin/kube-apiserver kubernetes/server/bin/kubectl
fi
var _ = BeforeSuite(func(done Done) {
Expect(os.Setenv("TEST_ASSET_KUBE_APISERVER", "../../testbin/kube-apiserver")).To(Succeed())
Expect(os.Setenv("TEST_ASSET_ETCD", "../../testbin/etcd")).To(Succeed())
Expect(os.Setenv("TEST_ASSET_KUBECTL", "../../testbin/kubectl")).To(Succeed())
logf.SetLogger(zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)))
testenv = &envtest.Environment{}
var err error
cfg, err = testenv.Start()
Expect(err).NotTo(HaveOccurred())
close(done)
}, 60)
var _ = AfterSuite(func() {
Expect(testenv.Stop()).To(Succeed())
Expect(os.Unsetenv("TEST_ASSET_KUBE_APISERVER")).To(Succeed())
Expect(os.Unsetenv("TEST_ASSET_ETCD")).To(Succeed())
Expect(os.Unsetenv("TEST_ASSET_KUBECTL")).To(Succeed())
}) Also, in order to help KB users are able to develop test we could/should at least doc it or provide the same feature in the default scaffolded project (Makefile target + script) Note that it shows the reason for I am unable to run a few tests in the controller-runtime on Mac OS X as well. See the issue: kubernetes-sigs/controller-runtime#907 Please, let me know if I am missing something here. c/c @joelanford |
Hi @DirectXMan12, The pr #1600 changes the setup for V3. However, I understand that for we are able to remove the k8s bins from the kubebuilder binary(releases) we would need to do the same for v2 which is a breaking change. Am I right? So, the change in the scripts used to do the KB release can be done just when no longer support V2 and it is removed. c/c @joelanford @estroz |
To give links to the references pointed out in today's community meeting, we have a custom The Makefile makes sure to set everything up https://github.com/kubernetes-sigs/cluster-api/blob/95fe9e2c2c48cb7c765e40fe97861f22765441ff/Makefile#L116-L118 |
Now that The installation docs need to be updated to say something like |
kubebuilder code doesn't have dependencies on k8s, since it's mostly scaffolding.
What have dependencies on k8s are controller-runtime and controller-tools. And both are not managed by kubebuilder.
Thus, IMO it's more reasonable to release kubebuilder and k8s bundle (i.e. kubeapiserver, kubectl and etcd) separately. We can have some mechanic to let the user to choose what k8s bundle to use with kubebuilder.
test
totestData
(part of ⚠️ controller-tools v0.2.0 scaffolding update #682), so go doesn't evaluate it to calculate deps.The text was updated successfully, but these errors were encountered: