-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add initial E2E test and structure #203
Conversation
- name: registry.k8s.io/cluster-api/kubeadm-bootstrap-controller:v1.1.5 | ||
loadBehavior: tryLoad | ||
- name: registry.k8s.io/cluster-api/kubeadm-control-plane-controller:v1.1.5 | ||
loadBehavior: tryLoad |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I found that the framework will load these images into the kind cluster, but wont pull them down. So you have to just have them on the machine first which is a pain.
IDK if the capa or oci providers use some extra framework trick, I have not found it yet. For now I am going to script something i guess, or write a helper func in the tests.
edit: reading the framework code, it looks like it should be pulling images that are not found locally 🤔 wonder why i was seeing errors
edit2: aha! because i am using an older version of capi that is why! We are using CAPI 1.1.5, which I left as I did not want to get into bumping that, and 1.1.5's franework stuff does not pull images for you. CAPA is using 1.1.2, which also does not pull images if not found, but I cannot find what they are doing to pull the images themselves. CAPOCI is using 1.2.0. which does have a framework bit which will pull down the images if they do not exist locally.
Do I want to get into bumping CAPI? Will have a go, but I tried that at the start of this and quickly decided it was a pain so we shall see.
final edit: decided not to get into that mess and just wrote a small func with a note that we can get rid of it when we bump CAPI
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In CAPA we have this: https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/main/Makefile#L440
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
eh mine is better ;)
@@ -4,6 +4,7 @@ go 1.17 | |||
|
|||
require ( | |||
github.com/go-logr/logr v1.2.3 | |||
github.com/onsi/ginkgo v1.16.5 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we are pinned to ginkgo v1 until CAPI and literally everything else bumps to v2
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this will be happening 22nd Aug: kubernetes-sigs/cluster-api#6906
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not holding my breath ;)
de59ad8
to
50f983e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally all looks ok to me, just one comment around the machine pool part.
My brain is a little fried (its been a long day) so will take another look in the morning when i'm more switched on.
fc09114
to
821ac40
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good, great work @Callisto13 .
I have run this locally across 2 machine and it all passed. I even had to use E2E_ARGS
because i have a different ip range for my machines.....E2E_ARGS="-e2e.capmvm.vip-address 192.168.9.10"
.
sorry! i forgot to push that small change i did yesterday afternoon thanks for checking that E2E_ARGS works fine 👍 I might change how the flintlock hosts are injected at some point |
This adds a single test to create a CAPMVM workload cluster and a bit of structure so that more tests can be added later. Note that only CAPMVM on the current branch is tested, not released versions or upgrades between versions. I have deliberately left it so we can add those bit by bit after the general setup is merged in. I have a lot more to do in the e2e saga, but this is a first step which is useable. I used the CAPI test framework[1] which seems to be something of a standard across providers, although it is a bit unwieldy. Everything can be found under `test/e2e/`. The contents of `test/e2e/data` are mostly yaml and can be ignored. The dir `test/e2e/config` contains the config file used by the CAPI test framework. Other config files can be passed in via test params but this is the default. The test itself is at `test/e2e/e2e_test.go` and `test/e2e/e2e_suite_test.go`. The suite performs some env setup and teardown by way of a manager. It is assumed for now that the tests are not run in parallel (there is only one of them) so I did not set up any synchronization between nodes, but that will need to be done as we add more tests and parallelization. The test creates a cilium flavour cluster, mainly because I could not get the other one to work and did not want to spend a lot of time fixing that. After the cluster is created it checks that all flintlock hosts were used, and that a simple application can be successfully deployed. In `test/e2e/utils` there are a bunch of helper things, which will probably end up being broken up as they expand: - `params.go` sets up test flags - `defaults.go` holds defaults for test flags - `utils.go` is a bunch of helper funcs used by the test - `clusterctl.go` contains some functions I copied from the framework because I had to edit them and they are not hugely flexible. - `manager.go` holds some objects and methods which handle environment setup/teardown and shared test information. The test can be run locally by: - Starting 1 or more flintlock servers - Exporting the server addresses `export FLINTLOCK_HOSTS="1.2.3.4:9090,5.6.7.8:9090"` - Running `make e2e` [1]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The additions to the readme are good.
This adds a single test to create a CAPMVM workload cluster and a bit of
structure so that more tests can be added later.
Note that only CAPMVM on the current branch is tested, not released
versions or upgrades between versions. I have deliberately left it so we
can add those bit by bit after the general setup is merged in.
I have a lot more to do in the e2e saga, but this is a first step which
is useable.
I used the CAPI test framework1 which seems to be something of a
standard across providers, although it is a bit unwieldy.
Everything can be found under
test/e2e/
.The contents of
test/e2e/data
are mostly yaml and can be ignored.The dir
test/e2e/config
contains the config file used by the CAPI testframework. Other config files can be passed in via test params but this
is the default.
The test itself is at
test/e2e/e2e_test.go
andtest/e2e/e2e_suite_test.go
. The suite performs some env setup andteardown by way of a manager. It is assumed for now that the tests are
not run in parallel (there is only one of them) so I did not set up any
synchronization between nodes, but that will need to be done as we add
more tests and parallelization.
The test creates a cilium flavour cluster, mainly because I could not
get the other one to work and did not want to spend a lot of time fixing
that. After the cluster is created it checks that all flintlock hosts
were used, and that a simple application can be successfully deployed.
In
test/e2e/utils
there are a bunch of helper things, which willprobably end up being broken up as they expand:
params.go
sets up test flagsdefaults.go
holds defaults for test flagsutils.go
is a bunch of helper funcs used by the testclusterctl.go
contains some functions I copied from the frameworkbecause I had to edit them and they are not hugely flexible.
manager.go
holds some objects and methods which handle environmentsetup/teardown and shared test information.
The test can be run locally by:
Pulling the images listed at the top of
this is no longer necessary, i fixed ittest/e2e/config/e2e_conf.yaml
to the local machine running the tests(except for the capmvm image, which will be built)
make e2e E2E_ARGS="-e2e.flintlock-hosts 1.2.3.4:9090,5.6.7.8:9090"
Checklist: