-
Notifications
You must be signed in to change notification settings - Fork 5.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Propose a test frameworks git repo #1524
Conversation
/cc @marun |
We propose a new git repository named `k8s.io/testframeworks`. This would be | ||
backed by a github repository `github.com/kubernetes/testframeworks`. | ||
|
||
### What should live in it? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might want to make explicit which other k8s.io/
repos this would be allowed to import
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good call. Which repos would folks like? :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally, none.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But the e2e might depend on the client? (Or does it shell out?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it does, then I don't think it can live outside of k/k.
avoid characters that are not lower case alphabetic characters. This precludes | ||
for example `k8s.io/test-frameworks`. | ||
|
||
### Who should own it? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, what does ownership mean?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this email when @spiffxp said "would naturally fall under the ownership of sig-testing", I presumed that "ownership" effectively meant something like "whose names go in the top level OWNERS file". I may have been wrong...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would expect the individuals proposing to work on this project to be the leads for it, and responsible for actually driving / implementing it. This is not a thing I have the capacity to staff or drive. So I'd like to see the "implementation owner TBD" part filled out.
But in the context of SIG's needing to own all things, in the same way that SIG Apps "owns" helm and charts, I think it makes sense for SIG Testing to own this. I would like to hear from @timothysc if this aligns with his proposal to expand the charter of sig-testing but I'm personally +1 here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the "individuals proposing to work on this project" are initially me, @hoegaarden and @apelisse. So far all the PRs on the integration framework have been authored by me and @hoegaarden, and reviewed by @apelisse.
I can't speak with as much certainty for @apelisse , but I know that @hoegaarden and I are happy to be responsible for driving/implementing this framework so long as that makes sense for everyone else.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added our (my, @apelisse, and @hoegaarden's) names to the proposal, and clarified my understanding of the difference between the "SIG's needing to own all things" context, and the "day to day looking after the code" context. Does that work for everyone?
/ok-to-test |
Specifically in the light of [this conversation](kubernetes#1524 (comment))
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: spiffxp, totherme The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these OWNERS Files:
You can indicate your approval by writing |
I think we may want to frame specifically which other repos this framework could import, if any, and enumerate them in the proposal. As it's meant to be helping with import cycles, that seems important to me as part of the charter for the repo. |
I can't edit the pull-request and its authors are on vacation. I'd still really like the repo to exists when they come back. I'll mention it here again, and I think it definitely should be printed in bold in the README at the root of the repo, but that new repo will NOT import any other kubernetes repository (maybe |
I've added a section saying explicitly that no k8s repos other than I can see (using |
@totherme you have understood correctly, thanks |
|
||
## Proposal | ||
|
||
We propose a new git repository named `k8s.io/testframeworks`. This would be |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd prefer just to call it the testing repo. b/c there could also be separate libraries and possible binary artifacts.
There also [seems to be a | ||
desire](https://groups.google.com/a/kubernetes.io/d/msg/steering/LA9WiFnl6PI/os48-c3HCgAJ) | ||
to get `kubernetes/test/e2e/framework` out into a more reusable place. It could | ||
live in `k8s.io/testframeworks/e2e` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a number of test-specific utility libraries that should be housed in this repo.
|
||
### What repos can be imported (and hence vendored) into this repo? | ||
|
||
No kubernetes repos other than [utils](https://github.com/kubernetes/utils) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you'll absolutely need either the internal or external client.
part of kubernetes has to do so through some CLI. The [CLI integration testing | ||
framework](https://github.com/kubernetes/kubectl/tree/master/pkg/framework/test) | ||
which originally motivated this repo certainly does not have any kubernetes | ||
dependencies. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's not possible, but it is possible to ensure it's a proper DAG.
This falls in line with the subgroup proposed for sig-testing that solely focuses on tests, and code organization as well. |
@timothysc @stevekuznetsov I'd like to leave it to you two to decide how much of this needs to be hashed out before /lgtm'ing In the meantime, I'm working on creating https://github.com/kubernetes-sig-testing to host a repo called "frameworks" that would live as |
Per discussions, I'm ok with this, but I don't think we should merge this proposal and we should use this example to serve as a test case for @brendandburns proposal. |
@timothysc I guess I would prefer to see this proposal amended to reference the repo I'm creating per said proposal, so there is written documentation somewhere explaining its motivation. |
I've amended this proposal to reference the repo that @spiffxp is creating, and to clarify my understanding of what happened in the recent sig-architecture meeting. |
Sure, but lets not merge this unless we have to, in preference for vetting on the official doc. |
@timothysc I'm not clear on what this means, can you elaborate? |
@timothysc: Github emailed me comments on this issue (dated Jan 9) from @sttts regarding how the new repo wouldn't be able to rely on any staging repos if it was to be consumed by k/k, but I don't see those comments reflected here. I think it may be worth reconsidering the use of an independent repo. I would like to ensure the evolution of test tooling for both k/k (where the bulk of testing work is currently done) and external repos (where the bulk of testing is likely to be done in the future), and from @sttts's comments only a repo staged out of k/k will accomplish both goals. |
So it already exists and the spec is in flight for a general-ish solution, closing. |
Following up on this email thread, this is a proposal for a new git repository to house:
kubernetes/test/e2e/framework