MCO-507: admin defined node disruption policy enhancement

[yuqi-zhang]

Enhancement for users to specify how MachineConfig updates should be acted upon.

See: https://issues.redhat.com/browse/OCPSTRAT-380

[openshift-ci-robot]

@yuqi-zhang: This pull request references MCO-507 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target either version "4.15." or "openshift-4.15.", but it targets "openshift-4.16" instead.

In response to this:

Draft enhancement for users to specify how MachineConfig updates should be acted upon.

See: https://issues.redhat.com/browse/OCPSTRAT-380

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[djoshy]

I like the idea of degrading the operator via conditions as this follows our existing pattern, but just to be super clear, are we talking about degrading the operator via new condition types on an object(say the RebootPolicy in this case), or a new condition type on the MCC itself? I don't think we do degrades via the MCC directly like that, but please correct me if I'm wrong!

You also mention degrading the daemon too, which makes sense. I assume both the controller and the daemon degrades will be simultaneous(barring sync delays), or would there be cases where each could happen independent of the other?

[yuqi-zhang]

So in this original model, I was thinking that the daemon would only error if it had an ongoing upgrade while the RebootPolicy was invalid. If it's not being currently used, the daemon wouldn't double report the error. In the case where both is erroring, it would be like:

1. controller is failing on validation of RebootPolicy
2. daemon is failing on waiting for validation of RebootPolicy

Hopefully that's not too confusing. As for the controller error, I was thinking of just erroring out the sync loop as a start. But maybe having the operator object itself sync the above RebootPolicyValidationFailing would be better, now I think about it

[djoshy]

This is probably out of my wheelhouse but I'm curious - are actions going to calculated and accumulated, and then applied at the end? Or would it be each action to take place in the order it came(regardless of the type of action)? Is there a disadvantage over doing one over the other? I know you mentioned checking for duplicates earlier, but just want to clarify that applies to actions or not

For example, consider these actions from two reboot policies(not applied at the same time)

• Policy 1
  reload crio
  reload crio
  reboot

• Policy 2
  reload crio
  reboot

Would policy 1 cause two crio reloads? Or would the actions take from both of them be the same(1 reload crio, 1 reboot)

[yuqi-zhang]

Great question, I was hoping we can get away with any merge logic and only validate that you don't have 2 entries for the same object (e.g. one file defined twice, once being a no-action and once being a reload, which self-contradicts). Then afterwards we just apply them in order from the daemon without any additional logic.

Is there a disadvantage over doing one over the other?

Other than doing unnecessary actions (reloading the same service multiple times, for example) there shouldn't be any real downsides. I guess there are potentially scenarios where you need to reload the service in a specific order, but I think we can preserve the definition ordering so that shouldn't be a big issue

[cgwalters]

So this enhancement doesn't yet mention the bootc configmap approach (we aren't using it yet I know, doesn't really exist)

But anyways in there I was straw-manning that we have an annotation in the configmap which specifies the "update policy" for it (apply live vs reboot) - there's some ugly bits around whether we need "apply live and execute this arbitrary command" or "apply live and reload these systemd units" that need to be defined.

But anyways to me, associating the update policy with the files makes a lot more sense than having another global list.

Now to implement this today we'd need to add a "reboot policy" section into the rendered config. But I think it would be a lot nicer in the future to get rid of the idea of the big "rendered config" and have the MCD fetch individual MachineConfig objects (like how kubelet fetches individual configmaps, there's no "rendered configmap!").

Doing that gets into versioning/race condition questions around "what happens when an input MC changes while we're doing a rollout" that I think are solvable, but do need some thought.

Now one thing @yuqi-zhang brought up when we were chatting about this is the overall question of "how would this work for files in the base image".

My offhand thoughts on that are that we may want to go in a direction of having "static configmaps" that we can inject into a container image in some way, similar to the arguments around containers/bootc#128

(These things connect because configmaps may specify quadlet units)

Then bootc would be the thing that executes on applying these note!

Anyways...I'm not totally sure we really need to immediately handle the case of handling policies for files embedded in the base image.

It feels like we'd get most of the way if we split out the things that need to be updated live outside of the base image.

[yuqi-zhang]

Great points 👍 , a few thoughts around this:

1. The global approach in a sense can help us merge the paths of MachineConfig and Configmap defined objects. So if you define a rebootpolicy, it can apply to both update paths, and eventually container image diffs potentially. So there is a singular API and the MCO can handle the specifics when doing the actual content update. I guess the downside would be that if bootc configmaps allow for that we'd have multiple locations where this can be defined, but that can also be an MCO implementation detail and this should give us enough flexibility
2. The timeline for this is going to be 4.16 which will likely predate any full implementation of bootc live-applies and configmaps going into the MCO, since there is some urgency around this, otherwise I'd love to wait and see what we finalize on in terms of bootc-based updates
3. The Configmap approach doesn't map as well to MachineConfigs (at least not the annotations) since a lot of MachineConfigs are a giant bundle of different files/units, due to our existing architecture, and not one-file-per-machineconfig
4. As a bridge gap, I think the idea of extending the MachineConfig API instead of a new API object is an alternative approach we can pursue. This may actually be easier to conceptualise for some users since I think most users think of "changing a machineconfig" as the catalyst to an update, and not necessarily "a new rendered config for the system was created and we're doing a diff update". I did consider this approach and originally rejected it because most of the "no-reboot updates" that are asked about are certificates, or registry configuration etc. which comes all in the cluster-supplied 00-worker and 00-master MCs, which you can't edit. And you definitely should not be providing your own machineconfigs for those files.
5. So then if we do opt for this instead I guess the user would end up defining a RebootPolicy as a separate MachineConfig still (for example, for image registry mirror changes, they would define a machineconfig that just has a "no-action" field). The downside would be... that it's a bit confusing for users to write a MachineConfig specifically to define the "how does my file reboot", but I don't see a great alternative. The upside is that it can be separated from the configmaps in the future and both would work separately.

I'm still on the fence about this. Do you think having the MCO write RebootPolicies into bootc configmaps instead eventually would work as well?

[cgwalters]

I did consider this approach and originally rejected it because most of the "no-reboot updates" that are asked about are certificates, or registry configuration etc. which comes all in the cluster-supplied 00-worker and 00-master MCs, which you can't edit.

But we can easily split out those files into a distinct e.g. 01-worker-configs MC today and aim for stuff that changes in -configs to have distinct "apply" policies.

[cgwalters]

I don't get why we'd mix "cluster" and "user" in one single CRD. Why not have RebootPolicy have multiple instances (like machineconfig and tons of other things) and just some of them are cluster owned (i.e. changes get overwritten)?

[yuqi-zhang]

I opted to define the initial approach this way for simplicity because it doesn't require us to:

1. merge multiple objects
2. validate separately

So in this way we can validate via the object via the singular cluster object status, and the daemon can just react to that, and we can re-evaluate for GA. We have a similar schema for the "node.config.openshift.io" object (only one "cluster" object is allowed")

But it is definitely more flexible and safer to have multiple... we can implement a merging schema. I don't really want us to have to create a "rendered-rebootpolicy" object but I guess we can have the controller validate each one separately, and the daemon to apply them in... alphanumerical order? We may not even need a merging schema and just have the daemon apply them in order, which at worst will be a few duplicated commands.

[JoelSpeed]

Do we think this needs to be a new CRD or could it be part of the MachineConfiguration operator CR?

[yuqi-zhang]

I defaulted to a machineconfig-like approach of a new CRD since it's a bit easier to manage, but I think technically speaking there's no reason why it can't be part of MachineConfiguration.

So then our options would be:

1. have it be part of MachineConfiguration knobs
2. have it be a new CRD
3. have it extend MachineConfig objects

I think if 1 is easier we can certainly take that approach, but I wonder if we'd be stuffing too many things into MachineConfiguration. What's the general guideline on what warrants a new CRD vs just a high level knob?

[JoelSpeed]

Do you expect there to be one configuration per cluster for this? If you do, then I think containing it in the MachineConfiguration singleton probably makes sense.
If the config might vary per machinepool and you expect we might need many duplicates, then perhaps we look at something separate.

The guidance I'd give is about the consumer. Group things together that the same consumers would want to look at/configure. I expect this reboot configuration will fit in nicely within either the MachineConfiguration singleton, or within the MachinePool if you wanted the MachinePools to have independent configs

[yuqi-zhang]

I think of the use cases we know of so far, there shouldn't be a need to specify this per-machineconfigpool. Doing it as a separate CRD will give us more flexibility (and the CRD name itself is a bit more direct on what this is configuring, vs the overall MachineConfiguration object) but I think perhaps the end goal is to make this work better with images, so in this sense, doing it via the MachineConfiguration object for cluster-wide, individual MachineConfig effects works best as a first implementation

[JoelSpeed]

TBH, we can start with either approach during tech preview, there's no need to promote whatever we do in tech preview to GA. But perhaps we need to explore the pros and cons of each approach first.
The one thing to bear in mind though, is that, once a name has bene used, even in techpreview, it cannot be re-used later with a different shape, so we should probably sync up on this at some point.

I think for now I lean towards seeing what the reboot policy would look like within the MachineConfiguration CR

[JoelSpeed]

Why does this need to be done via a controller rather than at admission time using CEL or a validating webhook?

[yuqi-zhang]

I wanted to leave this open for potentially merging multiple rebootpolicies, sort of like MachineConfigs, since I was a bit ambivalent on having just one.

So in that case, users can still provide any configuration of their own, cluster would be immutable and generated by default, and the controller will "merge" them in some fashion.

I suppose if we want to just have one object (or as a field in MachineConfiguration as mentioned above) having a CEL or validating webhook will be better. I assume those can't be used to cross-validate against other objects?

[JoelSpeed]

There are options for cross validating against other objects, we can explore that if it is a requirement.

In general, having the reboot policy defined in a single place would likely prevent duplication anyway no? For example, we can use map keys to make sure certain keys within the lists have unique entries

[JoelSpeed]

Typically we can prevent these objects from being created using validating webhooks, or now even a CEL validation. So perhaps restate this to say that CEL validation will prevent creating an object not named cluster

[JoelSpeed]

How will you stop users modifying these, but allow the controller to update these? Perhaps this should be enforced by the "have a spec, copy to status, observe from status" pattern

[yuqi-zhang]

I intended to have the controller validate and overwrite any changes to those fields, much like any other object the MCO manages (for example, if you try to overwrite any of the controller-managed MachineConfig objects, the controller will simply re-apply the old one back. There is a delay so the rendered config doesn't get generated.)

The status will just reflect that the controller has ack'ed whatever is in the spec in this current pattern, so we can save re-listing all the policies in the status. But if the copy-to-status pattern is better, I think we can save a bit of waiting in the daemon (the current approach has the daemon wait for the latest, vs using the last-validated in the status in this new method) so it might be better?

[JoelSpeed]

I would probably go for the copy to status approach.

The idea here is that the user can define what they want, and that is spec. What the user says should never be modified. A controller then validates the changes and copies to status, if the changes from the user are acceptable, rejecting and logging (via a condition maybe) why it's ignoring anything from the user spec, and merging anything that must exist.

Then whatever consumes the data, can always rely on the status fields, because they have already been validated.

[yuqi-zhang]

Ok, as an MVP, I think we can try for MachineConfiguration spec and status.

[JoelSpeed]

What does this mean for support cases? It sounds like we need to have an internal list of what can and cannot skip reboots, and force reboots where they are actually required.

[yuqi-zhang]

I think at least initially, the usage of rebootpolicies means that you the user take responsibility for them, so from a support perspective this shouldn't be any different than say, you modified a file manually via ssh, and broke your cluster as a result.

[JoelSpeed]

When I SSH into a node, it tells me that this has been logged and the node is now out of support. Will you have something similar for reboot policies?

[yuqi-zhang]

When I SSH into a node, it tells me that this has been logged and the node is now out of support. Will you have something similar for reboot policies?

So I think we actually removed the ssh warning a few versions ago. The whole ssh saga was a long history of errors and sub-optimal behaviour.

The SSH warning was part of a node annotation (and I guess, a daemon log entry) and didn't make it further than that. We can certainly add a node annotation when a rebootpolicy did take effect. Is that sufficient? We can also try to leverage the new MachineConfigNodes objects to reflect in their status instead

[JoelSpeed]

I think we need to come up with something to define our boundary for support. If you think there are going to be complex scenarios if users use certain features in tandem, then prevent them from doing so, or, flag it in a status so that we can say, oh well you did XYZ which we told you not to do, that's why it broke.

If we don't do this, I expect you'll see a number of bugs come through that are complicated, will require investigation, and then will conclude, oh they did something unsupported

[JoelSpeed]

How will you explain this to a user?

[yuqi-zhang]

Docs, hopefully :)

We can craft an alert or something to indicate when their rebootpolicy took effect upon an update. Without constraining the user to set based on a list of tested "safe" policies (Which goes back to what we already have today, and we don't want to continue extending necessarily) there isn't a way to guarantee (that I know of) user-defined policies to be always "Safe"

[JoelSpeed]

Providing some metric or alert that is easy for support to spot I think would have value

[JoelSpeed]

I would like to see the Go type for this, with godoc documentation and kubebuilder validations. Currently the API looks almost scary to me, in that I have very little context of what any of the things here mean. I'm concerned about how usable this will be to an average user so documentation on this API will be very important IMO. Perhaps we can discuss in the API extensions section

[cgwalters]

This one looks very similar to https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_handlers.html and would almost certainly face the same pressure to become similarly extensible.

[JoelSpeed]

Given that, I think this API is going to need to be thought out very carefully, we can take a look at the extensibility built into playbook API as inspiration, thanks for the link

[JoelSpeed]

A users expected to understand the changes introduced as part of the payload, and then add policies before they upgrade? If so, where will they get this information?

[yuqi-zhang]

Payload updates are not what this is intended for. Basically every payload update we have contains an OS update to RHCOS, which isn't rebootless.

The main use case is for specific machineconfig-defined file changes to skip this process, outside of major version upgrades

[JoelSpeed]

Oh ok! I didn't get that when reading the motivation/goals, perhaps an update there might make that more obvious.

Given that, I think this probably alleviates my concern. The user is making changes to lay specific files down or change specific files, so they know what they need to do.

Is there an ordering issue here? Do they need to make sure that the reboot policy change is applied before they apply the MachineConfig change?

[yuqi-zhang]

perhaps an update there might make that more obvious

I'll add a snippet to indicate that this is not for OCP major/minor upgrades, but instead for individual MachineConfig updates performed by the user

Is there an ordering issue here? Do they need to make sure that the reboot policy change is applied before they apply the MachineConfig change?

With the validation step above, I guess the user must ensure that the rebootpolicy they want has properly made it into the status of (either the MachineConfiguration or a new CR) object before starting their MachineConfig update

In practice if you apply them in quick succession, there is a small inbuilt delay into MachineConfig rendering -> pool acquiring -> daemon processing (of ~15 seconds) such that the rebootpolicy should take effect. I don't think we need to have addition inbuilt checks for this

[JoelSpeed]

This seems very very specific. Do we think users want this level of granularity, or do they want classes of changes to ignore reboots for?

[yuqi-zhang]

We can certainly support wildcarding paths, but this is mostly inline with the MachineConfig objects we have today.

What would you define as a "class of change" in this situation?

[JoelSpeed]

I was thinking about this applying to all changes, I think if we are saying this is only applicable to everyday changes and not updates, then I don't think this comment really applies

[cgwalters]

At an overall thing I think we don't want to hardcode the term "reboot" here because some changes today are still disruptive in terms of draining, but don't actually involve a reboot right? (Hm, the code flow in the MCO today is a bit confusing on this; I'd have expected postConfigChangeActionDrain but it's special cased)

Certainly in bootc I plan at some point to support systemd userspace-only restarts too which will act differently from reboots.

And then this whole enhancement is really about enabling special casing specific changes which are not reboots...

So maybe "Admin defined Host Change Apply Policy" ?

[yuqi-zhang]

I picked "reboots" since that's how most people view it :) I was considering the alternative of "Admin defined disruption policy" instead but disruption has multiple meanings

The title I can certainly change to be more descriptive, the naming of the api object (or other alternatives) is definitely more important

[yuqi-zhang]

Added 2 alternatives as discussed above, with some of the benefits and drawbacks to the approaches. Happy to discuss further

[yuqi-zhang]

Addressed most of the comments and updated the enhancement. Key points:

1. now is a sub-section of MachineConfiguration object instead of a new CRD
2. now called Node Disruption Policy instead of Reboot Policy, for clarity (since it encompasses drains, etc.)

[yuqi-zhang]

API draft here: openshift/api#1764

[dhellmann]

#1555 is changing the enhancement template in a way that will cause the header check in the linter job to fail for existing PRs. If this PR is merged within the development period for 4.16 you may override the linter if the only failures are caused by issues with the headers (please make sure the markdown formatting is correct). If this PR is not merged before 4.16 development closes, please update the enhancement to conform to the new template.

[sinnykumari]

ending of this sentence seems a bit confusing. Perhaps we wanted to write something after application

[yuqi-zhang]

Edited

[sinnykumari]

how are we envisioning applying kargs without performing reboot?

[yuqi-zhang]

Removed that from the list in the API and here

[sinnykumari]

linking to API PR would make sense here.

[yuqi-zhang]

Done

[sinnykumari]

I am a bit confused how pausing pool impact here. Are we allowing admin to skip drain/reboot for OS update or something else I missing?

[yuqi-zhang]

Reworded to remove confusion. I didn't mean to talk about paused pools

[sinnykumari]

Keeping policies at cluster level makes sense to me initially since in regular usecase if a change is safe, it would be safe to apply on all nodes. Maybe something we can fine tune afterward based on usecases.

[sinnykumari]

Can be useful for applying at directory level. Especically when adding drop-in files into drop-in directory.

[sinnykumari]

User can write a systemd unit to achieve that and that unit itself can be defined in the policy :)

[sinnykumari]

icsp changes are tricky. We may want to handle it as a special case. Something like ClusterDefaultPolicy for this would be set to no drain and reboot for addition. And userPolicy may specify deletion.

[JoelSpeed]

It's not clear to me (though I think from prior discussion this is true) that this policy only applies to users modifying MachineConfig objects, and doesn't apply in the general case for node upgrades. It would be good to clarify if users can prevent reboots if say, an upgrade says the kubelet service needs to be updated, and they specify to prevent that reboot or service restart.

Having users have the ability to prevent nodes from restarting for core updates to kubelet and crio could cause support concerns down the road.

Another thing I may have missed, but, will I be able to apply the policy to a subset of machine config pools? Is there any time where I may want to skip a reboot on workers, but I'm happy to do so on masters?

[JoelSpeed]

This probably wants to be a condition

[yuqi-zhang]

Updated and added a TODO in the API PR

[JoelSpeed]

This will need to be spelt out clearly in the API. I would like to understand more about the Special keyword so lets talk about that at some point, how it manifests and the implications of it

[yuqi-zhang]

The "special" keyword is reserved for the MCO-defined actions and we will reject any user policies with this keyword.

I was hoping to use the phrasing in describe to flesh out what this means to the user. There are basically additional logic for registries specifically around mirror-by-digest-only changes that exist in the MCD today. This is just for user awareness and not something they can set themselves.

[JoelSpeed]

I think the API change to make path the map key will mean this statement is null

[yuqi-zhang]

Removed

[JoelSpeed]

Do we have an API extension for this?

[yuqi-zhang]

Not today, since MCN is also tech preview.

Will make it a goal for GA

[JoelSpeed]

Please link to API PR at a minimum. Providing YAML examples to show usage would also be helpful for other readers. We can also talk about the validations that will apply here

[yuqi-zhang]

done. An example is provided above

[JoelSpeed]

I didn't think the disruption policy would apply to cluster version upgrades?

[yuqi-zhang]

It doesn't. Removed

[openshift-bot]

Inactive enhancement proposals go stale after 28d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle stale.
Stale proposals rot after an additional 7d of inactivity and eventually close.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale enhancement proposals rot after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Rotten proposals close after an additional 7d of inactivity.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[yuqi-zhang]

/remove-lifecycle rotten

Updated based on updates to the API PR

[djoshy]

Overall looks good to me! Just a few minor nits and some questions about the future work. Thanks for working on this, Jerry!

[djoshy]

Might need a newline here for better formatting

[yuqi-zhang]

done

[djoshy]

Probably worth adding directories/wildcarding plans here too?

[yuqi-zhang]

done

[djoshy]

I suppose this could change in the future, but as it stands we are completely guarded by API validation, no?

[yuqi-zhang]

modifed based on our current implementation

[djoshy]

Nit: better formatting

[yuqi-zhang]

done

[djoshy]

This makes sense to me. Off the top of my head, I guess we need:

• NodeDisruptionPolicyApplySuccessful - this will include the generation of the overall object and MCD can track/wait on this before starting to apply the actions in Status
• NodeDisruptionPolicyExecutionSuccessful - perhaps list the target config and the actions applied
• NodeDisruptionPolicyExecutionFailure- indicate why failure happened and that the legacy path was taken. Right now I can only think of this being the status not being generated correctly/in time..or in a buggy format that we did not anticipate?

[yuqi-zhang]

I think we can start off with just a success/fail when the node update gets to that stage. But agreed, we can add a lot of detail here for users to see, if they care

[djoshy]

/lgtm
Thanks for working on this with me @yuqi-zhang I believe this is now inline with our current MVP. I think the test can be overridden since we are still in 4.16 development window, but I don't believe I have that power. Would you mind overriding @JoelSpeed , assuming you have no other reservations? (:

[yuqi-zhang]

Thanks David :) Based on the current failure, seeing if we can override based on #1436 (comment)

/override ci/prow/markdownlint

[openshift-ci]

@yuqi-zhang: yuqi-zhang unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:openshift: openshift-release-oversight openshift-staff-engineers.

In response to this:

Thanks David :) Based on the current failure, seeing if we can override based on #1436 (comment)

/override ci/prow/markdownlint

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@yuqi-zhang: This pull request references MCO-507 which is a valid jira issue.

In response to this:

Enhancement for users to specify how MachineConfig updates should be acted upon.

See: https://issues.redhat.com/browse/OCPSTRAT-380

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sinnykumari]

All existing comments has been addressed and respective API and MCO implementation has been merged.
Let's take any follow-up feedback anyone may have later on in a follow-up PR.
/approve
Overriding test based on #1525 (comment)
/override ci/prow/markdownlint

[openshift-ci]

@sinnykumari: Overrode contexts on behalf of sinnykumari: ci/prow/markdownlint

In response to this:

All existing comments has been addressed and respective API and MCO implementation has been merged.
Let's take any follow-up feedback anyone may have later on in a follow-up PR.
/approve
Overriding test based on #1525 (comment)
/override ci/prow/markdownlint

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sinnykumari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sinnykumari]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@yuqi-zhang: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.