Add resource version checks #3993

johnbelamaric · 2023-06-30T16:54:45Z

Adds ResourceVersion handling, including using an annotation to manage it for rpkg pull and rpkg push.

johnbelamaric · 2023-06-30T16:57:16Z

Also adds some instrumentation in ensureSameOrigin. Right now, in Nephio, our build has that function disabled. However, I am curious if the problems we saw with that function were due to packages getting corrupted by the concurrency issues, so I am leaving it enabled in this PR.

johnbelamaric · 2023-06-30T19:01:26Z

Ok, I still get the error Error: Internal error occurred: cannot create revision of free5gc-upf with a different origin than other package revisions in the same package even with this fix. So we will still need to disable that for the Nephio build.

Should I just comment it out in this PR?

mortent · 2023-07-04T22:32:44Z

So this adds the resourceVersion to the Kptfile. Is that to make sure it handles the resourceVersion comparison even if it goes through a pull and then later a pull?

I think there are three separate issues related to this and we should probably make sure we address them all (although not necessarily in this PR):

Make sure we correctly check the resourceVersion for normal fields on the PackageRevision resource.
Make sure we handle this correctly in terms of concurrency so we don't have races between concurrent updates to the same resource. I think this can be challenging due to the way we split data about each resource across both git/oci and a CRD. But we should at least do what we can to minimize the chance of issues here and make sure we understand the cases where it might not be handled correctly.
Handle resourceVersion for updates to PackageRevisionResources (i.e. what this PR is about).

johnbelamaric · 2023-07-05T15:59:01Z

So this adds the resourceVersion to the Kptfile. Is that to make sure it handles the resourceVersion comparison even if it goes through a pull and then later a pull?

So, when using K8s clients, resourceVersion management works as expected simply by setting the metadata.ResourceVersion in the PackageRevision and PackageRevisionResources. However, if you do a pull, that metadata does not exist anywhere in the package. We do not store the source git commit ID from a pull, anywhere that I could see. So, in a pull I now stick it on an annotation in the Kptfile. But this is ephemeral; it is not stored in git. It is just added/removed by the API so that non-K8s clients (well, really just kpt) can work without modification.

Another option would be to have the kpt pull and push commands do this themselves. That may be preferable.

johnbelamaric · 2023-07-05T16:44:23Z

Make sure we correctly check the resourceVersion for normal fields on the PackageRevision resource.

This is done in this PR as well.

Make sure we handle this correctly in terms of concurrency so we don't have races between concurrent updates to the same resource. I think this can be challenging due to the way we split data about each resource across both git/oci and a CRD. But we should at least do what we can to minimize the chance of issues here and make sure we understand the cases where it might not be handled correctly.

The resourceVersion check done in this PR should handle this in the happy path. The failure paths may be more challenging (if we make a change to the CR, then cannot change Git, or vice-versa, they may get out of sync).

There is nothing we can do about someone out-of-band modifying Git. We could put in a webhook to prevent someone from out-of-band modifying the packagerev.

johnbelamaric · 2023-07-05T17:28:46Z

Another option would be to have the kpt pull and push commands do this themselves. That may be preferable.

I wasn't sure how the UI did things. So, I figured the server-side workflow would work for the UI as well. But maybe it's better to assume that the clients are properly behaving K8s clients. In that case, it should be implemented client side (in push and pull). @ChristopherFry do you know, does the UI do the equivalent of the kpt pull and push commands, OR does it use a normal K8s client to pull out the package contents and push new package contents? Specifically, does it handle metadata on the PackageRevision and PackageRevisionResource objects normally?

mortent · 2023-07-05T17:36:29Z

Yeah, I am also thinking that implementing the logic for including the resourceVersion as an annotation in the Kptfile would be better implemented client-side. That means that for Porch, the resourceVersion will always be available in the resourceVersion field from either the PackageRevision or PackageRevisionResources resources. Do you think there are any concerns about handling it that way?

johnbelamaric · 2023-07-05T17:37:31Z

Yeah, I am also thinking that implementing the logic for including the resourceVersion as an annotation in the Kptfile would be better implemented client-side. That means that for Porch, the resourceVersion will always be available in the resourceVersion field from either the PackageRevision or PackageRevisionResources resources. Do you think there are any concerns about handling it that way?

No, it's probably better.

johnbelamaric · 2023-07-06T01:15:12Z

/retest

johnbelamaric · 2023-07-06T16:32:56Z

Ok, I think this is ready for review. I updated it to do the resource version annotation in pull and use it in push, removing it before pushing.

I haven't tried the UI yet.

johnbelamaric · 2023-07-06T20:26:27Z

Ok, yeah, this break the UI. After a clone I get: "Internal error occurred: resourceVersion must be specified for an update". So, it's not using just the K8s API or if so it's not stashing away all the metadata. We'll need to drill into that.

mortent

Just a few small questions, but looks good.

mortent · 2023-07-07T18:53:35Z

porch/pkg/engine/engine.go

@@ -374,6 +383,11 @@ func ensureSameOrigin(ctx context.Context, repo repository.Repository, obj *api.
 		// If there are no tasks, or the first task is not init or clone, then this revision was not
 		// created from another package revision. That means we expect it to be the first revision
 		// for this package.
+		if len(tasks) == 0 {
+			klog.Warningf("not same origin due to no tasks")


Is it intentional that we add this logging in this PR?

I can remove it. This PR does not fix ensureSameOrigin; I think I will disable in a separate PR.

mortent · 2023-07-07T18:54:32Z

porch/pkg/engine/engine.go

@@ -127,7 +131,12 @@ func (p *PackageRevision) KubeObjectName() string {
 }

 func (p *PackageRevision) GetResources(ctx context.Context) (*api.PackageRevisionResources, error) {
-	return p.repoPackageRevision.GetResources(ctx)


Is this change just a leftover from previous drafts of the PR?

ugh, yes. I guess it's equivalent

I can clean it up though, if you want

Not critical, just wanted to make sure there wasn't something here I was missing.

mortent · 2023-07-07T18:55:54Z

porch/pkg/engine/engine.go

@@ -626,6 +644,15 @@ func (cad *cadEngine) UpdatePackageRevision(ctx context.Context, repositoryObj *
 	ctx, span := tracer.Start(ctx, "cadEngine::UpdatePackageRevision", trace.WithAttributes())
 	defer span.End()

+	newRV := newObj.GetResourceVersion()
+	if len(newRV) == 0 {
+		return nil, fmt.Errorf("resourceVersion must be specified for an update")


Is this the normal behavior for resources?

yes, standard k8s behavior

efiacor · 2023-07-20T13:06:51Z

Ok, yeah, this break the UI. After a clone I get: "Internal error occurred: resourceVersion must be specified for an update". So, it's not using just the K8s API or if so it's not stashing away all the metadata. We'll need to drill into that.

Hi @johnbelamaric , do we have an issue to address this? Seeing it now on deploying packages via UI. Also, blocks new revision creation.

johnbelamaric · 2023-07-20T14:12:27Z

The UI is fixed, we just need to use the new build in the install.

johnbelamaric · 2023-07-20T14:13:55Z

Sorry I forgot to bump that. Will send a PR today

efiacor · 2023-07-20T14:48:34Z

Sorry I forgot to bump that. Will send a PR today

No problem. Thank you sir. I should learn to use cli more in fairness.

johnbelamaric · 2023-07-20T19:46:06Z

nephio-project/nephio-packages#29

* Add resource version checks * Retrieve the updated object before a second update * gofmt * Increase timeout for cleanup wait * Ignore resource-version in test output * Oops * Move annotation handling to the client * Clean up go.mod * Do not swallow errors * Fix return value checking * Update e2e tests * Fix empty string issue with reorderYamlStdout * Revert some unnecessary changes

Add resource version checks

7e51e4a

johnbelamaric requested a review from a team as a code owner June 30, 2023 16:54

johnbelamaric requested a review from justinsb June 30, 2023 16:54

johnbelamaric added 2 commits June 30, 2023 10:40

Retrieve the updated object before a second update

799ac27

gofmt

23a83df

This was referenced Jun 30, 2023

Porch: creation time for package revisions continuously updates #3992

Open

Porch: concurrency issue #3991

Closed

Increase timeout for cleanup wait

649a172

This was referenced Jun 30, 2023

Enable running the e2e tests in parallel groups nephio-project/test-infra#149

Closed

Bump Kubernetes to v1.27.1 in management cluster nephio-project/test-infra#152

Merged

johnbelamaric added 2 commits July 1, 2023 11:29

Ignore resource-version in test output

f613c21

Oops

6c95ec8

johnbelamaric added 5 commits July 5, 2023 13:45

Move annotation handling to the client

b901e3d

Clean up go.mod

77a8b9a

Do not swallow errors

b7a9c33

Fix return value checking

fd2b138

Update e2e tests

95a22ef

Fix empty string issue with reorderYamlStdout

c2f8d01

ChristopherFry mentioned this pull request Jul 7, 2023

Sync upstream changes nephio-project/kpt-backstage-plugins#13

Merged

mortent reviewed Jul 7, 2023

View reviewed changes

Revert some unnecessary changes

05f20ab

mortent approved these changes Jul 7, 2023

View reviewed changes

johnbelamaric merged commit 70c5e27 into kptdev:main Jul 7, 2023
16 checks passed

johnbelamaric mentioned this pull request Jul 8, 2023

porch: additional concurrency tests #4003

Open

This was referenced Apr 8, 2024

porch: additional concurrency tests nephio-project/porch-issue-transfer#9

Closed

porch: additional concurrency tests nephio-project/porch-issue-transfer#50

Closed

porch: additional concurrency tests nephio-project/porch-issue-transfer#135

Closed

liamfallon mentioned this pull request Apr 23, 2024

porch: additional concurrency tests nephio-project/nephio#608

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add resource version checks #3993

Add resource version checks #3993

johnbelamaric commented Jun 30, 2023 •

edited

Loading

johnbelamaric commented Jun 30, 2023

johnbelamaric commented Jun 30, 2023

mortent commented Jul 4, 2023

johnbelamaric commented Jul 5, 2023

johnbelamaric commented Jul 5, 2023

johnbelamaric commented Jul 5, 2023

mortent commented Jul 5, 2023

johnbelamaric commented Jul 5, 2023

johnbelamaric commented Jul 6, 2023

johnbelamaric commented Jul 6, 2023

johnbelamaric commented Jul 6, 2023

mortent left a comment

mortent Jul 7, 2023

johnbelamaric Jul 7, 2023

mortent Jul 7, 2023

johnbelamaric Jul 7, 2023

johnbelamaric Jul 7, 2023

mortent Jul 7, 2023

mortent Jul 7, 2023

johnbelamaric Jul 7, 2023

efiacor commented Jul 20, 2023

johnbelamaric commented Jul 20, 2023

johnbelamaric commented Jul 20, 2023

efiacor commented Jul 20, 2023

johnbelamaric commented Jul 20, 2023

Add resource version checks #3993

Add resource version checks #3993

Conversation

johnbelamaric commented Jun 30, 2023 • edited Loading

johnbelamaric commented Jun 30, 2023

johnbelamaric commented Jun 30, 2023

mortent commented Jul 4, 2023

johnbelamaric commented Jul 5, 2023

johnbelamaric commented Jul 5, 2023

johnbelamaric commented Jul 5, 2023

mortent commented Jul 5, 2023

johnbelamaric commented Jul 5, 2023

johnbelamaric commented Jul 6, 2023

johnbelamaric commented Jul 6, 2023

johnbelamaric commented Jul 6, 2023

mortent left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efiacor commented Jul 20, 2023

johnbelamaric commented Jul 20, 2023

johnbelamaric commented Jul 20, 2023

efiacor commented Jul 20, 2023

johnbelamaric commented Jul 20, 2023

johnbelamaric commented Jun 30, 2023 •

edited

Loading