Why do Objects sometimes have empty `TypeMeta`? #1735

gallettilance · 2021-12-01T19:58:46Z

I built a controller that tracks a resource type called Mykind which creates/updates/deletes Pods.

// SetupWithManager sets up the controller with the Manager.
func (r *MykindReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&grpv1alpha1.Mykind{}).
		Owns(&core.Pod{}).
		WithEventFilter(predicates.MyPredicate()).
		Complete(r)
}

I created a predicate function as follows:

// MyPredicate is an example predicate function to filter events before they are
// handed to the reconciler
func MyPredicate() predicate.Predicate {
	return predicate.Funcs{
		CreateFunc: func(e event.CreateEvent) bool {
			fmt.Printf("Creating: %s\n", e.Object.GetName())
			fmt.Printf("Creating: %v\n", e.Object.GetObjectKind())
			return true
		},
		UpdateFunc: func(e event.UpdateEvent) bool {
			fmt.Printf("Updating: %s | prev generation = %d, new generation = %d\n", e.ObjectOld.GetName(), e.ObjectOld.GetGeneration(), e.ObjectNew.GetGeneration())
			fmt.Printf("Updating: %v\n", e.ObjectOld.GetObjectKind())
			fmt.Printf("Updating: %v\n", e.ObjectNew.GetObjectKind())
			// Ignore updates if metadata.Generation does not change
			return e.ObjectOld.GetGeneration() != e.ObjectNew.GetGeneration()
		},
		DeleteFunc: func(e event.DeleteEvent) bool {
			fmt.Printf("Deleting: %s\n", e.Object.GetName())
			fmt.Printf("Deleting: %v\n", e.Object.GetObjectKind())
			return !e.DeleteStateUnknown
		},
	}
}

Looking at the output logs I noticed that some event objects don't have TypeMeta populated while others do:

go run ./main.go
2021-12-01T12:27:41.090-0500	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-12-01T12:27:41.091-0500	INFO	setup	starting manager
2021-12-01T12:27:41.091-0500	INFO	starting metrics server	{"path": "/metrics"}
2021-12-01T12:27:41.091-0500	INFO	controller.mykind	Starting EventSource	{"reconciler group": "grp.example.com", "reconciler kind": "Mykind", "source": "kind source: /, Kind="}
2021-12-01T12:27:41.091-0500	INFO	controller.mykind	Starting EventSource	{"reconciler group": "grp.example.com", "reconciler kind": "Mykind", "source": "kind source: /, Kind="}
2021-12-01T12:27:41.091-0500	INFO	controller.mykind	Starting Controller	{"reconciler group": "grp.example.com", "reconciler kind": "Mykind"}
Creating: mykind-sample-74jw9
Creating: &TypeMeta{Kind:,APIVersion:,}
Creating: mykind-sample
Creating: coredns-78fcd69978-xgzx5
Creating: &TypeMeta{Kind:Mykind,APIVersion:grp.example.com/v1alpha1,}
Creating: &TypeMeta{Kind:,APIVersion:,}
2021-12-01T12:27:41.192-0500	INFO	controller.mykind	Starting workers	{"reconciler group": "grp.example.com", "reconciler kind": "Mykind", "worker count": 1}
Creating: etcd-minikube
Creating: &TypeMeta{Kind:,APIVersion:,}
Creating: kube-apiserver-minikube
Creating: &TypeMeta{Kind:,APIVersion:,}
Creating: kube-controller-manager-minikube
Creating: &TypeMeta{Kind:,APIVersion:,}
Creating: kube-proxy-mjh92
Creating: &TypeMeta{Kind:,APIVersion:,}
Creating: kube-scheduler-minikube
Creating: &TypeMeta{Kind:,APIVersion:,}
Creating: storage-provisioner
Creating: &TypeMeta{Kind:,APIVersion:,}
2021-12-01T12:27:41.192-0500	INFO	controllers.mykind	request	{"mykind-sample": "default"}
2021-12-01T12:27:41.192-0500	INFO	controllers.mykind	Initializing
2021-12-01T12:27:41.192-0500	INFO	mykind-resource	default	{"name": "mykind-sample"}
2021-12-01T12:27:41.192-0500	INFO	mykind-resource	validate create	{"name": "mykind-sample"}
2021-12-01T12:27:41.192-0500	INFO	controllers.mykind	Ensure Status
2021-12-01T12:27:41.192-0500	INFO	controllers.mykind	ensuring pod status is correct
Updating: mykind-sample | prev generation = 3, new generation = 3
Updating: &TypeMeta{Kind:Mykind,APIVersion:grp.example.com/v1alpha1,}
Updating: &TypeMeta{Kind:,APIVersion:,}

Why is that the case?

Looked through the repos and found a few related issues:

None of which seem to answer why this occurs.

The text was updated successfully, but these errors were encountered:

coderanger · 2021-12-01T22:42:07Z

Previously discussed in #1517 (comment) but that also doesn't really cover the "why".

coderanger · 2021-12-01T22:42:59Z

I think this should be closed as a dup as we've already established we don't plan to implement a workaround here.

gallettilance · 2021-12-02T00:41:58Z

I'm fine with closing this, I'm just trying to understand why this is happening (what makes it populated and what makes it empty). And if there's a way I can get these objects to be consistently one or the other.

Without knowing the above can occur, one might be tempted to do something like this:

UpdateFunc: func(e event.UpdateEvent) bool {
	if e.ObjectNew.GetObjectKind().GroupVersionKind().Kind == "Pod" {
		return true
	}
	return e.ObjectOld.GetGeneration() != e.ObjectNew.GetGeneration()
},

where you ignore status changes for one kind and not the other (which seems reasonable - correct me if that's wrong). Today this will sometimes work and sometimes not which makes it really tough to debug.

coderanger · 2021-12-02T00:56:45Z

Agreed it is an unfortunate footgun :-( I think the only way you're going to get a "why" is to just read all the relevant client-go code, which is a big task.

gallettilance · 2021-12-02T16:00:04Z

FWIW the following gets consistent behavior:

Add Pods to the scheme

utilruntime.Must(apps.AddToScheme(scheme))

Change the predicate function to take a pointer to scheme as a parameter

func MyPredicate(scheme *runtime.Scheme) predicate.Predicate {
	return predicate.Funcs{
		CreateFunc: func(e event.CreateEvent) bool {
			return true
		},
		UpdateFunc: func(e event.UpdateEvent) bool {
			objOld, _ := addGVK(e.ObjectOld, scheme)
			objNew, _ := addGVK(e.ObjectNew, scheme)
                        if objNew.GetObjectKind().GroupVersionKind().Kind == "Pod" {
		        	return true
	                }
			return e.ObjectOld.GetGeneration() != e.ObjectNew.GetGeneration()
		},
		DeleteFunc: func(e event.DeleteEvent) bool {
			return true
		},
	}
}

where addGVK is taken from here

func addGVK(obj client.Object, scheme *runtime.Scheme) (client.Object, error) {
	gvks, _, err := scheme.ObjectKinds(obj)
	if err != nil {
		return obj, fmt.Errorf("missing apiVersion or kind and cannot assign it; %w", err)
	}

	for _, gvk := range gvks {
		if len(gvk.Kind) == 0 {
			continue
		}
		if len(gvk.Version) == 0 || gvk.Version == runtime.APIVersionInternal {
			continue
		}
		obj.GetObjectKind().SetGroupVersionKind(gvk)
		break
	}
	return obj, nil
}

Update the controller set up

func (r *MykindReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&grpv1alpha1.Mykind{}).
		Owns(&core.Pod{}).
		WithEventFilter(predicates.MyPredicate(r.Scheme)).
		Complete(r)
}

Result:

go run ./main.go
2021-12-02T10:59:41.149-0500	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-12-02T10:59:41.149-0500	INFO	setup	starting manager
2021-12-02T10:59:41.150-0500	INFO	starting metrics server	{"path": "/metrics"}
2021-12-02T10:59:41.150-0500	INFO	controller.mykind	Starting EventSource	{"reconciler group": "grp.example.com", "reconciler kind": "Mykind", "source": "kind source: /, Kind="}
2021-12-02T10:59:41.150-0500	INFO	controller.mykind	Starting EventSource	{"reconciler group": "grp.example.com", "reconciler kind": "Mykind", "source": "kind source: /, Kind="}
2021-12-02T10:59:41.150-0500	INFO	controller.mykind	Starting Controller	{"reconciler group": "grp.example.com", "reconciler kind": "Mykind"}
Creating: mykind-sample
2021-12-02T10:59:41.251-0500	INFO	controller.mykind	Starting workers	{"reconciler group": "grp.example.com", "reconciler kind": "Mykind", "worker count": 1}
Creating: &TypeMeta{Kind:Mykind,APIVersion:grp.example.com/v1alpha1,}
Creating: kube-scheduler-minikube
Creating: &TypeMeta{Kind:Pod,APIVersion:v1,}
Creating: storage-provisioner
Creating: &TypeMeta{Kind:Pod,APIVersion:v1,}
2021-12-02T10:59:41.252-0500	INFO	controllers.mykind	request	{"mykind-sample": "default"}
Creating: mykind-sample-74jw9
Creating: &TypeMeta{Kind:Pod,APIVersion:v1,}
2021-12-02T10:59:41.252-0500	INFO	controllers.mykind	Initializing
2021-12-02T10:59:41.252-0500	INFO	mykind-resource	default	{"name": "mykind-sample"}
2021-12-02T10:59:41.252-0500	INFO	mykind-resource	validate create	{"name": "mykind-sample"}
Creating: coredns-78fcd69978-xgzx5
Creating: &TypeMeta{Kind:Pod,APIVersion:v1,}
Creating: etcd-minikube
Creating: &TypeMeta{Kind:Pod,APIVersion:v1,}
Creating: kube-apiserver-minikube
2021-12-02T10:59:41.252-0500	INFO	controllers.mykind	Ensure Status
Creating: &TypeMeta{Kind:Pod,APIVersion:v1,}
2021-12-02T10:59:41.252-0500	INFO	controllers.mykind	ensuring pod status is correct
Creating: kube-controller-manager-minikube
Creating: &TypeMeta{Kind:Pod,APIVersion:v1,}
Creating: kube-proxy-mjh92
Creating: &TypeMeta{Kind:Pod,APIVersion:v1,}
Updating: mykind-sample | prev generation = 3, new generation = 3
Updating: &TypeMeta{Kind:Mykind,APIVersion:grp.example.com/v1alpha1,}
Updating: &TypeMeta{Kind:Mykind,APIVersion:grp.example.com/v1alpha1,}
2021-12-02T10:59:41.263-0500	INFO	controllers.mykind	request	{"mykind-sample": "default"}
2021-12-02T10:59:41.264-0500	INFO	controllers.mykind	Initializing
2021-12-02T10:59:41.264-0500	INFO	mykind-resource	default	{"name": "mykind-sample"}
2021-12-02T10:59:41.264-0500	INFO	mykind-resource	validate create	{"name": "mykind-sample"}
2021-12-02T10:59:41.264-0500	INFO	controllers.mykind	Ensure Status
2021-12-02T10:59:41.264-0500	INFO	controllers.mykind	ensuring pod status is correct

gallettilance · 2021-12-02T16:53:21Z

Actually you can do even simpler:

if _, ok := e.Object.(*core.Pod); !ok {
	fmt.Printf("Not Pod\n")
} else {
	fmt.Printf("Pod\n")
}

gallettilance closed this as completed Dec 2, 2021

This was referenced Jun 28, 2022

Make resources return non-empty resource kind sky-uk/kfp-operator#136

Merged

Fetched resources do not contain GroupVersionKind information sky-uk/kfp-operator#137

Open

moolen mentioned this issue Oct 14, 2022

Added namespace condition to ClusterSecretStore external-secrets/external-secrets#1635

Merged

Karthik-K-N mentioned this issue Jul 30, 2024

🌱 Log controller name in predicates kubernetes-sigs/cluster-api#10959

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Why do Objects sometimes have empty `TypeMeta`? #1735

Why do Objects sometimes have empty `TypeMeta`? #1735

gallettilance commented Dec 1, 2021 •

edited

Loading

coderanger commented Dec 1, 2021

coderanger commented Dec 1, 2021

gallettilance commented Dec 2, 2021 •

edited

Loading

coderanger commented Dec 2, 2021

gallettilance commented Dec 2, 2021

gallettilance commented Dec 2, 2021

Why do Objects sometimes have empty TypeMeta? #1735

Why do Objects sometimes have empty TypeMeta? #1735

Comments

gallettilance commented Dec 1, 2021 • edited Loading

coderanger commented Dec 1, 2021

coderanger commented Dec 1, 2021

gallettilance commented Dec 2, 2021 • edited Loading

coderanger commented Dec 2, 2021

gallettilance commented Dec 2, 2021

gallettilance commented Dec 2, 2021

Why do Objects sometimes have empty `TypeMeta`? #1735

Why do Objects sometimes have empty `TypeMeta`? #1735

gallettilance commented Dec 1, 2021 •

edited

Loading

gallettilance commented Dec 2, 2021 •

edited

Loading