[WIP] feat: add constructor for full event lineage graph #7921

Cali0707 · 2024-05-13T18:40:52Z

Fixes #7694

Proposed Changes

Get resources from cluster and construct a graph

Signed-off-by: Calum Murray <cmurray@redhat.com>

Cali0707 · 2024-05-13T18:41:02Z

/cc @pierDipi

knative-prow · 2024-05-13T18:41:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Cali0707

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Cali0707]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/graph/constructor.go

Cali0707 · 2024-05-13T18:42:12Z

This is WIP because I want feedback on the approach before adding unit tests

Signed-off-by: Calum Murray <cmurray@redhat.com>

codecov · 2024-05-13T18:57:36Z

Codecov Report

Attention: Patch coverage is 9.67742% with 224 lines in your changes missing coverage. Please review.

Project coverage is 67.04%. Comparing base (2cc0a5b) to head (d60a069).
Report is 5 commits behind head on main.

Files	Patch %	Lines
pkg/graph/constructor.go	10.90%	196 Missing ⚠️
pkg/graph/types.go	0.00%	28 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7921      +/-   ##
==========================================
- Coverage   67.87%   67.04%   -0.83%     
==========================================
  Files         368      368              
  Lines       17551    17777     +226     
==========================================
+ Hits        11912    11918       +6     
- Misses       4894     5113     +219     
- Partials      745      746       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pkg/graph/constructor.go

Signed-off-by: Calum Murray <cmurray@redhat.com>

Cali0707 · 2024-07-23T03:51:43Z

/cc @pierDipi @aliok

aliok · 2024-07-23T08:17:55Z

pkg/graph/constructor.go

+
+	g := NewGraph()
+
+	for _, ns := range config.Namespaces {


In Backstage backend, we fetch brokers for all namespaces. It is a readonly view, so, should be fine.

We might add a config option in Backstage backend to consider specific namespaces.
However, there's still a need to fetch all brokers, all triggers, etc.

That can't be done by fetching all namespaces and then iterating over them though.

We should use brokers, err := clientset.EventingV1().Brokers(metav1.NamespaceAll).List(context.Background(), metav1.ListOptions{}).

Can we add a config option for this?

I was thinking for this you could just provide metav1.NamespaceAll as the only namespace in the config list. Would that work?

I think it would...

We might then need add a validation in the code to check if the list of namespaces contain "metav1.NamespaceAll" and if so, it should not have anything else.

aliok · 2024-07-23T08:19:42Z

pkg/graph/constructor.go

+		if err != nil && !apierrs.IsNotFound(err) && !apierrs.IsUnauthorized(err) && !apierrs.IsForbidden(err) {
+			return nil, err
+		}


We should log with info/warn in case of apierrs.IsUnauthorized(err) or apierrs.IsForbidden(err)

Signed-off-by: Calum Murray <cmurray@redhat.com>

aliok · 2024-07-25T12:40:01Z

pkg/graph/constructor.go

+	ShouldAddBroker       func(b eventingv1.Broker) bool
+	ShouldAddChannel      func(c messagingv1.Channel) bool
+	ShouldAddSource       func(s duckv1.Source) bool
+	ShouldAddTrigger      func(t eventingv1.Trigger) bool
+	ShouldAddSubscription func(s messagingv1.Subscription) bool
+	ShouldAddEventType    func(et eventingv1beta3.EventType) bool


These are nice, but the resources are fetched anyway. If I am not interested in channels, that shouldn't be fetched at all.

Can you create separate entries like fetchBrokers, etc defaulting to true?

aliok · 2024-07-25T12:43:03Z

pkg/graph/constructor.go

+	if err != nil {
+		return nil, fmt.Errorf("unable to list source CRDs: %w", err)
+	}


This returns error in case of 404 (when there are no source CRDs)

Another thing, in case of 404s or Auth issues when fetching resources other than sources, we write a Warn log and continue. It might be nice to be more consistent.

aliok · 2024-07-25T13:00:29Z

pkg/graph/constructor.go

+
+	return g, nil
+}
+
 func (g *Graph) AddBroker(broker eventingv1.Broker) {


Can we actually embed the full broker resource in the graph here? Or make it a config option?

Because, for example in the Backstage backend, if I only have references, I will need to fetch the brokers again. But, we have the full resource here fetched anyway.

aliok · 2024-07-25T14:19:30Z

Kind of irrelevant for this PR, but it would be super nice if we have a stringer for graph and vertices for debugging purposes. A text like

A (broker.eventing.knative.dev) --> B (foo.bar)
...etc

Cali0707 · 2024-07-25T18:43:43Z

/cc @aliok
Thanks for all the feedback!!

Cali0707 · 2024-07-25T18:57:18Z

Kind of irrelevant for this PR, but it would be super nice if we have a stringer for graph and vertices for debugging purposes. A text like
A (broker.eventing.knative.dev) --> B (foo.bar)
...etc

Yeah 100%, I thought about this too but I wasn't sure how to show a vertex with an arbitrary number of edges using just strings...

aliok · 2024-07-26T09:45:47Z

Yeah 100%, I thought about this too but I wasn't sure how to show a vertex with an arbitrary number of edges using just strings...

I added stringer impls for types now. It is ok-ish. Example:

Kind = Broker, Namespace = default, Name = test-broker, APIVersion = eventing.knative.dev/v1, Group = , Address = 
Kind = Broker, Namespace = default, Name = default, APIVersion = eventing.knative.dev/v1, Group = , Address = 
Kind = EventType, Namespace = default, Name = test-event-type, APIVersion = eventing.knative.dev/v1beta3, Group = , Address = 
Out Edges:
        [Kind = EventType, Namespace = default, Name = test-event-type, APIVersion = eventing.knative.dev/v1beta3, Group = , Address = ] --> [Kind = Broker, Namespace = default, Name = test-broker, APIVersion = eventing.knative.dev/v1, Group = , Address = ]

aliok · 2024-07-26T09:52:46Z

@Cali0707

Say we have an EventType:

kind: EventType
metadata:
  name: test-event-type
  namespace: default
spec:
  reference:
    apiVersion: eventing.knative.dev/v1
    kind: Broker
    name: test-broker

Currently the graph is constructed in a way that there is an edge from ET to Broker.

I think it should be the other way around.

Events of EventType is coming out of broker.

cc @pierDipi

aliok · 2024-07-26T12:28:22Z

pkg/graph/constructor.go


 	v.AddEdge(to, dest, NoTransform{}, true)
 }

-func (g *Graph) AddEventType(et *eventingv1beta3.EventType) error {
+func (g *Graph) AddEventType(et eventingv1beta3.EventType) error {
 	ref := &duckv1.KReference{
 		Name:       et.Name,
 		Namespace:  et.Namespace,


Below this line, we have a APIVersion: "eventing.knative.dev/v1beta3".

This v1beta3 information is probably needed for this case for using as a map key. Group (eventing.k.d) should be enough, as there can't be another resource with the same name and group, but in a different version.

Ignoring the version can make things easier. Would we lose important information to be provided to the clients of this library?

cc @pierDipi @Cali0707

My only concern here is that if we get rid of the version, it might make it harder for clients of the library to understand what type to cast the interface{} value we are storing with the object to...

for 3 entries, it was creating an array of 6 Signed-off-by: Ali Ok <aliok@redhat.com>

Signed-off-by: Calum Murray <cmurray@redhat.com>

Signed-off-by: Ali Ok <aliok@redhat.com>

knative-prow · 2024-07-29T12:51:13Z

@Cali0707: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
reconciler-tests_eventing_main	`d60a069`	link	true	`/test reconciler-tests`

Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Cali0707 · 2024-07-29T19:15:52Z

@aliok @pierDipi is there anything left for this PR? Should we merge this and move to some follow up issues/PRs for anything that comes up?

feat: add constructor for full graph

698a50e

Signed-off-by: Calum Murray <cmurray@redhat.com>

knative-prow bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 13, 2024

knative-prow bot requested a review from pierDipi May 13, 2024 18:41

knative-prow bot requested review from aslom and matzew May 13, 2024 18:41

knative-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 13, 2024

Cali0707 commented May 13, 2024

View reviewed changes

pkg/graph/constructor.go Outdated Show resolved Hide resolved

fix: tests build again

582ca21

Signed-off-by: Calum Murray <cmurray@redhat.com>

pierDipi reviewed May 31, 2024

View reviewed changes

pkg/graph/constructor.go Outdated Show resolved Hide resolved

pierDipi reviewed May 31, 2024

View reviewed changes

pkg/graph/constructor.go Outdated Show resolved Hide resolved

fix: handle fetching sources properly

67fefb7

Signed-off-by: Calum Murray <cmurray@redhat.com>

knative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 16, 2024

fix merge conflicts

853b831

Signed-off-by: Calum Murray <cmurray@redhat.com>

knative-prow-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 16, 2024

Cali0707 added 2 commits July 16, 2024 16:23

fix: don't return early for auth errors

ab9870d

Signed-off-by: Calum Murray <cmurray@redhat.com>

feat: added config for graph constructor

43cd8e8

Signed-off-by: Calum Murray <cmurray@redhat.com>

knative-prow bot requested review from aliok and pierDipi July 23, 2024 03:51

aliok reviewed Jul 23, 2024

View reviewed changes

feat: add logging when unauthorized/forbidden to list resources

0173c1a

Signed-off-by: Calum Murray <cmurray@redhat.com>

aliok reviewed Jul 25, 2024

View reviewed changes

knative-prow bot requested a review from aliok July 25, 2024 18:43

aliok reviewed Jul 26, 2024

View reviewed changes

aliok and others added 5 commits July 29, 2024 15:04

Fix vertices fn creating nil pointers

bdc959b

for 3 entries, it was creating an array of 6 Signed-off-by: Ali Ok <aliok@redhat.com>

fix review comments from @aliok

3fb2c40

Signed-off-by: Calum Murray <cmurray@redhat.com>

Add stringer impl to types

64dd3f6

Signed-off-by: Ali Ok <aliok@redhat.com>

No panic if source filter is not specified

b68d839

Signed-off-by: Ali Ok <aliok@redhat.com>

Replace some hardcoded strings

d60a069

Signed-off-by: Ali Ok <aliok@redhat.com>

aliok force-pushed the create-lineage-graph branch from 192021c to d60a069 Compare July 29, 2024 12:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] feat: add constructor for full event lineage graph #7921

[WIP] feat: add constructor for full event lineage graph #7921

Cali0707 commented May 13, 2024

Cali0707 commented May 13, 2024

knative-prow bot commented May 13, 2024

Cali0707 commented May 13, 2024

codecov bot commented May 13, 2024 •

edited

Loading

Cali0707 commented Jul 23, 2024

aliok Jul 23, 2024

Cali0707 Jul 23, 2024

aliok Jul 24, 2024

aliok Jul 23, 2024

aliok Jul 25, 2024

aliok Jul 25, 2024

aliok Jul 25, 2024

aliok Jul 25, 2024

aliok commented Jul 25, 2024

Cali0707 commented Jul 25, 2024

Cali0707 commented Jul 25, 2024

aliok commented Jul 26, 2024

aliok commented Jul 26, 2024

aliok Jul 26, 2024

aliok Jul 26, 2024

Cali0707 Jul 29, 2024

knative-prow bot commented Jul 29, 2024

Cali0707 commented Jul 29, 2024

[WIP] feat: add constructor for full event lineage graph #7921

Are you sure you want to change the base?

[WIP] feat: add constructor for full event lineage graph #7921

Conversation

Cali0707 commented May 13, 2024

Proposed Changes

Cali0707 commented May 13, 2024

knative-prow bot commented May 13, 2024

Cali0707 commented May 13, 2024

codecov bot commented May 13, 2024 • edited Loading

Codecov Report

Cali0707 commented Jul 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aliok commented Jul 25, 2024

Cali0707 commented Jul 25, 2024

Cali0707 commented Jul 25, 2024

aliok commented Jul 26, 2024

aliok commented Jul 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knative-prow bot commented Jul 29, 2024

Cali0707 commented Jul 29, 2024

codecov bot commented May 13, 2024 •

edited

Loading