feat: Adding pubsub interface #2538

JaydipGabani · 2023-01-25T23:32:23Z

Signed-off-by: Jaydip Gabani gabanijaydip@gmail.com

What this PR does / why we need it: This PR introduces a way to export violations and provides a way to consume all the violations without scanning the logs. For more details, please refer to this doc

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):
Fixes #

Special notes for your reviewer:

JaydipGabani · 2023-01-25T23:38:58Z

Open questions:

Should we include a way for us to batch the messages in the interface itself, or rely on the batching mechanism of individual underlying tools?
How to implement per-request routing based on the relevant queues, for different enforcement actions?

codecov-commenter · 2023-01-25T23:40:39Z

Codecov Report

Patch coverage: 16.51% and project coverage change: -0.79 ⚠️

Comparison is base (a5ebec1) 53.46% compared to head (17e0271) 52.68%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2538      +/-   ##
==========================================
- Coverage   53.46%   52.68%   -0.79%     
==========================================
  Files         127      131       +4     
  Lines       11373    11591     +218     
==========================================
+ Hits         6081     6107      +26     
- Misses       4817     5005     +188     
- Partials      475      479       +4

Flag	Coverage Δ
unittests	`52.68% <16.51%> (-0.79%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/audit/controller.go	`0.00% <ø> (ø)`
pkg/audit/manager.go	`9.64% <0.00%> (-0.41%)`	⬇️
pkg/pubsub/system.go	`3.92% <3.92%> (ø)`
pkg/controller/pubsub/pubsub_config_controller.go	`11.68% <11.68%> (ø)`
pkg/pubsub/dapr/dapr.go	`23.80% <23.80%> (ø)`
pkg/pubsub/provider/provider.go	`100.00% <100.00%> (ø)`

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

pkg/pubsub/pubsub_client.go

JaydipGabani · 2023-01-26T00:06:38Z

Here is a PR I came across that defined method definition for batching in dapr. This is a complete proposal for batching in dapr pub sub

JaydipGabani · 2023-01-27T23:30:01Z

@maxsmythe according to this PPT, it seems I was wrong and at least with dapr, the subscriber will have a choice of bulk subscribe or simple subscribe (i.e how the subscriber can get messages).

maxsmythe · 2023-01-28T00:49:02Z

It looks like Dapr's response to the possibility of some clients not supporting bulk publishing is to have their library turn a bulk publish into a loop over the non-supporting client's publish call.

JaydipGabani · 2023-01-30T20:20:08Z

Yes! According to this comment, non-supporting client's publish call can be made either parallelly or serially (by default it will be made parallelly

pkg/pubsub/pubsub_client.go

pkg/pubsub/rabbitmq/rabbitmq.go

pkg/pubsub/pubsub_client.go

pkg/pubsub/dapr/dapr.go

davis-haba · 2023-02-08T22:50:48Z

We want to add this to the validation webhook too, right? Or did you plan to do this as a follow up?

JaydipGabani · 2023-02-08T23:39:22Z

We want to add this to the validation webhook too, right? Or did you plan to do this as a follow up?

Initially we will enable this with audit and then, later on, can integrate to different parts such as validation webhook.

JaydipGabani · 2023-02-09T02:06:27Z

@sozercan @maxsmythe @nilekhc @ritazh I am working on fixing CI errors, but this PR is ready for review

maxsmythe

Sorry for the delay, added a few comments.

pkg/pubsub/pubsub_client.go

pkg/pubsub/dapr/dapr.go

pkg/pubsub/pubsub_client.go

pkg/pubsub/pubsub_init.go

pkg/pubsub/pubsub_client.go

pkg/pubsub/dapr/dapr.go

pkg/pubsub/pubsub_init.go

pkg/pubsub/client/pubsub_client.go

pkg/audit/manager.go

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

…ror reporting on publish Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

maxsmythe

Code LGTM, last nit is on the Helm chart

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

charts/gatekeeper/values.yaml

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

JaydipGabani · 2023-05-24T18:51:10Z

@maxsmythe @sozercan made the change, PTAL.

maxsmythe

LGTM!

Co-authored-by: Sertaç Özercan <852750+sozercan@users.noreply.github.com>

JaydipGabani commented Jan 25, 2023

View reviewed changes

pkg/pubsub/pubsub_client.go Outdated Show resolved Hide resolved

JaydipGabani force-pushed the pubsub branch from 9afc217 to 5103cf0 Compare February 3, 2023 01:52

JaydipGabani commented Feb 3, 2023

View reviewed changes

pkg/pubsub/pubsub_client.go Outdated Show resolved Hide resolved

JaydipGabani commented Feb 3, 2023

View reviewed changes

pkg/pubsub/rabbitmq/rabbitmq.go Outdated Show resolved Hide resolved

pkg/pubsub/pubsub_client.go Outdated Show resolved Hide resolved

JaydipGabani commented Feb 6, 2023

View reviewed changes

pkg/pubsub/pubsub_client.go Outdated Show resolved Hide resolved

JaydipGabani force-pushed the pubsub branch 2 times, most recently from d200ae9 to 1440f84 Compare February 8, 2023 19:21

JaydipGabani commented Feb 8, 2023

View reviewed changes

pkg/pubsub/dapr/dapr.go Outdated Show resolved Hide resolved

JaydipGabani marked this pull request as ready for review February 9, 2023 01:32

JaydipGabani force-pushed the pubsub branch from 9e9b4b3 to 5aa53bf Compare February 9, 2023 19:44

sozercan requested review from maxsmythe, ritazh and sozercan February 9, 2023 23:09

maxsmythe reviewed Feb 11, 2023

View reviewed changes

JaydipGabani force-pushed the pubsub branch 2 times, most recently from c92a982 to 6320696 Compare February 14, 2023 23:25

maxsmythe reviewed Feb 15, 2023

View reviewed changes

JaydipGabani force-pushed the pubsub branch from 7518c38 to e581384 Compare February 28, 2023 02:44

JaydipGabani commented Feb 28, 2023

View reviewed changes

pkg/audit/manager.go Outdated Show resolved Hide resolved

JaydipGabani removed request for sozercan and ritazh February 28, 2023 02:46

JaydipGabani added 4 commits May 18, 2023 21:44

renaming dummy to fake-subscriber

f8a60f6

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

removing common package, checking pubsub status before publishing, er…

f3ddfb9

…ror reporting on publish Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

creating annotations in tmp dir

76af8d0

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

Merge branch 'pubsub' of github.com:JaydipGabani/gatekeeper into pubsub

9be106a

JaydipGabani requested a review from maxsmythe May 18, 2023 21:44

maxsmythe reviewed May 19, 2023

View reviewed changes

JaydipGabani added 2 commits May 19, 2023 23:39

renaming pubsub-annotation to audit annotations

561839a

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

Merge branch 'master' into pubsub

17e0271

sozercan reviewed May 20, 2023

View reviewed changes

charts/gatekeeper/values.yaml Outdated Show resolved Hide resolved

JaydipGabani added 3 commits May 22, 2023 17:31

changing back auditPodAnnotations to podAnnotations

2cf8b7c

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

Merge branch 'master' into pubsub

6bf81ca

Merge branch 'pubsub' of github.com:JaydipGabani/gatekeeper into pubsub

43acf63

JaydipGabani requested a review from maxsmythe May 22, 2023 17:33

JaydipGabani added 2 commits May 24, 2023 18:49

nit: helm variable audit pod annotations

a09ff0e

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com>

Merge branch 'master' into pubsub

1319b9e

maxsmythe approved these changes May 25, 2023

View reviewed changes

sozercan merged commit f485634 into open-policy-agent:master May 25, 2023

JaydipGabani deleted the pubsub branch May 30, 2023 18:19

acpana mentioned this pull request May 31, 2023

Update gatekeeper.yml to use go version 1.20 open-policy-agent/frameworks#333

Merged

JaydipGabani mentioned this pull request Jun 1, 2023

docs: adding doc for pubsub #2808

Merged

Hy3n4 pushed a commit to Hy3n4/gatekeeper that referenced this pull request Jun 30, 2023

feat: Adding pubsub interface (open-policy-agent#2538)

ee683de

Co-authored-by: Sertaç Özercan <852750+sozercan@users.noreply.github.com>

ralgozino mentioned this pull request Jul 26, 2023

No any logs of violations sighupio/gatekeeper-policy-manager#433

Open

This was referenced Nov 22, 2023

Master 314 stolostron/gatekeeper#185

Closed

Replace master with 3.14 stolostron/gatekeeper#189

Closed

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Adding pubsub interface #2538

feat: Adding pubsub interface #2538

JaydipGabani commented Jan 25, 2023 •

edited

Loading

JaydipGabani commented Jan 25, 2023

codecov-commenter commented Jan 25, 2023 •

edited

Loading

JaydipGabani commented Jan 26, 2023

JaydipGabani commented Jan 27, 2023

maxsmythe commented Jan 28, 2023

JaydipGabani commented Jan 30, 2023

davis-haba commented Feb 8, 2023

JaydipGabani commented Feb 8, 2023

JaydipGabani commented Feb 9, 2023 •

edited

Loading

maxsmythe left a comment

maxsmythe left a comment

JaydipGabani commented May 24, 2023

maxsmythe left a comment

feat: Adding pubsub interface #2538

feat: Adding pubsub interface #2538

Conversation

JaydipGabani commented Jan 25, 2023 • edited Loading

JaydipGabani commented Jan 25, 2023

codecov-commenter commented Jan 25, 2023 • edited Loading

Codecov Report

JaydipGabani commented Jan 26, 2023

JaydipGabani commented Jan 27, 2023

maxsmythe commented Jan 28, 2023

JaydipGabani commented Jan 30, 2023

davis-haba commented Feb 8, 2023

JaydipGabani commented Feb 8, 2023

JaydipGabani commented Feb 9, 2023 • edited Loading

maxsmythe left a comment

Choose a reason for hiding this comment

maxsmythe left a comment

Choose a reason for hiding this comment

JaydipGabani commented May 24, 2023

maxsmythe left a comment

Choose a reason for hiding this comment

JaydipGabani commented Jan 25, 2023 •

edited

Loading

codecov-commenter commented Jan 25, 2023 •

edited

Loading

JaydipGabani commented Feb 9, 2023 •

edited

Loading