feat: filter events support #3882
Conversation
chandumlg
force-pushed
the
feat.filter-events
branch
from
d0aea5c to
5dc6c7d
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #3882 +/- ##
==========================================
+ Coverage 70.20% 70.22% +0.01%
==========================================
Files 357 357
Lines 53160 53244 +84
==========================================
+ Hits 37321 37388 +67
- Misses 13585 13596 +11
- Partials 2254 2260 +6
☔ View full report in Codecov by Sentry. |
chandumlg
force-pushed
the
feat.filter-events
branch
2 times, most recently
from
50a9ec1 to
04bcd43
Compare
processor/processor.go
Outdated
|
|
||
| var filtered, failed []transformer.TransformerResponse | ||
| for _, event := range response.FailedEvents { | ||
| if event.StatusCode == types.FilterEventCode { |
There was a problem hiding this comment.
Shouldn't the filtered event code be returned by the user transformer as well in case the event is dropped?
IMO to be confident we need integration tests, with an actual transformer, reporting enabled (but not sending reports to a remote server) to verify that filtered events are being recorded in reporting in various stages of the processor's pipeline:
- 1 scenario for message filter
- 1 scenario for destination transformation
- 1 scenario for user transformation
- 1 scenario for tracking plan(?)
cc @pChondros
There was a problem hiding this comment.
Yes. We can use the same filtered event code for dropped events during user transformation as well. When the changes for that are done by the respective team, rudder-server will report them as filtered. (This will be communicated to them. CC @sivashanmukh.) Till then, things will continue to work as is.
chandumlg
force-pushed
the
feat.filter-events
branch
from
04bcd43 to
7ead941
Compare
chandumlg
force-pushed
the
feat.filter-events
branch
from
7ead941 to
16e9ad1
Compare
| @@ -1187,26 +1230,26 @@ func (proc *Handle) getFailedEventJobs(response transformer.Response, commonMeta | |||
| ) | |||
| payload, err := jsonfast.Marshal(messages) | |||
There was a problem hiding this comment.
Any idea why we are storing a single job containing an array of failed payloads to proc_errors? @cisse21?
And we are using this single job to calculate dropped events as well. Wouldn't that cause skewed rsources metrics @Sidddddarth?
| @@ -806,7 +821,7 @@ func (w *worker) sendRouterResponseCountStat(status *jobsdb.JobStatusT, destinat | |||
| destinationTag := misc.GetTagName(destination.ID, destination.Name) | |||
| var alert bool | |||
| alert = w.allowRouterAbortedAlert(errorAt) | |||
| if status.JobState == jobsdb.Succeeded.State { | |||
| if status.JobState == jobsdb.Succeeded.State || status.JobState == jobsdb.Filtered.State { | |||
There was a problem hiding this comment.
Any adaptations needed in sendEventDeliveryStat?
There was a problem hiding this comment.
IMO, No.
router response counts -> responses router received for an event. respStatusCode label will reveal about the status of the delivery.
event delivery stats -> stats for successfully delivered events (count & time spent by the event in the pipeline since it is received.)
chandumlg
force-pushed
the
feat.filter-events
branch
from
a17c58f to
ebbae96
Compare
Description
This pr adds the following features:
If transformer returns a statusCode: 298 after transforming an event, that event will be treated as a filtered event. Job (corresponding to that event) will be marked as filtered in jobsdb and the same state is forwarded to reporting service.
If transformer returns a statusCode: 299 after transforming an event (esp: during router transform), that event will be treated as a successful one and router doesn't need to perform any more actions. Job will be marked as succeeded in jobsdb.
Event filter module in processor, instead of silently filtering the events, returns the filtered events with statusCode: 298. Processor will handle them accordingly.
More details: https://www.notion.so/rudderstacks/Introduce-new-code-b-w-transformer-server-to-filter-out-events-8e8faea134474ac08bd85d63d00eceab
Linear Ticket
https://linear.app/rudderstack/issue/INT-515/implement-server-side-changes-to-accept-new-status-code-from
Security