Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Random Condition #18

Merged
merged 5 commits into from
Aug 29, 2022
Merged

feat: Add Random Condition #18

merged 5 commits into from
Aug 29, 2022

Conversation

jshlbrd
Copy link
Collaborator

@jshlbrd jshlbrd commented Aug 26, 2022

Description

  • Adds a Random choice condition
  • Adds a patterns.libsonnet file
  • Removes raw metadata support from the Copy processor

Motivation and Context

We have some use cases where we need to integration test new configurations and deployments with high-volume datasets, but we don't want to fully process the entire dataset (which would cause a 2x increase in cost during the test), so this PR adds a "random choice" condition that can be used to selectively process data.

The intended use of this for integration testing is to pair it with the Drop processor, which would cause the application to randomly drop events from a workflow. To support that, I added the patterns.libsonnet file (this mimics other files we've used internally) -- this file contains pre-configured and commonly used data processing patterns that can be applied across many pipelines. As a bonus, I added a pattern for calculating SHA256 hashes for both plaintext and binary data.

This also includes an update to the Copy processor and jsonnetfmt'ing of the libsonnet files.

How Has This Been Tested?

Tested locally using the file app.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@jshlbrd jshlbrd marked this pull request as ready for review August 26, 2022 21:32
@jshlbrd jshlbrd requested a review from a team as a code owner August 26, 2022 21:32
Copy link
Contributor

@julieagnessparks julieagnessparks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jshlbrd jshlbrd merged commit 302f24a into main Aug 29, 2022
@jshlbrd jshlbrd deleted the jshlbrd/random-condition branch August 29, 2022 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants