Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Motivation and Context
We have some use cases where we need to integration test new configurations and deployments with high-volume datasets, but we don't want to fully process the entire dataset (which would cause a 2x increase in cost during the test), so this PR adds a "random choice" condition that can be used to selectively process data.
The intended use of this for integration testing is to pair it with the Drop processor, which would cause the application to randomly drop events from a workflow. To support that, I added the patterns.libsonnet file (this mimics other files we've used internally) -- this file contains pre-configured and commonly used data processing patterns that can be applied across many pipelines. As a bonus, I added a pattern for calculating SHA256 hashes for both plaintext and binary data.
This also includes an update to the Copy processor and jsonnetfmt'ing of the libsonnet files.
How Has This Been Tested?
Tested locally using the file app.
Types of changes
Checklist: