-
Notifications
You must be signed in to change notification settings - Fork 14
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(transform): Add Storage Class Support to AWS S3 (#189)
* feat(transform): Add Storage Class Support to S3 * chore: Add Check for Valid Storage Class * docs: Add Example for Glacier IR
- Loading branch information
Showing
6 changed files
with
70 additions
and
25 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
23 changes: 23 additions & 0 deletions
23
examples/config/transform/send/aws_s3_glacier/config.jsonnet
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
// This example configures a storage class for the AWS S3 destination transform. | ||
// The Glacier Instant Retrieval class is recommended for archival data that is | ||
// compatible with Substation's serverless architecture; this class can be read | ||
// directly by a Lambda function triggered by an SNS notification. | ||
local sub = import '../../../../../build/config/substation.libsonnet'; | ||
|
||
{ | ||
concurrency: 1, | ||
transforms: [ | ||
sub.tf.send.aws.s3({ | ||
// Glacier Instant Retrieval charges a minimum of 128KB per object, otherwise | ||
// the other values are set to impossibly high values to ensure all events are | ||
// written to the same file. | ||
batch: { size: 128 * 1000, count: 1000 * 1000, duration: '60m' }, | ||
bucket_name: 'substation', | ||
storage_class: 'GLACIER_IR', // Glacier Instant Retrieval. | ||
// S3 objects are organized by time to the nearest hour and have a UUID filename. | ||
file_path: { time_format: '2006/01/02/15', uuid: true, suffix: '.jsonl.gz' }, | ||
// This example formats the data as JSONL and compresses it with Gzip. | ||
aux_tforms: sub.pattern.tf.fmt.jsonl + [sub.tf.fmt.to.gzip()], | ||
}), | ||
], | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters