-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
S3 destination: updating specs regarding certification #11917
Merged
sashaNeshcheret
merged 11 commits into
master
from
omneshcheret/11835-s3-destination-spec-update
May 3, 2022
Merged
Changes from 3 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
3baf426
S3 destination: updating specs regarding certification
sashaNeshcheret 6bb7afe
S3 destination: updating required fields
sashaNeshcheret 8980a0e
S3 destination: updating required fields
sashaNeshcheret ef28540
Apply suggestions from code review
sashaNeshcheret 7151d5e
Merge remote-tracking branch 'origin/master' into omneshcheret/11835-…
sashaNeshcheret 8d4e61d
Merge remote-tracking branch 'origin/omneshcheret/11835-s3-destinatio…
sashaNeshcheret bc20abb
S3 destination: updating specs with links
sashaNeshcheret de4aa79
S3 destination: updating specs with links
sashaNeshcheret a1e7220
Apply suggestions from code review
sashaNeshcheret 97bd0ce
S3 destination: added links to the fields
sashaNeshcheret cffca0b
S3 destination: updating specs with links
sashaNeshcheret File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,33 +16,34 @@ | |
], | ||
"additionalProperties": false, | ||
"properties": { | ||
"s3_endpoint": { | ||
"title": "Endpoint", | ||
"access_key_id": { | ||
"type": "string", | ||
"default": "", | ||
"description": "This is your S3 endpoint url.(if you are working with AWS S3, just leave empty).", | ||
"examples": ["http://localhost:9000"], | ||
"description": "The access key id to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket.", | ||
sashaNeshcheret marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"title": "S3 Key Id *", | ||
sashaNeshcheret marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"airbyte_secret": true, | ||
"examples": ["A012345678910EXAMPLE"], | ||
"order": 0 | ||
}, | ||
"secret_access_key": { | ||
"type": "string", | ||
"description": "The corresponding secret to the access key id", | ||
sashaNeshcheret marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"title": "S3 Access Key *", | ||
"airbyte_secret": true, | ||
"examples": ["a012345678910ABCDEFGH/AbCdEfGhEXAMPLEKEY"], | ||
"order": 1 | ||
}, | ||
"s3_bucket_name": { | ||
"title": "S3 Bucket Name", | ||
"type": "string", | ||
"description": "The name of the S3 bucket.", | ||
"examples": ["airbyte_sync"], | ||
"order": 1 | ||
"order": 2 | ||
}, | ||
"s3_bucket_path": { | ||
"title": "S3 Bucket Path", | ||
"description": "Directory under the S3 bucket where data will be written.", | ||
"type": "string", | ||
"examples": ["data_sync/test"], | ||
"order": 2 | ||
}, | ||
"s3_path_format": { | ||
"description": "Format string on how data will be organized inside the S3 bucket directory", | ||
"type": "string", | ||
"examples": [ | ||
"${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_" | ||
], | ||
"order": 3 | ||
}, | ||
"s3_bucket_region": { | ||
|
@@ -80,24 +81,8 @@ | |
], | ||
"order": 4 | ||
}, | ||
"access_key_id": { | ||
"type": "string", | ||
"description": "The access key id to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket, if not set, Airbyte will rely on Instance Profile.", | ||
"title": "S3 Key Id", | ||
"airbyte_secret": true, | ||
"examples": ["A012345678910EXAMPLE"], | ||
"order": 5 | ||
}, | ||
"secret_access_key": { | ||
"type": "string", | ||
"description": "The corresponding secret to the access key id, if S3 Key Id is set, then S3 Access Key must also be provided", | ||
"title": "S3 Access Key", | ||
"airbyte_secret": true, | ||
"examples": ["a012345678910ABCDEFGH/AbCdEfGhEXAMPLEKEY"], | ||
"order": 6 | ||
}, | ||
"format": { | ||
"title": "Output Format", | ||
"title": "Output Format *", | ||
"type": "object", | ||
"description": "Output data format", | ||
"oneOf": [ | ||
|
@@ -106,12 +91,14 @@ | |
"required": ["format_type", "compression_codec"], | ||
"properties": { | ||
"format_type": { | ||
"title": "Format Type *", | ||
"type": "string", | ||
"enum": ["Avro"], | ||
"default": "Avro" | ||
"default": "Avro", | ||
"order": 0 | ||
}, | ||
"compression_codec": { | ||
"title": "Compression Codec", | ||
"title": "Compression Codec *", | ||
"description": "The compression algorithm used to compress data. Default to no compression.", | ||
"type": "object", | ||
"oneOf": [ | ||
|
@@ -211,14 +198,16 @@ | |
} | ||
} | ||
} | ||
] | ||
], | ||
"order": 1 | ||
}, | ||
"part_size_mb": { | ||
"title": "Block Size (MB) for Amazon S3 multipart upload", | ||
"title": "dddBlock Size (MB) for Amazon S3 multipart upload (Optional)", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think you added some random 'd's to this description? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems yes, it was attempt to recognize whether specs is reloaded. |
||
"description": "This is the size of a \"Part\" being buffered in memory. It limits the memory usage when writing. Larger values will allow to upload a bigger files and improve the speed, but consumes9 more memory. Allowed values: min=5MB, max=525MB Default: 5MB.", | ||
"type": "integer", | ||
"default": 5, | ||
"examples": [5] | ||
"examples": [5], | ||
"order": 2 | ||
} | ||
} | ||
}, | ||
|
@@ -227,6 +216,7 @@ | |
"required": ["format_type", "flattening"], | ||
"properties": { | ||
"format_type": { | ||
"title": "Format Type *", | ||
"type": "string", | ||
"enum": ["CSV"], | ||
"default": "CSV" | ||
|
@@ -239,7 +229,7 @@ | |
"enum": ["No flattening", "Root level flattening"] | ||
}, | ||
"part_size_mb": { | ||
"title": "Block Size (MB) for Amazon S3 multipart upload", | ||
"title": "Block Size (MB) for Amazon S3 multipart upload (Optional)", | ||
"description": "This is the size of a \"Part\" being buffered in memory. It limits the memory usage when writing. Larger values will allow to upload a bigger files and improve the speed, but consumes9 more memory. Allowed values: min=5MB, max=525MB Default: 5MB.", | ||
"type": "integer", | ||
"default": 5, | ||
|
@@ -252,12 +242,13 @@ | |
"required": ["format_type"], | ||
"properties": { | ||
"format_type": { | ||
"title": "Format Type *", | ||
"type": "string", | ||
"enum": ["JSONL"], | ||
"default": "JSONL" | ||
}, | ||
"part_size_mb": { | ||
"title": "Block Size (MB) for Amazon S3 multipart upload", | ||
"title": "Block Size (MB) for Amazon S3 multipart upload (Optional)", | ||
"description": "This is the size of a \"Part\" being buffered in memory. It limits the memory usage when writing. Larger values will allow to upload a bigger files and improve the speed, but consumes9 more memory. Allowed values: min=5MB, max=525MB Default: 5MB.", | ||
"type": "integer", | ||
"default": 5, | ||
|
@@ -270,12 +261,13 @@ | |
"required": ["format_type"], | ||
"properties": { | ||
"format_type": { | ||
"title": "Format Type *", | ||
"type": "string", | ||
"enum": ["Parquet"], | ||
"default": "Parquet" | ||
}, | ||
"compression_codec": { | ||
"title": "Compression Codec", | ||
"title": "Compression Codec (Optional)", | ||
"description": "The compression algorithm used to compress data pages.", | ||
"type": "string", | ||
"enum": [ | ||
|
@@ -290,42 +282,59 @@ | |
"default": "UNCOMPRESSED" | ||
}, | ||
"block_size_mb": { | ||
"title": "Block Size (Row Group Size) (MB)", | ||
"title": "Block Size (Row Group Size) (MB) (Optional)", | ||
"description": "This is the size of a row group being buffered in memory. It limits the memory usage when writing. Larger values will improve the IO when reading, but consume more memory when writing. Default: 128 MB.", | ||
"type": "integer", | ||
"default": 128, | ||
"examples": [128] | ||
}, | ||
"max_padding_size_mb": { | ||
"title": "Max Padding Size (MB)", | ||
"title": "Max Padding Size (MB) (Optional)", | ||
"description": "Maximum size allowed as padding to align row groups. This is also the minimum size of a row group. Default: 8 MB.", | ||
"type": "integer", | ||
"default": 8, | ||
"examples": [8] | ||
}, | ||
"page_size_kb": { | ||
"title": "Page Size (KB)", | ||
"title": "Page Size (KB) (Optional)", | ||
"description": "The page size is for compression. A block is composed of pages. A page is the smallest unit that must be read fully to access a single record. If this value is too small, the compression will deteriorate. Default: 1024 KB.", | ||
"type": "integer", | ||
"default": 1024, | ||
"examples": [1024] | ||
}, | ||
"dictionary_page_size_kb": { | ||
"title": "Dictionary Page Size (KB)", | ||
"title": "Dictionary Page Size (KB) (Optional)", | ||
"description": "There is one dictionary page per column per row group when dictionary encoding is used. The dictionary page size works like the page size but for dictionary. Default: 1024 KB.", | ||
"type": "integer", | ||
"default": 1024, | ||
"examples": [1024] | ||
}, | ||
"dictionary_encoding": { | ||
"title": "Dictionary Encoding", | ||
"title": "Dictionary Encoding (Optional)", | ||
"description": "Default: true.", | ||
"type": "boolean", | ||
"default": true | ||
} | ||
} | ||
} | ||
], | ||
"order": 5 | ||
}, | ||
"s3_endpoint": { | ||
"title": "Endpoint (Optional)", | ||
"type": "string", | ||
"default": "", | ||
"description": "This is your S3 endpoint url.(if you are working with AWS S3, just leave empty).", | ||
sashaNeshcheret marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"examples": ["http://localhost:9000"], | ||
"order": 6 | ||
}, | ||
"s3_path_format": { | ||
"title": "s3_path_format (Optional)", | ||
sashaNeshcheret marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"description": "Format string on how data will be organized inside the S3 bucket directory", | ||
"type": "string", | ||
"examples": [ | ||
"${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_" | ||
], | ||
"order": 7 | ||
} | ||
} | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For all of these fields, where are the relevant docs linked so a user can find more information as part of this checklist item?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added links for required and optional first level fields.