aws s3 exporter second PR, implementation #10000
Conversation
|
|
pdelewski
changed the title
pdelewski
force-pushed
the
s3exporter-2
branch
8 times, most recently
from
1cf1f74
to
15a612e
Compare
pdelewski
force-pushed
the
s3exporter-2
branch
7 times, most recently
from
1e6f853
to
3ddf560
Compare
pdelewski
force-pushed
the
s3exporter-2
branch
7 times, most recently
from
208d0b3
to
bfe1d9b
Compare
|
can we get this merged in? |
|
and how could I test this out locally? |
first, it needs to be reviewed |
exporter/awss3exporter/s3_writer.go
Outdated
| year, month, day := time.Date() | ||
| hour, minute, _ := time.Clock() | ||
|
|
||
| rand.Int() |
There was a problem hiding this comment.
Not sure what's the purpose of this line
| @@ -0,0 +1,31 @@ | |||
| # AWS S3 Exporter for OpenTelemetry Collector | |||
There was a problem hiding this comment.
I think some discussion (or example) on partitioning could be helpful here
exporter/awss3exporter/s3_writer.go
Outdated
|
|
||
| func getS3Key(time time.Time, bucket string, keyPrefix string, partition string, filePrefix string, metadata string, fileformat string) string { | ||
| timeKey := getTimeKey(time, partition) | ||
| randomID := rand.Int() |
There was a problem hiding this comment.
Not sure if that matters much, but perhaps it would be nice to have padded number (so always the same number of characters). Also, some encoding (even base64) could be used to shorten the random part. S3 has max object length of 1024 so we should be well within the limit, but, just in case, might be worth considering
|
@pdelewski is it possible to provide multiple buckets to export to? one of the reasons for this is because we normally provide a backup bucket for logs in case there is an outage with one region, we have another region & bucket to drop logs into |
@thomasbaldwin Sure, but this requires configuration refactoring/extension. I will think about it. |
sounds good @pdelewski, thanks for all your work on this. And regarding the path name, we generally have all our logging structured in such a way that the root folder name is usually "AWSLogs" within the bucket. Not a blocker by any means but would be nice to be able to control that |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
Can we merge this? I’m using this on staging environments and it runs without issue |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
| metric/year=XXXX/month=XX/day=XX/hour=XX/minute=XX | ||
| ``` | ||
|
|
||
| ## AWS Credential Configuration |
There was a problem hiding this comment.
About AWS Credentials config, i think need more example
|
Given the conflicts, I have opened #20912 instead to follow up based on the work done in this PR. Closing this PR for good. |
Description:
awss3exporter exports logs and traces
Link to tracking Issue:
#2835
Testing:
Unit tests covering exporter code
Documentation:
Readme describing exporter functionality and configuration.