Implements s3 exporter #372
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| //+kubebuilder:default:="" | ||
| // username to connect to server | ||
| AccessKeyID string `json:"accessKeyId"` | ||
|
|
||
| //+kubebuilder:default:="" | ||
| // password to connect to server | ||
| SecretAccessKey string `json:"secretAccessKey"` |
There was a problem hiding this comment.
It would be better to load a secret here but it's useless for now as FLP doesn't.
In result, the credentials would be in clear text in the FLP configmap.
There was a problem hiding this comment.
Well, I think we should read it from a secret/CM anyway: not because it makes it more hidden (you're right that it will be visible in FLP anyway - which is ok as long as FLP is only accessible for cluster admins) , but in a gitops perspective users would like to decouple a config such as FlowCollector CR, and secrets management, for instance by using stuff like Vault: so that the FlowCollector CR can be safely stored in config repos (e.g. in git) without compromising its secrets, and secrets are generated/injected at runtime.
I have a commit for Kafka / SASL that I haven't pushed as a PR yet (I should do that) where there's a similar problem to solve: jotak@17a2ae9 - maybe I should push it and we can align our 2 PRs & reuse some structs / functions
There was a problem hiding this comment.
(edit: rebased my sasl branch, new one is jotak@3a4f105 )
There was a problem hiding this comment.
I was considering replacing these creds by a secret and mounting the secret inside FLP.
Do you feel it's not necessary here ?
There was a problem hiding this comment.
yes it can be done that way but I guess then you need to update FLP to read from file?
That's also what I did for sasl:
jotak@3a4f105#diff-c761638f65b982f9ecf3717616920a5a3c0bb1670fd8ee7e9ab51afc4f166581R578-R593
There was a problem hiding this comment.
yeah I'm fine updating FLP to add this capability if we are aligned
| ) | ||
|
|
||
| // FlowCollectorExporter defines an additional exporter to send enriched flows to. | ||
| type FlowCollectorExporter struct { | ||
| // type selects the type of exporters. The available options are "KAFKA" and "IPFIX". "IPFIX" is <i>unsupported (*)</i>. | ||
| // type selects the type of exporters. The available options are "KAFKA", "IPFIX" and "S3". "IPFIX" is <i>unsupported (*)</i>. |
There was a problem hiding this comment.
I guess S3 would be unsupported too, or is there a request to support it?
| @@ -733,6 +765,10 @@ type FlowCollectorExporter struct { | |||
| // IPFIX configuration, such as the IP address and port to send enriched IPFIX flows to. <i>Unsupported (*)</i>. | |||
| // +optional | |||
| IPFIX FlowCollectorIPFIXReceiver `json:"ipfix,omitempty"` | |||
|
|
|||
| // S3 configuration, such as the endpoint, credentials and bucket name | |||
There was a problem hiding this comment.
Same here: should be said as "unsupported" ?
|
/hold |
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Closing this as it's quite outdated |
This PR implements S3 exporter using the already existing s3 encode stage.
Here is a sample of generated files in the bucket:
sample.txt
Rely on netobserv/flowlogs-pipeline#442