feat(aws_cloudwatch_logs): Add retention policy for cloudwatch logs sink #15955

dizlv · 2023-01-13T20:02:22Z

Solves #12038

…te for updating retention policy

netlify · 2023-01-13T20:02:27Z

✅ Deploy Preview for vector-project ready!

Name	Link
🔨 Latest commit	`2836c14`
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/63c5c7cd359aa0000923239a
😎 Deploy Preview	https://deploy-preview-15955--vector-project.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2023-01-13T20:02:35Z

✅ Deploy Preview for vrl-playground canceled.

Name	Link
🔨 Latest commit	`2836c14`
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/63c5c7cd7078740008b795d9

dizlv

Add tests
Add validation for days field
Add documentation examples

dizlv · 2023-01-13T20:03:32Z

src/sinks/aws_cloudwatch_logs/config.rs

+    pub enabled: bool,
+
+    #[configurable(derived)]
+    pub days: i32,


Add validation as this parameter supports only limited values set https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutRetentionPolicy.html#API_PutRetentionPolicy_RequestSyntax

github-actions · 2023-01-13T20:48:52Z

Regression Test Results

Run ID: b17b2389-fd35-4020-9473-06c5e2a60f35
Baseline: 6d05e13
Comparison: 097f499
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

Changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%:

experiment	Δ mean	Δ mean %	confidence
http_text_to_http_json	1.22MiB/CPU-s	5.05	100.00%

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_text_to_http_json	1.22MiB/CPU-s	5.05	100.00%	24.16MiB/CPU-s	898.54KiB/CPU-s	11.6KiB/CPU-s	0.036322	25.38MiB/CPU-s	926.21KiB/CPU-s	11.95KiB/CPU-s	0.035642	False	False
datadog_agent_remap_blackhole	1.04MiB/CPU-s	3.48	100.00%	29.96MiB/CPU-s	1.54MiB/CPU-s	20.42KiB/CPU-s	0.051552	31.01MiB/CPU-s	1.24MiB/CPU-s	16.45KiB/CPU-s	0.040137	False	False
datadog_agent_remap_datadog_logs_acks	904.89KiB/CPU-s	2.62	100.00%	33.69MiB/CPU-s	1.38MiB/CPU-s	18.29KiB/CPU-s	0.0411	34.57MiB/CPU-s	901.15KiB/CPU-s	11.63KiB/CPU-s	0.025455	False	False
splunk_hec_route_s3	262.15KiB/CPU-s	2.23	100.00%	11.5MiB/CPU-s	588.89KiB/CPU-s	7.6KiB/CPU-s	0.05	11.76MiB/CPU-s	590.03KiB/CPU-s	7.61KiB/CPU-s	0.049007	False	False
syslog_splunk_hec_logs	164.57KiB/CPU-s	1.82	100.00%	8.85MiB/CPU-s	318.13KiB/CPU-s	4.11KiB/CPU-s	0.03512	9.01MiB/CPU-s	224.65KiB/CPU-s	2.9KiB/CPU-s	0.024357	False	False
socket_to_socket_blackhole	184.49KiB/CPU-s	1.36	100.00%	13.28MiB/CPU-s	457.14KiB/CPU-s	5.9KiB/CPU-s	0.03361	13.46MiB/CPU-s	167.67KiB/CPU-s	2.17KiB/CPU-s	0.012163	False	False
datadog_agent_remap_datadog_logs	366.16KiB/CPU-s	1.06	100.00%	33.64MiB/CPU-s	1.67MiB/CPU-s	22.12KiB/CPU-s	0.049778	34.0MiB/CPU-s	1.86MiB/CPU-s	24.53KiB/CPU-s	0.054609	False	False
syslog_humio_logs	96.83KiB/CPU-s	1.05	100.00%	9.03MiB/CPU-s	261.8KiB/CPU-s	3.38KiB/CPU-s	0.028318	9.12MiB/CPU-s	151.52KiB/CPU-s	1.96KiB/CPU-s	0.016219	False	False
http_to_http_acks	18.15KiB/CPU-s	0.34	28.08%	5.25MiB/CPU-s	2.71MiB/CPU-s	35.83KiB/CPU-s	0.516818	5.26MiB/CPU-s	2.69MiB/CPU-s	35.57KiB/CPU-s	0.511451	True	False
enterprise_http_to_http	6.01KiB/CPU-s	0.04	77.55%	13.61MiB/CPU-s	308.48KiB/CPU-s	3.98KiB/CPU-s	0.022126	13.62MiB/CPU-s	227.84KiB/CPU-s	2.94KiB/CPU-s	0.016335	False	False
http_to_http_noack	6.27KiB/CPU-s	0.04	75.69%	13.61MiB/CPU-s	330.61KiB/CPU-s	4.27KiB/CPU-s	0.023717	13.62MiB/CPU-s	253.05KiB/CPU-s	3.27KiB/CPU-s	0.018145	False	False
splunk_hec_to_splunk_hec_logs_acks	3.67KiB/CPU-s	0.03	42.59%	13.61MiB/CPU-s	370.17KiB/CPU-s	4.78KiB/CPU-s	0.02655	13.62MiB/CPU-s	344.64KiB/CPU-s	4.45KiB/CPU-s	0.024712	False	False
splunk_hec_to_splunk_hec_logs_noack	4.69KiB/CPU-s	0.03	77.97%	13.62MiB/CPU-s	228.36KiB/CPU-s	2.95KiB/CPU-s	0.016372	13.62MiB/CPU-s	188.51KiB/CPU-s	2.43KiB/CPU-s	0.01351	False	False
splunk_hec_indexer_ack_blackhole	-604.48B/CPU-s	-0.0	10.62%	13.62MiB/CPU-s	241.47KiB/CPU-s	3.12KiB/CPU-s	0.017314	13.62MiB/CPU-s	243.15KiB/CPU-s	3.14KiB/CPU-s	0.017435	False	False
http_to_http_json	153.11B/CPU-s	0.0	3.10%	13.62MiB/CPU-s	210.26KiB/CPU-s	2.71KiB/CPU-s	0.015072	13.62MiB/CPU-s	210.89KiB/CPU-s	2.72KiB/CPU-s	0.015117	False	False
fluent_elasticsearch	-102.33B/CPU-s	-0.0	14.79%	45.41MiB/CPU-s	29.76KiB/CPU-s	389.18B/CPU-s	0.00064	45.41MiB/CPU-s	29.61KiB/CPU-s	387.23B/CPU-s	0.000637	False	False
file_to_blackhole	-11.76KiB/CPU-s	-0.02	40.57%	54.5MiB/CPU-s	1.17MiB/CPU-s	15.44KiB/CPU-s	0.021445	54.49MiB/CPU-s	1.2MiB/CPU-s	15.79KiB/CPU-s	0.021955	False	False
syslog_loki	-8.55KiB/CPU-s	-0.09	97.99%	8.86MiB/CPU-s	236.54KiB/CPU-s	3.05KiB/CPU-s	0.026081	8.85MiB/CPU-s	158.96KiB/CPU-s	2.05KiB/CPU-s	0.017544	False	False
syslog_log2metric_splunk_hec_metrics	-20.93KiB/CPU-s	-0.22	99.91%	9.41MiB/CPU-s	272.2KiB/CPU-s	3.51KiB/CPU-s	0.028244	9.39MiB/CPU-s	402.44KiB/CPU-s	5.2KiB/CPU-s	0.04185	False	False
otlp_grpc_to_blackhole	-3.4KiB/CPU-s	-0.32	100.00%	1.05MiB/CPU-s	38.58KiB/CPU-s	510.02B/CPU-s	0.036021	1.04MiB/CPU-s	47.61KiB/CPU-s	629.21B/CPU-s	0.044592	False	False
otlp_http_to_blackhole	-9.34KiB/CPU-s	-0.59	100.00%	1.54MiB/CPU-s	106.83KiB/CPU-s	1.38KiB/CPU-s	0.067725	1.53MiB/CPU-s	113.48KiB/CPU-s	1.46KiB/CPU-s	0.07237	False	False
syslog_log2metric_humio_metrics	-81.06KiB/CPU-s	-1.31	100.00%	6.07MiB/CPU-s	278.25KiB/CPU-s	3.59KiB/CPU-s	0.044793	5.99MiB/CPU-s	270.23KiB/CPU-s	3.49KiB/CPU-s	0.044076	False	False
datadog_agent_remap_blackhole_acks	-667.32KiB/CPU-s	-2.09	100.00%	31.11MiB/CPU-s	1.05MiB/CPU-s	13.89KiB/CPU-s	0.033767	30.46MiB/CPU-s	1.52MiB/CPU-s	20.04KiB/CPU-s	0.049766	False	False
syslog_regex_logs2metric_ddmetrics	-151.77KiB/CPU-s	-4.06	100.00%	3.65MiB/CPU-s	415.8KiB/CPU-s	5.37KiB/CPU-s	0.111288	3.5MiB/CPU-s	439.23KiB/CPU-s	5.67KiB/CPU-s	0.122536	True	False

…watch-logs

github-actions · 2023-01-16T22:44:21Z

Regression Test Results

Run ID: f6fc201e-2e04-4ee3-adc4-97faa658dca8
Baseline: 889c167
Comparison: 2836c14
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparison SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_text_to_http_json	998.67KiB/CPU-s	4.09	100.00%	23.83MiB/CPU-s	833.52KiB/CPU-s	10.76KiB/CPU-s	0.034155	24.81MiB/CPU-s	811.92KiB/CPU-s	10.48KiB/CPU-s	0.031962	False	False
syslog_log2metric_splunk_hec_metrics	279.76KiB/CPU-s	3.08	100.00%	8.86MiB/CPU-s	376.39KiB/CPU-s	4.86KiB/CPU-s	0.041471	9.14MiB/CPU-s	340.89KiB/CPU-s	4.4KiB/CPU-s	0.036436	False	False
syslog_humio_logs	185.74KiB/CPU-s	1.99	100.00%	9.11MiB/CPU-s	180.91KiB/CPU-s	2.33KiB/CPU-s	0.019382	9.3MiB/CPU-s	95.38KiB/CPU-s	1.23KiB/CPU-s	0.010019	False	False
syslog_regex_logs2metric_ddmetrics	69.13KiB/CPU-s	1.79	100.00%	3.77MiB/CPU-s	441.22KiB/CPU-s	5.7KiB/CPU-s	0.11441	3.83MiB/CPU-s	475.49KiB/CPU-s	6.14KiB/CPU-s	0.121126	True	False
socket_to_socket_blackhole	230.81KiB/CPU-s	1.69	100.00%	13.32MiB/CPU-s	190.27KiB/CPU-s	2.46KiB/CPU-s	0.013953	13.54MiB/CPU-s	181.26KiB/CPU-s	2.34KiB/CPU-s	0.013071	False	False
otlp_http_to_blackhole	23.6KiB/CPU-s	1.53	100.00%	1.51MiB/CPU-s	135.39KiB/CPU-s	1.75KiB/CPU-s	0.087682	1.53MiB/CPU-s	119.24KiB/CPU-s	1.54KiB/CPU-s	0.076064	False	False
datadog_agent_remap_blackhole_acks	464.94KiB/CPU-s	1.47	100.00%	30.8MiB/CPU-s	1.02MiB/CPU-s	13.5KiB/CPU-s	0.033155	31.25MiB/CPU-s	932.22KiB/CPU-s	12.03KiB/CPU-s	0.02913	False	False
datadog_agent_remap_datadog_logs_acks	391.44KiB/CPU-s	1.16	100.00%	32.87MiB/CPU-s	1.62MiB/CPU-s	21.35KiB/CPU-s	0.049172	33.25MiB/CPU-s	1.34MiB/CPU-s	17.68KiB/CPU-s	0.040234	False	False
datadog_agent_remap_datadog_logs	377.53KiB/CPU-s	1.1	100.00%	33.59MiB/CPU-s	1.35MiB/CPU-s	17.85KiB/CPU-s	0.040221	33.96MiB/CPU-s	1.52MiB/CPU-s	20.05KiB/CPU-s	0.044685	False	False
http_to_http_acks	52.46KiB/CPU-s	0.98	69.38%	5.23MiB/CPU-s	2.71MiB/CPU-s	35.83KiB/CPU-s	0.518288	5.28MiB/CPU-s	2.77MiB/CPU-s	36.67KiB/CPU-s	0.525059	True	False
otlp_grpc_to_blackhole	9.75KiB/CPU-s	0.92	100.00%	1.03MiB/CPU-s	39.47KiB/CPU-s	521.72B/CPU-s	0.037279	1.04MiB/CPU-s	51.7KiB/CPU-s	683.07B/CPU-s	0.048387	False	False
syslog_splunk_hec_logs	54.74KiB/CPU-s	0.6	100.00%	8.98MiB/CPU-s	289.03KiB/CPU-s	3.73KiB/CPU-s	0.031434	9.03MiB/CPU-s	279.34KiB/CPU-s	3.61KiB/CPU-s	0.030201	False	False
enterprise_http_to_http	18.58KiB/CPU-s	0.13	99.94%	13.61MiB/CPU-s	388.84KiB/CPU-s	5.02KiB/CPU-s	0.027907	13.62MiB/CPU-s	150.29KiB/CPU-s	1.94KiB/CPU-s	0.010772	False	False
syslog_loki	6.44KiB/CPU-s	0.07	82.61%	8.74MiB/CPU-s	258.86KiB/CPU-s	3.34KiB/CPU-s	0.028916	8.75MiB/CPU-s	260.22KiB/CPU-s	3.36KiB/CPU-s	0.029048	False	False
http_to_http_noack	6.47KiB/CPU-s	0.05	73.45%	13.61MiB/CPU-s	353.22KiB/CPU-s	4.56KiB/CPU-s	0.025343	13.62MiB/CPU-s	278.96KiB/CPU-s	3.6KiB/CPU-s	0.020006	False	False
splunk_hec_to_splunk_hec_logs_noack	3.63KiB/CPU-s	0.03	66.69%	13.62MiB/CPU-s	218.2KiB/CPU-s	2.82KiB/CPU-s	0.015642	13.63MiB/CPU-s	192.11KiB/CPU-s	2.48KiB/CPU-s	0.013768	False	False
http_to_http_json	-192.86B/CPU-s	-0.0	3.89%	13.62MiB/CPU-s	211.29KiB/CPU-s	2.73KiB/CPU-s	0.015146	13.62MiB/CPU-s	212.1KiB/CPU-s	2.74KiB/CPU-s	0.015204	False	False
fluent_elasticsearch	62.97B/CPU-s	0.0	9.00%	45.41MiB/CPU-s	29.73KiB/CPU-s	388.71B/CPU-s	0.000639	45.41MiB/CPU-s	30.51KiB/CPU-s	398.77B/CPU-s	0.000656	False	False
file_to_blackhole	992.69B/CPU-s	0.0	3.68%	54.49MiB/CPU-s	1.09MiB/CPU-s	14.36KiB/CPU-s	0.019949	54.49MiB/CPU-s	1.16MiB/CPU-s	15.31KiB/CPU-s	0.021281	False	False
splunk_hec_to_splunk_hec_logs_acks	-3.43KiB/CPU-s	-0.02	40.75%	13.62MiB/CPU-s	339.55KiB/CPU-s	4.38KiB/CPU-s	0.02435	13.61MiB/CPU-s	361.83KiB/CPU-s	4.67KiB/CPU-s	0.025954	False	False
splunk_hec_indexer_ack_blackhole	-4.58KiB/CPU-s	-0.03	69.08%	13.62MiB/CPU-s	236.45KiB/CPU-s	3.05KiB/CPU-s	0.016952	13.62MiB/CPU-s	256.49KiB/CPU-s	3.31KiB/CPU-s	0.018395	False	False
splunk_hec_route_s3	-27.5KiB/CPU-s	-0.23	98.45%	11.77MiB/CPU-s	562.08KiB/CPU-s	7.25KiB/CPU-s	0.046616	11.75MiB/CPU-s	677.62KiB/CPU-s	8.74KiB/CPU-s	0.056326	False	False
datadog_agent_remap_blackhole	-621.5KiB/CPU-s	-2.01	100.00%	30.26MiB/CPU-s	1.87MiB/CPU-s	24.74KiB/CPU-s	0.061837	29.65MiB/CPU-s	2.25MiB/CPU-s	29.74KiB/CPU-s	0.07591	False	False
syslog_log2metric_humio_metrics	-143.1KiB/CPU-s	-2.3	100.00%	6.08MiB/CPU-s	179.14KiB/CPU-s	2.31KiB/CPU-s	0.028795	5.94MiB/CPU-s	210.69KiB/CPU-s	2.72KiB/CPU-s	0.034662	False	False

spencergilbert · 2023-01-30T15:51:14Z

@dizlv just confirming you're working on this and didn't need any review or feedback yet?

dizlv · 2023-01-30T21:45:38Z

@spencergilbert yeah, got burden with regular stuff previous week, most likely will get back on this tomorrow

spencergilbert · 2023-01-30T22:08:37Z

@spencergilbert yeah, got burden with regular stuff previous week, most likely will get back on this tomorrow

No rush, just wanted to make sure I wasn't blocking you on something 😅

vtatarin · 2023-07-22T01:32:55Z

hello @dizlv ! Do plan to continue the development of this feature or maybe you need help?

AndrewChubatiuk · 2023-10-17T15:36:40Z

#18865

jszwedko · 2023-12-01T19:19:10Z

Closed by #18865

dizlv added 4 commits January 13, 2023 21:54

Add retention configuration structures

5e92ba5

Pass default values for retention in integration tests

f470cd6

Add put retention and delete retention actions for client and new sta…

cc48cb8

…te for updating retention policy

Updated docstring for Retention structure

097f499

github-actions bot added the domain: sinks Anything related to the Vector's sinks label Jan 13, 2023

dizlv commented Jan 13, 2023

View reviewed changes

spencergilbert added the sink: aws_cloudwatch_logs Anything `aws_cloudwatch_logs` sink related label Jan 13, 2023

spencergilbert self-assigned this Jan 13, 2023

spencergilbert self-requested a review January 13, 2023 20:39

Merge branch 'master' into feature/add-log-retention-policy-for-cloud…

2836c14

…watch-logs

spencergilbert removed their request for review October 6, 2023 13:04

spencergilbert removed their assignment Oct 6, 2023

jszwedko closed this Dec 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(aws_cloudwatch_logs): Add retention policy for cloudwatch logs sink #15955

feat(aws_cloudwatch_logs): Add retention policy for cloudwatch logs sink #15955

dizlv commented Jan 13, 2023

netlify bot commented Jan 13, 2023 •

edited

Loading

netlify bot commented Jan 13, 2023 •

edited

Loading

dizlv left a comment •

edited

Loading

dizlv Jan 13, 2023

github-actions bot commented Jan 13, 2023

github-actions bot commented Jan 16, 2023

spencergilbert commented Jan 30, 2023

dizlv commented Jan 30, 2023

spencergilbert commented Jan 30, 2023

vtatarin commented Jul 22, 2023

AndrewChubatiuk commented Oct 17, 2023

jszwedko commented Dec 1, 2023

feat(aws_cloudwatch_logs): Add retention policy for cloudwatch logs sink #15955

feat(aws_cloudwatch_logs): Add retention policy for cloudwatch logs sink #15955

Conversation

dizlv commented Jan 13, 2023

netlify bot commented Jan 13, 2023 • edited Loading

✅ Deploy Preview for vector-project ready!

netlify bot commented Jan 13, 2023 • edited Loading

✅ Deploy Preview for vrl-playground canceled.

dizlv left a comment • edited Loading

Choose a reason for hiding this comment

dizlv Jan 13, 2023

Choose a reason for hiding this comment

github-actions bot commented Jan 13, 2023

Regression Test Results

github-actions bot commented Jan 16, 2023

Regression Test Results

spencergilbert commented Jan 30, 2023

dizlv commented Jan 30, 2023

spencergilbert commented Jan 30, 2023

vtatarin commented Jul 22, 2023

AndrewChubatiuk commented Oct 17, 2023

jszwedko commented Dec 1, 2023

netlify bot commented Jan 13, 2023 •

edited

Loading

netlify bot commented Jan 13, 2023 •

edited

Loading

dizlv left a comment •

edited

Loading