Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(new source): add AWS SQS source #9968

Merged
merged 19 commits into from
Nov 12, 2021
Merged

feat(new source): add AWS SQS source #9968

merged 19 commits into from
Nov 12, 2021

Conversation

fuchsnj
Copy link
Member

@fuchsnj fuchsnj commented Nov 9, 2021 •
edited
Loading

closes #6830
This introduces a new AWS SQS source.

This also is the first source to start using the AWS official rust SDK. There are a few small rough edges around config between the new/old SDK, but for the most part it just works. The biggest downside I've seen is that the new SDK currently doesn't allow using a custom file for auth (it can only use the default file locations).

E2E acking, and setting concurrency limits were included. Emitting latency metrics was considered out of scope for this PR. This is something we are looking at introducing in a more generic way in the future.

EventsReceived was also moved to vector_core (we are slowly moving all of the "core" events into vector_core)

@fuchsnj
save
67334fb 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
Merge branch 'master' into fuchsnj/sqs_source
b6ba85a
@fuchsnj
save
457471a 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
add E2E acking
89779c8 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
safe
a78b444 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
tests pass
0649320 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
save
9ca3303 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
cleanup
68676b6 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
Merge branch 'master' into fuchsnj/sqs_source
de93e7d 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@netlify
Copy link

netlify bot commented Nov 9, 2021 •
edited
Loading

✔️ Deploy Preview for vector-project ready!

🔨 Explore the source changes: b6662b1

🔍 Inspect the deploy log: https://app.netlify.com/sites/vector-project/deploys/618d7303e705f2000945faeb

😎 Browse the preview: https://deploy-preview-9968--vector-project.netlify.app/reports/lighthouse

@github-actions github-actions bot added domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks domain: sources Anything related to the Vector's sources labels Nov 9, 2021
@fuchsnj fuchsnj requested review from bruceg and blt November 9, 2021 14:37
@fuchsnj fuchsnj changed the title feature(new source): add AWS SQS source feat(new source): add AWS SQS source Nov 9, 2021
@github-actions
Copy link

github-actions bot commented Nov 9, 2021

Soak Test Results

Baseline: f957dfe
Comparison: de93e7d

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 41.75Mi 41.89Mi 41.92Mi 41.93Mi 0.41 -0.33
comparison 40.00Mi 40.61Mi 40.66Mi 40.67Mi 1.02 -0.68

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 73.86Mi 74.03Mi 74.06Mi 74.07Mi 1.01 0.40
comparison 75.62Mi 76.50Mi 76.52Mi 76.52Mi -9.07 94.52

syslog_humio_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 28.16Mi 28.46Mi 28.49Mi 28.49Mi -0.25 -0.63
comparison 28.02Mi 28.07Mi 28.08Mi 28.08Mi 0.22 -1.06

syslog_log2metric_humio_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 19.84Mi 19.91Mi 19.92Mi 19.92Mi 0.13 -1.12
comparison 20.28Mi 20.44Mi 20.45Mi 20.45Mi 0.42 -1.47

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 20.94Mi 21.16Mi 21.18Mi 21.18Mi -0.07 -1.38
comparison 21.12Mi 21.20Mi 21.20Mi 21.21Mi -0.54 0.37

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 15.86Mi 17.01Mi 17.27Mi 17.29Mi 0.18 -0.84
comparison 15.79Mi 17.35Mi 17.57Mi 17.59Mi 0.71 -0.95

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 15.48Mi 15.59Mi 15.62Mi 15.62Mi 0.35 -0.94
comparison 15.43Mi 15.47Mi 15.48Mi 15.48Mi -0.02 -1.04

syslog_splunk_hec_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 28.07Mi 28.47Mi 28.49Mi 28.49Mi 0.39 -1.31
comparison 28.97Mi 29.20Mi 29.21Mi 29.21Mi -0.16 -1.05

@fuchsnj
fix tests
e3e1eb3 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@github-actions
Copy link

github-actions bot commented Nov 9, 2021

Soak Test Results

Baseline: f957dfe
Comparison: e3e1eb3

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 40.20Mi 42.59Mi 42.62Mi 42.63Mi -7.14 54.21
comparison 42.26Mi 42.39Mi 42.41Mi 42.41Mi -0.98 0.69

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 75.67Mi 76.48Mi 76.51Mi 76.52Mi -8.10 80.79
comparison 74.53Mi 76.31Mi 76.34Mi 76.37Mi -9.23 95.75

syslog_humio_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 27.21Mi 27.35Mi 27.41Mi 27.41Mi 1.32 1.12
comparison 27.81Mi 28.06Mi 28.08Mi 28.08Mi 0.13 -1.41

syslog_log2metric_humio_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 19.84Mi 19.93Mi 19.95Mi 19.95Mi 0.48 -0.79
comparison 19.85Mi 19.89Mi 19.91Mi 19.91Mi 0.21 -0.27

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 21.24Mi 21.29Mi 21.31Mi 21.31Mi 0.10 -0.62
comparison 20.67Mi 20.79Mi 20.81Mi 20.81Mi 0.35 -0.84

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 15.33Mi 15.99Mi 16.08Mi 16.09Mi 0.63 -1.01
comparison 15.79Mi 17.35Mi 17.57Mi 17.59Mi 0.71 -0.95

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 15.26Mi 15.36Mi 15.38Mi 15.39Mi 0.02 -0.43
comparison 15.09Mi 15.17Mi 15.19Mi 15.19Mi -0.17 -0.58

syslog_splunk_hec_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 29.10Mi 29.39Mi 29.43Mi 29.44Mi 0.55 -0.97
comparison 28.97Mi 29.20Mi 29.21Mi 29.21Mi -0.16 -1.05

@fuchsnj
cargo fmt
1630b1d 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@github-actions
Copy link

github-actions bot commented Nov 9, 2021

Soak Test Results

Baseline: f957dfe
Comparison: 1630b1d

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 40.20Mi 42.59Mi 42.62Mi 42.63Mi -7.14 54.21
comparison 42.26Mi 42.39Mi 42.41Mi 42.41Mi -0.98 0.69

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 75.67Mi 76.48Mi 76.51Mi 76.52Mi -8.10 80.79
comparison 74.58Mi 75.78Mi 75.80Mi 75.81Mi -8.85 92.06

syslog_humio_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 27.21Mi 27.35Mi 27.41Mi 27.41Mi 1.32 1.12
comparison 28.83Mi 29.25Mi 29.27Mi 29.27Mi -0.15 -1.64

syslog_log2metric_humio_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 20.02Mi 20.11Mi 20.12Mi 20.12Mi 0.09 -1.36
comparison 19.85Mi 19.89Mi 19.91Mi 19.91Mi 0.21 -0.27

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 19.90Mi 21.20Mi 21.22Mi 21.23Mi -6.86 50.47
comparison 20.47Mi 20.61Mi 20.63Mi 20.63Mi -0.44 -0.60

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 15.81Mi 16.77Mi 16.85Mi 16.85Mi 0.51 -1.36
comparison 15.79Mi 17.35Mi 17.57Mi 17.59Mi 0.71 -0.95

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 15.60Mi 15.67Mi 15.69Mi 15.69Mi 0.10 -0.93
comparison 15.43Mi 15.47Mi 15.48Mi 15.48Mi -0.02 -1.04

syslog_splunk_hec_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 28.70Mi 28.78Mi 28.79Mi 28.79Mi 0.27 -1.41
comparison 29.22Mi 29.34Mi 29.35Mi 29.35Mi -0.88 -0.84

blt
blt reviewed Nov 9, 2021
View reviewed changes
src/aws/rusoto/region.rs Outdated Show resolved Hide resolved
blt
blt reviewed Nov 9, 2021
View reviewed changes
Copy link
Contributor

@blt blt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems straightforward and uncontroversial to me. I've got some comments but nothing major.

src/sources/aws_sqs/integration_tests.rs Outdated Show resolved Hide resolved
website/cue/reference/components/sources/aws_sqs.cue Outdated Show resolved Hide resolved
@fuchsnj
cleanup
30ec387 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj fuchsnj removed the request for review from bruceg November 9, 2021 19:49
@github-actions
Copy link

github-actions bot commented Nov 9, 2021

Soak Test Results

Baseline: 99e7549
Comparison: 30ec387
Total Vector CPUs: 4

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 10.50Mi 10.53Mi 10.53Mi 10.53Mi -0.05 -0.11
comparison 10.63Mi 10.65Mi 10.66Mi 10.66Mi 0.60 0.95

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 18.71Mi 18.74Mi 18.75Mi 18.75Mi -0.16 -0.40
comparison 19.31Mi 19.36Mi 19.36Mi 19.37Mi -0.25 -0.69

syslog_humio_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 7.16Mi 7.27Mi 7.28Mi 7.28Mi -0.51 -1.15
comparison 7.34Mi 7.38Mi 7.39Mi 7.39Mi -0.71 -0.55

syslog_log2metric_humio_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.06Mi 5.10Mi 5.11Mi 5.11Mi -0.68 0.21
comparison 4.95Mi 4.97Mi 4.97Mi 4.98Mi -1.01 0.40

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.32Mi 5.35Mi 5.36Mi 5.36Mi -0.20 -1.39
comparison 5.31Mi 5.37Mi 5.37Mi 5.37Mi 0.20 -0.83

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.91Mi 4.11Mi 4.18Mi 4.19Mi -0.05 -0.99
comparison 3.89Mi 4.04Mi 4.15Mi 4.16Mi 0.54 1.06

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.92Mi 3.94Mi 3.94Mi 3.94Mi 0.22 -0.37
comparison 3.89Mi 3.91Mi 3.91Mi 3.91Mi -0.37 -0.51

syslog_splunk_hec_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 7.25Mi 7.28Mi 7.28Mi 7.28Mi 0.21 -1.06
comparison 7.21Mi 7.25Mi 7.25Mi 7.25Mi -1.14 0.14

@fuchsnj
update docs
50440ab 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
fix import
d929e52 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@github-actions
Copy link

Soak Test Results

Baseline: a9c310c
Comparison: d929e52
Total Vector CPUs: 4

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 10.67Mi 10.70Mi 10.70Mi 10.70Mi -0.14 -0.29
comparison 10.65Mi 10.68Mi 10.69Mi 10.69Mi -0.13 -0.57

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 18.71Mi 18.74Mi 18.75Mi 18.75Mi -0.16 -0.40
comparison 19.31Mi 19.36Mi 19.36Mi 19.37Mi -0.25 -0.69

splunk_hec_route_s3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.27Mi 5.59Mi 5.66Mi 5.68Mi 0.26 -0.43
comparison 5.68Mi 6.02Mi 6.12Mi 6.12Mi 0.10 -0.47

splunk_transforms_splunk3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 2.65Mi 2.78Mi 2.82Mi 2.84Mi 0.45 -0.48
comparison 2.62Mi 2.76Mi 2.78Mi 2.81Mi 0.53 -0.28

syslog_humio_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 7.16Mi 7.27Mi 7.28Mi 7.28Mi -0.51 -1.15
comparison 7.34Mi 7.38Mi 7.39Mi 7.39Mi -0.71 -0.55

syslog_log2metric_humio_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.06Mi 5.10Mi 5.11Mi 5.11Mi -0.68 0.21
comparison 4.78Mi 4.81Mi 4.82Mi 4.82Mi 1.07 -0.23

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.32Mi 5.35Mi 5.36Mi 5.36Mi -0.20 -1.39
comparison 5.12Mi 5.15Mi 5.16Mi 5.16Mi 0.62 -0.12

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.91Mi 4.11Mi 4.18Mi 4.19Mi -0.05 -0.99
comparison 3.72Mi 4.07Mi 4.13Mi 4.13Mi -0.34 -0.68

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.70Mi 3.71Mi 3.72Mi 3.72Mi -0.18 -0.93
comparison 3.87Mi 3.88Mi 3.89Mi 3.89Mi -0.08 -0.78

syslog_splunk_hec_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 7.27Mi 7.35Mi 7.36Mi 7.36Mi -0.32 -1.53
comparison 7.17Mi 7.25Mi 7.26Mi 7.26Mi -1.24 0.56

@fuchsnj
appease the clippy gods
c0ac9fb 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
fix cue docs
568c892 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
actually add SQS to the docs...
e3c6d86 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@fuchsnj
improve docs
79e2fa0 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@github-actions
Copy link

Soak Test Results

Baseline: 6cfb28d
Comparison: 79e2fa0
Total Vector CPUs: 4

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 10.19Mi 10.23Mi 10.23Mi 10.23Mi 0.26 -0.46
comparison 10.63Mi 10.65Mi 10.66Mi 10.66Mi 0.60 0.95

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 19.24Mi 19.29Mi 19.30Mi 19.30Mi -0.14 -0.71
comparison 19.53Mi 19.58Mi 19.58Mi 19.58Mi -0.19 -0.83

fluent_remap_aws_firehose

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 39.22Mi 39.40Mi 39.48Mi 39.49Mi 0.61 0.42
comparison 37.13Mi 39.00Mi 39.10Mi 39.11Mi 0.06 -1.48

splunk_hec_route_s3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.70Mi 6.01Mi 6.09Mi 6.12Mi 0.05 -0.53
comparison 5.68Mi 6.02Mi 6.12Mi 6.12Mi 0.10 -0.47

splunk_transforms_splunk3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 2.49Mi 2.74Mi 2.76Mi 2.77Mi -0.53 0.40
comparison 2.55Mi 2.68Mi 2.74Mi 2.76Mi 0.07 0.94

syslog_humio_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 7.16Mi 7.27Mi 7.28Mi 7.28Mi -0.51 -1.15
comparison 6.94Mi 7.07Mi 7.08Mi 7.08Mi 0.04 -1.10

syslog_log2metric_humio_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.06Mi 5.10Mi 5.11Mi 5.11Mi -0.68 0.21
comparison 4.94Mi 4.97Mi 4.98Mi 4.98Mi -0.10 -1.54

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.24Mi 5.27Mi 5.27Mi 5.27Mi -0.69 -0.51
comparison 5.03Mi 5.26Mi 5.26Mi 5.27Mi 0.04 -1.36

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.97Mi 4.12Mi 4.20Mi 4.21Mi 0.24 0.11
comparison 3.72Mi 4.07Mi 4.13Mi 4.13Mi -0.34 -0.68

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.90Mi 3.92Mi 3.93Mi 3.93Mi -0.09 -0.78
comparison 3.82Mi 3.86Mi 3.87Mi 3.87Mi -1.35 0.55

syslog_splunk_hec_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 6.90Mi 6.93Mi 6.93Mi 6.93Mi -0.24 -1.49
comparison 7.12Mi 7.22Mi 7.25Mi 7.25Mi 1.03 0.01

@fuchsnj
fix cue formatting
b6662b1 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
@github-actions
Copy link

Soak Test Results

Baseline: 6cfb28d
Comparison: b6662b1
Total Vector CPUs: 4

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 10.33Mi 10.74Mi 10.74Mi 10.74Mi -1.77 1.82
comparison 10.63Mi 10.65Mi 10.66Mi 10.66Mi 0.60 0.95

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 19.24Mi 19.29Mi 19.30Mi 19.30Mi -0.14 -0.71
comparison 19.80Mi 20.04Mi 20.05Mi 20.05Mi -0.34 -1.60

fluent_remap_aws_firehose

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 39.22Mi 39.40Mi 39.48Mi 39.49Mi 0.61 0.42
comparison 37.13Mi 39.00Mi 39.10Mi 39.11Mi 0.06 -1.48

splunk_hec_route_s3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.70Mi 6.01Mi 6.09Mi 6.12Mi 0.05 -0.53
comparison 5.50Mi 5.93Mi 6.03Mi 6.05Mi 0.14 -0.48

splunk_transforms_splunk3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 2.43Mi 2.73Mi 2.75Mi 2.77Mi -0.43 -0.65
comparison 2.50Mi 2.67Mi 2.71Mi 2.72Mi 0.10 -0.69

syslog_humio_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 7.15Mi 7.32Mi 7.33Mi 7.33Mi -0.50 -1.22
comparison 6.94Mi 7.07Mi 7.08Mi 7.08Mi 0.04 -1.10

syslog_log2metric_humio_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.03Mi 5.05Mi 5.06Mi 5.06Mi 0.59 -0.91
comparison 4.94Mi 4.97Mi 4.98Mi 4.98Mi -0.10 -1.54

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.33Mi 5.35Mi 5.36Mi 5.36Mi -0.25 -1.25
comparison 5.12Mi 5.15Mi 5.16Mi 5.16Mi 0.62 -0.12

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.97Mi 4.12Mi 4.20Mi 4.21Mi 0.24 0.11
comparison 3.84Mi 4.10Mi 4.11Mi 4.12Mi 0.08 -1.40

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.90Mi 3.92Mi 3.93Mi 3.93Mi -0.09 -0.78
comparison 3.89Mi 3.91Mi 3.91Mi 3.91Mi -0.37 -0.51

syslog_splunk_hec_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 6.90Mi 6.93Mi 6.93Mi 6.93Mi -0.24 -1.49
comparison 7.17Mi 7.25Mi 7.26Mi 7.26Mi -1.24 0.56

@fuchsnj fuchsnj requested a review from blt November 11, 2021 22:07
@fuchsnj fuchsnj merged commit b273b78 into master Nov 12, 2021
@fuchsnj fuchsnj deleted the fuchsnj/sqs_source branch November 12, 2021 13:56
jszwedko
jszwedko reviewed Nov 12, 2021
View reviewed changes
website/cue/reference/components/sources/aws_sqs.cue
Comment on lines +65 to +74
// client_concurrency: {
// common: true
// description: "How many clients are receiving / acking SQS messages. Increasing may allow higher throughput. Note: the default is 1 / CPU core"
// required: false
// warnings: []
// type: uint: {
// default: 1
// unit: "# of clients"
// }
// }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we end up leaving this option out?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was not intentional. I'll add this back in.

jszwedko
jszwedko reviewed Nov 12, 2021
View reviewed changes
src/aws/aws_sdk/auth.rs
credentials_file,
profile,
} => {
warn!("Overriding the credentials file is not supported. `~/.aws/config` and `~/.aws/credentials` will be used instead of \"{}\"", credentials_file);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we note this in the docs for this source?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure of a good place to put this. All the AWS docs are shared, and I think this is a pretty minor edge case that very few would hit.

@jszwedko
Copy link
Member

Just noticing that I don't think we added this new source's integration tests to the integration test workflow: https://github.com/vectordotdev/vector/blob/master/.github/workflows/integration-test.yml

I wonder if we could get the makefile to output the integration test targets and dynamically build the set to run (similar to the soak tests dynamically building their matrix) 🤔

@fuchsnj
Copy link
Member Author

fuchsnj commented Nov 13, 2021

Just noticing that I don't think we added this new source's integration tests to the integration test workflow: https://github.com/vectordotdev/vector/blob/master/.github/workflows/integration-test.yml

I wonder if we could get the makefile to output the integration test targets and dynamically build the set to run (similar to the soak tests dynamically building their matrix) thinking

aws covers all of the aws integration tests (including this one).

@jszwedko
Copy link
Member

Aha, you are right! Thanks!

jdrouet pushed a commit that referenced this pull request Nov 18, 2021
@fuchsnj @jdrouet
feat(new source): add AWS SQS source (#9968)
1e1e66c 
Signed-off-by: Nathan Fox <fuchsnj@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jszwedko jszwedko jszwedko left review comments

@blt blt blt approved these changes

Assignees
No one assigned
Labels
domain: core Anything related to core crates i.e. vector-core, core-common, etc domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks domain: sources Anything related to the Vector's sources
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Data source: AWS SQS
3 participants
@fuchsnj @jszwedko @blt