Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Upgrade lading #10141

Merged
merged 5 commits into from
Nov 23, 2021
Merged

chore: Upgrade lading #10141

merged 5 commits into from
Nov 23, 2021

Conversation

blt
Copy link
Contributor

@blt blt commented Nov 23, 2021

This commit upgrades the soaks to use lading 0.6 -- currently targetting a SHA
release -- in preparation for soak testing with static files. The reason an
upgrade to lading 0.6 is necessary is outlined in
DataDog/lading#119. I have extracted common configuration
files upward into soaks/common/configs, making the transition easier.

Signed-off-by: Brian L. Troutwine brian@troutwine.us

@blt
Upgrade lading
1e64dc2 
This commit upgrades the soaks to use lading 0.6 -- currently targetting a SHA
release -- in preparation for soak testing with static files. The reason an
upgrade to lading 0.6 is necessary is outlined in
DataDog/lading#119. I have extracted common configuration
files upward into `soaks/common/configs`, making the transition easier.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
@netlify
Copy link

netlify bot commented Nov 23, 2021 •
edited
Loading

✔️ Deploy Preview for vector-project canceled.

🔨 Explore the source changes: 923c002

🔍 Inspect the deploy log: https://app.netlify.com/sites/vector-project/deploys/619c54d7ed80960007bc2420

blt added 4 commits November 22, 2021 18:09
@blt
missed some changes
f909b9c 
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
@blt
path correction
8dac774 
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
@blt
file extensions
c0a23c7 
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
@blt
missed more config changes
923c002 
Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
@github-actions
Copy link

Soak Test Results

Baseline: e399c84
Comparison: 923c002
Total Vector CPUs: 4

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 10.43Mi 10.45Mi 10.46Mi 10.46Mi 0.24 -0.44
comparison 10.80Mi 10.82Mi 10.83Mi 10.83Mi -0.36 -0.17

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 19.55Mi 19.62Mi 19.62Mi 19.63Mi -0.28 -1.03
comparison 19.37Mi 19.41Mi 19.42Mi 19.42Mi 0.46 -0.60

fluent_elasticsearch

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 54.53Mi 54.96Mi 55.03Mi 55.04Mi -0.01 -0.54
comparison 52.55Mi 52.72Mi 52.75Mi 52.75Mi -0.06 -0.83

fluent_remap_aws_firehose

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 40.50Mi 40.69Mi 40.70Mi 40.70Mi -0.90 0.58
comparison 39.16Mi 39.31Mi 39.37Mi 39.38Mi 0.55 0.40

splunk_hec_route_s3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.18Mi 5.38Mi 5.48Mi 5.52Mi 0.87 0.84
comparison 5.41Mi 5.66Mi 5.70Mi 5.71Mi 0.38 -0.34

splunk_transforms_splunk3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 2.41Mi 2.60Mi 2.66Mi 2.66Mi 0.14 -0.99
comparison 2.46Mi 2.68Mi 2.72Mi 2.75Mi 0.06 -0.66

syslog_humio_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 7.26Mi 7.31Mi 7.31Mi 7.31Mi -0.47 -0.85
comparison 7.25Mi 7.29Mi 7.30Mi 7.30Mi -0.27 -0.98

syslog_log2metric_humio_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.05Mi 5.08Mi 5.08Mi 5.08Mi 0.21 -0.77
comparison 5.01Mi 5.05Mi 5.05Mi 5.06Mi -0.33 -0.60

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.08Mi 5.09Mi 5.09Mi 5.09Mi -0.05 -0.91
comparison 5.12Mi 5.15Mi 5.15Mi 5.16Mi 0.44 -0.81

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.81Mi 4.14Mi 4.19Mi 4.19Mi 0.20 -1.00
comparison 4.39Mi 4.58Mi 4.62Mi 4.62Mi 0.26 -0.46

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.87Mi 3.89Mi 3.89Mi 3.89Mi -0.74 -0.32
comparison 3.73Mi 3.75Mi 3.76Mi 3.76Mi 0.44 -0.15

syslog_splunk_hec_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 6.96Mi 7.03Mi 7.04Mi 7.04Mi -1.08 -0.21
comparison 7.29Mi 7.31Mi 7.32Mi 7.32Mi 0.36 -0.31

blt added a commit that referenced this pull request Nov 23, 2021
@blt
Introduce a new http -> pipelines -> blackhole soak
570ce52 
This commit introduces a new soak to investigate our pipelines in an ongoing
basis. The pipeline configuration for vector and the sample data was contributed
by work was contributed @vladimir-dd. I have expanded the soak to allow mounting
a TESTNAME/data to smuggle static data into the minikube, by which we feed
http_gen. Unfortunately this does not work as we cannot use the Virtual Box VM
for soak testing and kubernetes/minikube#12301 is
open. We will have to figure out some other way to smuggle data into the
minikube for this soak to function.

This commit depends on #10141 which
depends on DataDog/lading#119.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
@blt blt mentioned this pull request Nov 23, 2021
Merged
@blt blt merged commit 3c2469c into master Nov 23, 2021
@blt blt deleted the soak_lading_06 branch November 23, 2021 15:02
blt added a commit that referenced this pull request Nov 23, 2021
@blt
Introduce a new http -> pipelines -> blackhole soak
ac55a60 
This commit introduces a new soak to investigate our pipelines in an ongoing
basis. The pipeline configuration for vector and the sample data was contributed
by work was contributed @vladimir-dd. I have expanded the soak to allow mounting
a TESTNAME/data to smuggle static data into the minikube, by which we feed
http_gen. Unfortunately this does not work as we cannot use the Virtual Box VM
for soak testing and kubernetes/minikube#12301 is
open. We will have to figure out some other way to smuggle data into the
minikube for this soak to function.

This commit depends on #10141 which
depends on DataDog/lading#119.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
blt added a commit that referenced this pull request Nov 24, 2021
@blt
chore: Introduce a new http -> pipelines -> blackhole soak (#10142)
808c1e3 
* Introduce a new http -> pipelines -> blackhole soak

This commit introduces a new soak to investigate our pipelines in an ongoing
basis. The pipeline configuration for vector and the sample data was contributed
by work was contributed @vladimir-dd. I have expanded the soak to allow mounting
a TESTNAME/data to smuggle static data into the minikube, by which we feed
http_gen. Unfortunately this does not work as we cannot use the Virtual Box VM
for soak testing and kubernetes/minikube#12301 is
open. We will have to figure out some other way to smuggle data into the
minikube for this soak to function.

This commit depends on #10141 which
depends on DataDog/lading#119.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>

* Use configmap hack

It turns out that the problem discussed in the last commit is fixed by
kubernetes/minikube#13013. This is not present in a
minikube release yet, so we can't rely on it. That said, this commit introduces
a hack whereby the bootstrap is passed in a configmap. This limits the size of
the bootstrap with all the problems that come along with a low-entropy
experiment.

Signed-off-by: Brian L. Troutwine <brian@troutwine.us>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tobz tobz tobz approved these changes

@gaylatea gaylatea Awaiting requested review from gaylatea

@spencergilbert spencergilbert Awaiting requested review from spencergilbert

@wbew wbew Awaiting requested review from wbew

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@blt @tobz