chore(vdev): Rewrite scripts in vdev #16661

jonathanpv · 2023-03-02T08:31:39Z

Problem

It is requested to rewrite some of the shell scripts into Rust so that it is supported with vdev

Solution

I rewrote test-vrl.sh, install-git-hooks.sh, release-github.sh and check-component-docs.sh in Rust and put them in the vdev command hierarchy I saw fit.

This involved adding a mod <name of command>, implementing the Cli command interface, translating the shell code into Rust, deleting the shell script, adding helper git functions, modifying the workflow to point to vdev cargo commands.

Testing

I require help verifying the following commands work:

`cargo vdev test-vrl`

This works and tests vrl if ran through the repository directory.

`cargo vdev meta install-git-hooks`

I verified this works by running it and then looking at the .git directory for the shell script in question to be copied into there.

`cargo vdev release github`

@jszwedko I request help with testing the release function

`cargo vdev check component-docs`

@jszwedko @bruceg I believe the only way to test this one will be through the CI as running generate component docs results in an error in the local machine, and upon reading the source it seems this is only to be ran in the CI, making it difficult for me to test all the cases of component-docs is supposed to hit.

Related Issues

#16383

Checklist

Add the ci-condition integration tests enable
Delete any rewritten scripts
Modify makefile to point to the new cargo vdev
Modify workflow if need be
Compare the output of the local script to the newly created cargo vdev Rust version
All CI integration tests passing
Test newly created cargo vdev commands

Signed-off-by: jonathanpv <jonpad512@gmail.com>

netlify · 2023-03-02T08:31:44Z

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	`da4337a`
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/640907cdb3721e0008bf7dc7

netlify · 2023-03-02T08:31:46Z

✅ Deploy Preview for vrl-playground canceled.

Name	Link
🔨 Latest commit	`da4337a`
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/640907cd5cc0ff0008795b53

Signed-off-by: jonathanpv <jonpad512@gmail.com>

github-actions · 2023-03-02T09:15:26Z

Regression Detector Results

Run ID: d6bed955-ba30-4822-9b34-ee883663d5e4
Baseline: 2e0de1f
Comparison: 6aab34a
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

Changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%:

experiment	goal	Δ mean	Δ mean %	confidence
file_to_blackhole	egress throughput	-859.66KiB/CPU-s	-12.09	100.00%

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
syslog_humio_logs	ingress throughput	189.84KiB/CPU-s	2.13	100.00%	8.69MiB/CPU-s	229.48KiB/CPU-s	2.82KiB/CPU-s	0.025793	8.87MiB/CPU-s	199.95KiB/CPU-s	2.46KiB/CPU-s	0.0	0.022005	False	False
datadog_agent_remap_datadog_logs	ingress throughput	307.55KiB/CPU-s	0.91	100.00%	33.13MiB/CPU-s	1.2MiB/CPU-s	15.08KiB/CPU-s	0.036135	33.43MiB/CPU-s	872.47KiB/CPU-s	10.73KiB/CPU-s	0.0	0.025488	False	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	82.28KiB/CPU-s	0.90	100.00%	8.97MiB/CPU-s	312.87KiB/CPU-s	3.85KiB/CPU-s	0.034063	9.05MiB/CPU-s	317.33KiB/CPU-s	3.9KiB/CPU-s	0.0	0.034242	False	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	32.54KiB/CPU-s	0.89	100.00%	3.56MiB/CPU-s	460.08KiB/CPU-s	5.66KiB/CPU-s	0.126143	3.59MiB/CPU-s	352.52KiB/CPU-s	4.34KiB/CPU-s	0.0	0.095797	True	False
datadog_agent_remap_blackhole_acks	ingress throughput	248.82KiB/CPU-s	0.79	100.00%	30.89MiB/CPU-s	1.43MiB/CPU-s	17.99KiB/CPU-s	0.046211	31.13MiB/CPU-s	844.64KiB/CPU-s	10.39KiB/CPU-s	0.0	0.026493	False	False
otlp_grpc_to_blackhole	ingress throughput	7.09KiB/CPU-s	0.67	100.00%	1.04MiB/CPU-s	49.95KiB/CPU-s	629.4B/CPU-s	0.047068	1.04MiB/CPU-s	44.79KiB/CPU-s	564.35B/CPU-s	0.0	0.041919	False	False
otlp_http_to_blackhole	ingress throughput	9.39KiB/CPU-s	0.59	100.00%	1.56MiB/CPU-s	108.82KiB/CPU-s	1.34KiB/CPU-s	0.067952	1.57MiB/CPU-s	98.18KiB/CPU-s	1.21KiB/CPU-s	0.0	0.060949	False	False
datadog_agent_remap_blackhole	ingress throughput	149.69KiB/CPU-s	0.47	100.00%	31.03MiB/CPU-s	1.05MiB/CPU-s	13.28KiB/CPU-s	0.033964	31.17MiB/CPU-s	966.24KiB/CPU-s	11.89KiB/CPU-s	0.0	0.030268	False	False
http_to_http_acks	ingress throughput	16.15KiB/CPU-s	0.30	25.85%	5.31MiB/CPU-s	2.76MiB/CPU-s	34.79KiB/CPU-s	0.520407	5.32MiB/CPU-s	2.73MiB/CPU-s	34.45KiB/CPU-s	0.0	0.513777	True	False
http_text_to_http_json	ingress throughput	62.77KiB/CPU-s	0.25	100.00%	24.51MiB/CPU-s	577.6KiB/CPU-s	7.11KiB/CPU-s	0.023008	24.58MiB/CPU-s	653.57KiB/CPU-s	8.04KiB/CPU-s	0.0	0.025969	False	False
syslog_loki	ingress throughput	15.95KiB/CPU-s	0.18	99.99%	8.51MiB/CPU-s	240.11KiB/CPU-s	2.95KiB/CPU-s	0.027554	8.52MiB/CPU-s	217.58KiB/CPU-s	2.68KiB/CPU-s	0.0	0.024923	False	False
enterprise_http_to_http	ingress throughput	3.89KiB/CPU-s	0.03	75.61%	13.62MiB/CPU-s	223.99KiB/CPU-s	2.76KiB/CPU-s	0.016059	13.62MiB/CPU-s	153.32KiB/CPU-s	1.89KiB/CPU-s	0.0	0.010989	False	False
http_to_http_noack	ingress throughput	2.62KiB/CPU-s	0.02	33.25%	13.61MiB/CPU-s	357.31KiB/CPU-s	4.4KiB/CPU-s	0.025638	13.61MiB/CPU-s	328.53KiB/CPU-s	4.24KiB/CPU-s	0.0	0.023567	False	False
fluent_elasticsearch	ingress throughput	3.09KiB/CPU-s	0.01	62.42%	45.41MiB/CPU-s	284.71KiB/CPU-s	3.47KiB/CPU-s	0.006122	45.41MiB/CPU-s	30.08KiB/CPU-s	375.03B/CPU-s	0.0	0.000647	False	False
http_to_http_json	ingress throughput	437.18B/CPU-s	0.00	8.90%	13.62MiB/CPU-s	219.78KiB/CPU-s	2.7KiB/CPU-s	0.015757	13.62MiB/CPU-s	219.51KiB/CPU-s	2.7KiB/CPU-s	0.0	0.015738	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-349.39B/CPU-s	-0.00	4.35%	13.61MiB/CPU-s	360.91KiB/CPU-s	4.44KiB/CPU-s	0.025887	13.61MiB/CPU-s	358.54KiB/CPU-s	4.41KiB/CPU-s	0.0	0.025718	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	580.94B/CPU-s	0.00	9.86%	13.61MiB/CPU-s	265.65KiB/CPU-s	3.27KiB/CPU-s	0.019054	13.61MiB/CPU-s	260.67KiB/CPU-s	3.21KiB/CPU-s	0.0	0.018696	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-2.15KiB/CPU-s	-0.02	36.07%	13.62MiB/CPU-s	257.32KiB/CPU-s	3.17KiB/CPU-s	0.018453	13.61MiB/CPU-s	271.11KiB/CPU-s	3.33KiB/CPU-s	0.0	0.019445	False	False
socket_to_socket_blackhole	ingress throughput	-11.19KiB/CPU-s	-0.08	99.79%	13.4MiB/CPU-s	230.79KiB/CPU-s	2.84KiB/CPU-s	0.016817	13.39MiB/CPU-s	184.54KiB/CPU-s	2.27KiB/CPU-s	0.0	0.013458	False	False
syslog_splunk_hec_logs	ingress throughput	-22.01KiB/CPU-s	-0.25	100.00%	8.7MiB/CPU-s	230.25KiB/CPU-s	2.83KiB/CPU-s	0.02585	8.68MiB/CPU-s	199.67KiB/CPU-s	2.46KiB/CPU-s	0.0	0.022473	False	False
splunk_hec_route_s3	ingress throughput	-65.33KiB/CPU-s	-0.54	100.00%	11.74MiB/CPU-s	513.09KiB/CPU-s	6.31KiB/CPU-s	0.04268	11.68MiB/CPU-s	527.79KiB/CPU-s	6.49KiB/CPU-s	0.0	0.044143	False	False
datadog_agent_remap_datadog_logs_acks	ingress throughput	-283.1KiB/CPU-s	-0.82	100.00%	33.69MiB/CPU-s	877.17KiB/CPU-s	10.79KiB/CPU-s	0.025425	33.41MiB/CPU-s	971.93KiB/CPU-s	11.96KiB/CPU-s	0.0	0.028404	False	False
syslog_log2metric_humio_metrics	ingress throughput	-52.5KiB/CPU-s	-0.82	100.00%	6.25MiB/CPU-s	72.81KiB/CPU-s	917.71B/CPU-s	0.011376	6.2MiB/CPU-s	134.83KiB/CPU-s	1.66KiB/CPU-s	0.0	0.021241	False	False
file_to_blackhole	egress throughput	-859.66KiB/CPU-s	-12.09	100.00%	6.94MiB/CPU-s	3.71MiB/CPU-s	121.36KiB/CPU-s	0.53382	6.1MiB/CPU-s	4.14MiB/CPU-s	120.19KiB/CPU-s	2.004812	0.678801	True	False

Signed-off-by: jonathanpv <jonpad512@gmail.com>

github-actions · 2023-03-02T11:17:56Z

Regression Detector Results

Run ID: 4d983d3b-b9de-4528-ad29-d6227090c406
Baseline: d0ed464
Comparison: 278fd74
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

Changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%:

experiment	goal	Δ mean	Δ mean %	confidence
file_to_blackhole	egress throughput	1.11MiB/CPU-s	19.65	100.00%

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
file_to_blackhole	egress throughput	1.11MiB/CPU-s	19.65	100.00%	5.66MiB/CPU-s	4.27MiB/CPU-s	107.8KiB/CPU-s	0.753283	6.78MiB/CPU-s	3.89MiB/CPU-s	124.68KiB/CPU-s	0.573596	True	False
datadog_agent_remap_blackhole_acks	ingress throughput	867.83KiB/CPU-s	2.79	100.00%	30.38MiB/CPU-s	1.43MiB/CPU-s	18.08KiB/CPU-s	0.047224	31.23MiB/CPU-s	707.37KiB/CPU-s	8.71KiB/CPU-s	0.022121	False	False
datadog_agent_remap_datadog_logs_acks	ingress throughput	797.53KiB/CPU-s	2.39	100.00%	32.54MiB/CPU-s	1.69MiB/CPU-s	21.26KiB/CPU-s	0.051856	33.32MiB/CPU-s	1.07MiB/CPU-s	13.47KiB/CPU-s	0.032097	False	False
http_text_to_http_json	ingress throughput	540.31KiB/CPU-s	2.18	100.00%	24.15MiB/CPU-s	943.98KiB/CPU-s	11.62KiB/CPU-s	0.038162	24.68MiB/CPU-s	547.26KiB/CPU-s	6.74KiB/CPU-s	0.021651	False	False
otlp_grpc_to_blackhole	ingress throughput	17.5KiB/CPU-s	1.68	100.00%	1.02MiB/CPU-s	51.42KiB/CPU-s	647.94B/CPU-s	0.049252	1.04MiB/CPU-s	49.18KiB/CPU-s	619.76B/CPU-s	0.046327	False	False
socket_to_socket_blackhole	ingress throughput	177.52KiB/CPU-s	1.31	100.00%	13.2MiB/CPU-s	240.54KiB/CPU-s	2.96KiB/CPU-s	0.017794	13.37MiB/CPU-s	261.86KiB/CPU-s	3.22KiB/CPU-s	0.01912	False	False
otlp_http_to_blackhole	ingress throughput	15.28KiB/CPU-s	0.96	100.00%	1.55MiB/CPU-s	114.69KiB/CPU-s	1.41KiB/CPU-s	0.072237	1.57MiB/CPU-s	109.74KiB/CPU-s	1.35KiB/CPU-s	0.068461	False	False
syslog_splunk_hec_logs	ingress throughput	42.97KiB/CPU-s	0.48	100.00%	8.7MiB/CPU-s	257.91KiB/CPU-s	3.17KiB/CPU-s	0.028932	8.75MiB/CPU-s	169.9KiB/CPU-s	2.09KiB/CPU-s	0.018968	False	False
syslog_log2metric_humio_metrics	ingress throughput	26.78KiB/CPU-s	0.42	100.00%	6.17MiB/CPU-s	184.12KiB/CPU-s	2.27KiB/CPU-s	0.029118	6.2MiB/CPU-s	155.93KiB/CPU-s	1.92KiB/CPU-s	0.024556	False	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	12.58KiB/CPU-s	0.34	94.23%	3.66MiB/CPU-s	394.58KiB/CPU-s	4.86KiB/CPU-s	0.105366	3.67MiB/CPU-s	366.63KiB/CPU-s	4.51KiB/CPU-s	0.097576	True	False
datadog_agent_remap_datadog_logs	ingress throughput	81.98KiB/CPU-s	0.24	100.00%	33.5MiB/CPU-s	792.91KiB/CPU-s	9.76KiB/CPU-s	0.023114	33.58MiB/CPU-s	777.25KiB/CPU-s	9.56KiB/CPU-s	0.022604	False	False
enterprise_http_to_http	ingress throughput	3.56KiB/CPU-s	0.03	72.30%	13.62MiB/CPU-s	217.99KiB/CPU-s	2.68KiB/CPU-s	0.015628	13.62MiB/CPU-s	152.18KiB/CPU-s	1.87KiB/CPU-s	0.010907	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	-101.36B/CPU-s	-0.00	1.73%	13.62MiB/CPU-s	263.05KiB/CPU-s	3.24KiB/CPU-s	0.018866	13.62MiB/CPU-s	262.61KiB/CPU-s	3.23KiB/CPU-s	0.018835	False	False
fluent_elasticsearch	ingress throughput	72.4B/CPU-s	0.00	9.87%	45.41MiB/CPU-s	36.13KiB/CPU-s	450.29B/CPU-s	0.000777	45.41MiB/CPU-s	29.79KiB/CPU-s	371.41B/CPU-s	0.000641	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	587.88B/CPU-s	0.00	7.57%	13.61MiB/CPU-s	340.5KiB/CPU-s	4.19KiB/CPU-s	0.024423	13.61MiB/CPU-s	354.07KiB/CPU-s	4.35KiB/CPU-s	0.025395	False	False
http_to_http_noack	ingress throughput	-1.87KiB/CPU-s	-0.01	22.14%	13.61MiB/CPU-s	376.47KiB/CPU-s	4.63KiB/CPU-s	0.027016	13.61MiB/CPU-s	389.54KiB/CPU-s	4.79KiB/CPU-s	0.027957	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-4.87KiB/CPU-s	-0.03	71.87%	13.62MiB/CPU-s	241.07KiB/CPU-s	2.97KiB/CPU-s	0.017285	13.61MiB/CPU-s	277.0KiB/CPU-s	3.41KiB/CPU-s	0.019869	False	False
http_to_http_json	ingress throughput	-36.8KiB/CPU-s	-0.26	100.00%	13.62MiB/CPU-s	218.84KiB/CPU-s	2.69KiB/CPU-s	0.01569	13.58MiB/CPU-s	264.36KiB/CPU-s	3.25KiB/CPU-s	0.019003	False	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	-29.33KiB/CPU-s	-0.32	100.00%	9.03MiB/CPU-s	218.79KiB/CPU-s	2.69KiB/CPU-s	0.02367	9.0MiB/CPU-s	281.54KiB/CPU-s	3.46KiB/CPU-s	0.030556	False	False
syslog_loki	ingress throughput	-57.57KiB/CPU-s	-0.66	100.00%	8.54MiB/CPU-s	220.68KiB/CPU-s	2.71KiB/CPU-s	0.02523	8.49MiB/CPU-s	263.45KiB/CPU-s	3.24KiB/CPU-s	0.030319	False	False
http_to_http_acks	ingress throughput	-45.84KiB/CPU-s	-0.84	65.01%	5.31MiB/CPU-s	2.76MiB/CPU-s	34.78KiB/CPU-s	0.519599	5.27MiB/CPU-s	2.74MiB/CPU-s	34.56KiB/CPU-s	0.520639	True	False
splunk_hec_route_s3	ingress throughput	-110.8KiB/CPU-s	-0.92	100.00%	11.79MiB/CPU-s	516.3KiB/CPU-s	6.35KiB/CPU-s	0.042756	11.68MiB/CPU-s	653.74KiB/CPU-s	8.04KiB/CPU-s	0.054639	False	False
syslog_humio_logs	ingress throughput	-114.02KiB/CPU-s	-1.26	100.00%	8.83MiB/CPU-s	191.88KiB/CPU-s	2.36KiB/CPU-s	0.021224	8.72MiB/CPU-s	263.73KiB/CPU-s	3.24KiB/CPU-s	0.029543	False	False
datadog_agent_remap_blackhole	ingress throughput	-1.3MiB/CPU-s	-4.14	100.00%	31.42MiB/CPU-s	1.0MiB/CPU-s	12.6KiB/CPU-s	0.031843	30.12MiB/CPU-s	2.46MiB/CPU-s	31.01KiB/CPU-s	0.081687	False	False

jszwedko

Good start! Just a small recommendation to do one pull request per-script. This will help them get reviewed and merged in more quickly.

jonathanpv · 2023-03-02T13:28:58Z

Good start! Just a small recommendation to do one pull request per-script. This will help them get reviewed and merged in more quickly.

Ah I see I'll do one more script here that I am almost finished with and then do a PR by PR basis

Signed-off-by: jonathanpv <jonpad512@gmail.com>

jonathanpv · 2023-03-02T14:18:01Z

@jszwedko how can I test the
cargo vdev release github command. I fear if I run it I will do something irreversible to something in Github.
Is it possible for you to let me know if the command works?

github-actions · 2023-03-02T14:56:24Z

Regression Detector Results

Run ID: 5ad8e58d-6694-40e1-89f6-c8d486db93e3
Baseline: d0ed464
Comparison: c08f74c
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_regex_logs2metric_ddmetrics	ingress throughput	132.63KiB/CPU-s	3.67	100.00%	3.53MiB/CPU-s	471.4KiB/CPU-s	5.8KiB/CPU-s	0.0	0.130427	3.66MiB/CPU-s	404.63KiB/CPU-s	4.98KiB/CPU-s	0.107992	True	False
socket_to_socket_blackhole	ingress throughput	332.13KiB/CPU-s	2.49	100.00%	13.0MiB/CPU-s	377.4KiB/CPU-s	4.64KiB/CPU-s	0.0	0.028344	13.33MiB/CPU-s	280.4KiB/CPU-s	3.45KiB/CPU-s	0.020547	False	False
otlp_grpc_to_blackhole	ingress throughput	19.91KiB/CPU-s	1.89	100.00%	1.03MiB/CPU-s	51.64KiB/CPU-s	650.53B/CPU-s	0.0	0.049126	1.05MiB/CPU-s	44.83KiB/CPU-s	564.93B/CPU-s	0.041859	False	False
datadog_agent_remap_datadog_logs	ingress throughput	399.93KiB/CPU-s	1.18	100.00%	33.16MiB/CPU-s	1.04MiB/CPU-s	13.07KiB/CPU-s	0.0	0.03129	33.55MiB/CPU-s	1.09MiB/CPU-s	13.78KiB/CPU-s	0.032587	False	False
http_text_to_http_json	ingress throughput	233.1KiB/CPU-s	0.92	100.00%	24.68MiB/CPU-s	567.01KiB/CPU-s	6.98KiB/CPU-s	0.0	0.022432	24.91MiB/CPU-s	555.13KiB/CPU-s	6.83KiB/CPU-s	0.021761	False	False
otlp_http_to_blackhole	ingress throughput	14.43KiB/CPU-s	0.92	100.00%	1.54MiB/CPU-s	115.55KiB/CPU-s	1.42KiB/CPU-s	0.0	0.07326	1.55MiB/CPU-s	111.64KiB/CPU-s	1.37KiB/CPU-s	0.070139	False	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	66.85KiB/CPU-s	0.73	100.00%	8.98MiB/CPU-s	342.47KiB/CPU-s	4.21KiB/CPU-s	0.0	0.037236	9.05MiB/CPU-s	277.94KiB/CPU-s	3.42KiB/CPU-s	0.030002	False	False
syslog_loki	ingress throughput	37.96KiB/CPU-s	0.44	100.00%	8.47MiB/CPU-s	309.92KiB/CPU-s	3.81KiB/CPU-s	0.0	0.035735	8.51MiB/CPU-s	239.98KiB/CPU-s	2.95KiB/CPU-s	0.02755	False	False
enterprise_http_to_http	ingress throughput	9.44KiB/CPU-s	0.07	97.85%	13.61MiB/CPU-s	296.76KiB/CPU-s	3.65KiB/CPU-s	0.0	0.021284	13.62MiB/CPU-s	152.91KiB/CPU-s	1.88KiB/CPU-s	0.01096	False	False
http_to_http_noack	ingress throughput	6.0KiB/CPU-s	0.04	63.29%	13.6MiB/CPU-s	409.8KiB/CPU-s	5.04KiB/CPU-s	0.0	0.029415	13.61MiB/CPU-s	352.42KiB/CPU-s	4.34KiB/CPU-s	0.025286	False	False
datadog_agent_remap_blackhole_acks	ingress throughput	2.0KiB/CPU-s	0.01	8.80%	31.26MiB/CPU-s	940.01KiB/CPU-s	11.57KiB/CPU-s	0.0	0.029361	31.26MiB/CPU-s	1.1MiB/CPU-s	13.9KiB/CPU-s	0.035268	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-8.1B/CPU-s	-0.00	0.13%	13.61MiB/CPU-s	269.69KiB/CPU-s	3.32KiB/CPU-s	0.0	0.019343	13.61MiB/CPU-s	268.69KiB/CPU-s	3.3KiB/CPU-s	0.019271	False	False
fluent_elasticsearch	ingress throughput	1.57KiB/CPU-s	0.00	99.73%	45.41MiB/CPU-s	29.87KiB/CPU-s	372.17B/CPU-s	0.0	0.000642	45.42MiB/CPU-s	30.85KiB/CPU-s	384.43B/CPU-s	0.000663	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-1.51KiB/CPU-s	-0.01	19.67%	13.62MiB/CPU-s	352.3KiB/CPU-s	4.33KiB/CPU-s	0.0	0.025267	13.61MiB/CPU-s	345.85KiB/CPU-s	4.25KiB/CPU-s	0.024807	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	-2.15KiB/CPU-s	-0.02	37.21%	13.62MiB/CPU-s	249.9KiB/CPU-s	3.07KiB/CPU-s	0.0	0.01792	13.62MiB/CPU-s	260.23KiB/CPU-s	3.2KiB/CPU-s	0.018664	False	False
http_to_http_json	ingress throughput	-39.69KiB/CPU-s	-0.28	100.00%	13.62MiB/CPU-s	218.12KiB/CPU-s	2.68KiB/CPU-s	0.0	0.015637	13.58MiB/CPU-s	280.1KiB/CPU-s	3.45KiB/CPU-s	0.020138	False	False
syslog_humio_logs	ingress throughput	-25.77KiB/CPU-s	-0.29	100.00%	8.63MiB/CPU-s	331.05KiB/CPU-s	4.07KiB/CPU-s	0.0	0.037448	8.61MiB/CPU-s	224.07KiB/CPU-s	2.76KiB/CPU-s	0.025421	False	False
splunk_hec_route_s3	ingress throughput	-55.02KiB/CPU-s	-0.47	100.00%	11.37MiB/CPU-s	599.24KiB/CPU-s	7.37KiB/CPU-s	0.0	0.051449	11.32MiB/CPU-s	678.92KiB/CPU-s	8.36KiB/CPU-s	0.058567	False	False
http_to_http_acks	ingress throughput	-34.95KiB/CPU-s	-0.64	52.42%	5.31MiB/CPU-s	2.76MiB/CPU-s	34.83KiB/CPU-s	0.0	0.520176	5.28MiB/CPU-s	2.74MiB/CPU-s	34.47KiB/CPU-s	0.518013	True	False
file_to_blackhole	egress throughput	-56.69KiB/CPU-s	-0.83	25.23%	6.69MiB/CPU-s	4.18MiB/CPU-s	128.59KiB/CPU-s	5.786618	0.623576	6.64MiB/CPU-s	4.08MiB/CPU-s	120.44KiB/CPU-s	0.614247	True	False
datadog_agent_remap_blackhole	ingress throughput	-267.69KiB/CPU-s	-0.84	100.00%	30.95MiB/CPU-s	1.53MiB/CPU-s	19.21KiB/CPU-s	0.0	0.049273	30.69MiB/CPU-s	1.29MiB/CPU-s	16.2KiB/CPU-s	0.041898	False	False
syslog_splunk_hec_logs	ingress throughput	-184.52KiB/CPU-s	-2.07	100.00%	8.73MiB/CPU-s	287.16KiB/CPU-s	3.53KiB/CPU-s	0.0	0.032138	8.55MiB/CPU-s	251.54KiB/CPU-s	3.09KiB/CPU-s	0.028745	False	False
datadog_agent_remap_datadog_logs_acks	ingress throughput	-828.82KiB/CPU-s	-2.48	100.00%	32.66MiB/CPU-s	1.33MiB/CPU-s	16.72KiB/CPU-s	0.0	0.040638	31.85MiB/CPU-s	1.51MiB/CPU-s	19.05KiB/CPU-s	0.047457	False	False
syslog_log2metric_humio_metrics	ingress throughput	-278.56KiB/CPU-s	-4.43	100.00%	6.14MiB/CPU-s	216.42KiB/CPU-s	2.66KiB/CPU-s	0.0	0.034416	5.87MiB/CPU-s	270.12KiB/CPU-s	3.32KiB/CPU-s	0.044947	False	False

spencergilbert

A small nit, we tend to group imports in this style - but it's not enforced (the option hasn't landed in stable).

scripts/signoff-git-hook.sh

vdev/src/commands/install_git_hooks.rs

vdev/src/commands/test_vrl.rs

vdev/src/commands/check/component_docs.rs

vdev/src/commands/install_git_hooks.rs

bruceg

This is a great start. Make sure to also replace the use of these scripts in, for example, the Makefile with running cargo vdev ….

vdev/src/commands/install_git_hooks.rs

vdev/src/commands/check/component_docs.rs

jsoref · 2023-03-02T21:03:20Z

check-spelling should be happy if you merge w/ master again.

…ndline

github-actions · 2023-03-02T23:06:55Z

Regression Detector Results

Run ID: 092eade9-d6b5-4e27-9355-bba8bc32eb3c
Baseline: 8747a69
Comparison: 24c3e91
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
otlp_grpc_to_blackhole	ingress throughput	36.1KiB/CPU-s	3.48	100.00%	1.01MiB/CPU-s	56.28KiB/CPU-s	709.02B/CPU-s	0.0	0.054274	1.05MiB/CPU-s	38.66KiB/CPU-s	487.23B/CPU-s	0.0	0.036023	False	False
datadog_agent_remap_datadog_logs_acks	ingress throughput	824.99KiB/CPU-s	2.48	100.00%	32.47MiB/CPU-s	1.24MiB/CPU-s	15.61KiB/CPU-s	0.0	0.038161	33.28MiB/CPU-s	798.95KiB/CPU-s	9.83KiB/CPU-s	0.0	0.023445	False	False
otlp_http_to_blackhole	ingress throughput	34.62KiB/CPU-s	2.22	100.00%	1.53MiB/CPU-s	123.16KiB/CPU-s	1.52KiB/CPU-s	0.0	0.07884	1.56MiB/CPU-s	105.81KiB/CPU-s	1.3KiB/CPU-s	0.0	0.066265	False	False
socket_to_socket_blackhole	ingress throughput	264.66KiB/CPU-s	1.98	100.00%	13.07MiB/CPU-s	347.26KiB/CPU-s	4.27KiB/CPU-s	0.0	0.025936	13.33MiB/CPU-s	297.26KiB/CPU-s	3.66KiB/CPU-s	0.0	0.021771	False	False
http_text_to_http_json	ingress throughput	437.09KiB/CPU-s	1.78	100.00%	24.03MiB/CPU-s	838.68KiB/CPU-s	10.32KiB/CPU-s	0.0	0.034075	24.46MiB/CPU-s	713.06KiB/CPU-s	8.78KiB/CPU-s	0.0	0.028465	False	False
syslog_loki	ingress throughput	141.5KiB/CPU-s	1.62	100.00%	8.52MiB/CPU-s	351.72KiB/CPU-s	4.33KiB/CPU-s	0.0	0.040293	8.66MiB/CPU-s	162.07KiB/CPU-s	1.99KiB/CPU-s	0.0	0.018271	False	False
datadog_agent_remap_datadog_logs	ingress throughput	404.94KiB/CPU-s	1.18	100.00%	33.38MiB/CPU-s	1.27MiB/CPU-s	16.01KiB/CPU-s	0.0	0.038055	33.78MiB/CPU-s	967.14KiB/CPU-s	11.9KiB/CPU-s	0.0	0.027958	False	False
syslog_splunk_hec_logs	ingress throughput	81.13KiB/CPU-s	0.94	100.00%	8.46MiB/CPU-s	371.91KiB/CPU-s	4.57KiB/CPU-s	0.0	0.042934	8.54MiB/CPU-s	225.61KiB/CPU-s	2.78KiB/CPU-s	0.0	0.025803	False	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	79.48KiB/CPU-s	0.88	100.00%	8.85MiB/CPU-s	376.82KiB/CPU-s	4.64KiB/CPU-s	0.0	0.041591	8.92MiB/CPU-s	204.5KiB/CPU-s	2.52KiB/CPU-s	0.0	0.022375	False	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	33.16KiB/CPU-s	0.86	100.00%	3.75MiB/CPU-s	448.62KiB/CPU-s	5.52KiB/CPU-s	0.0	0.116803	3.78MiB/CPU-s	431.72KiB/CPU-s	5.31KiB/CPU-s	0.0	0.11144	True	False
syslog_humio_logs	ingress throughput	49.82KiB/CPU-s	0.56	100.00%	8.68MiB/CPU-s	280.15KiB/CPU-s	3.45KiB/CPU-s	0.0	0.031533	8.72MiB/CPU-s	273.32KiB/CPU-s	3.36KiB/CPU-s	0.0	0.030592	False	False
http_to_http_acks	ingress throughput	7.57KiB/CPU-s	0.14	12.29%	5.22MiB/CPU-s	2.75MiB/CPU-s	34.63KiB/CPU-s	0.0	0.526857	5.22MiB/CPU-s	2.74MiB/CPU-s	34.58KiB/CPU-s	0.0	0.525344	True	False
enterprise_http_to_http	ingress throughput	5.56KiB/CPU-s	0.04	81.94%	13.62MiB/CPU-s	277.33KiB/CPU-s	3.41KiB/CPU-s	0.0	0.019888	13.62MiB/CPU-s	192.55KiB/CPU-s	2.37KiB/CPU-s	0.0	0.013803	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	480.5B/CPU-s	0.00	5.99%	13.61MiB/CPU-s	359.74KiB/CPU-s	4.42KiB/CPU-s	0.0	0.025804	13.61MiB/CPU-s	358.69KiB/CPU-s	4.41KiB/CPU-s	0.0	0.025728	False	False
fluent_elasticsearch	ingress throughput	364.84B/CPU-s	0.00	50.75%	45.41MiB/CPU-s	30.24KiB/CPU-s	376.88B/CPU-s	0.0	0.00065	45.41MiB/CPU-s	30.08KiB/CPU-s	374.87B/CPU-s	0.0	0.000647	False	False
http_to_http_json	ingress throughput	105.15B/CPU-s	0.00	2.14%	13.62MiB/CPU-s	219.85KiB/CPU-s	2.7KiB/CPU-s	0.0	0.015762	13.62MiB/CPU-s	220.63KiB/CPU-s	2.71KiB/CPU-s	0.0	0.015818	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	-753.4B/CPU-s	-0.01	12.90%	13.62MiB/CPU-s	260.21KiB/CPU-s	3.2KiB/CPU-s	0.0	0.018662	13.62MiB/CPU-s	260.8KiB/CPU-s	3.21KiB/CPU-s	0.0	0.018705	False	False
http_to_http_noack	ingress throughput	-1.5KiB/CPU-s	-0.01	19.93%	13.61MiB/CPU-s	334.54KiB/CPU-s	4.12KiB/CPU-s	0.0	0.024	13.61MiB/CPU-s	348.44KiB/CPU-s	4.29KiB/CPU-s	0.0	0.025	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-2.21KiB/CPU-s	-0.02	36.83%	13.62MiB/CPU-s	257.77KiB/CPU-s	3.17KiB/CPU-s	0.0	0.018485	13.61MiB/CPU-s	271.9KiB/CPU-s	3.34KiB/CPU-s	0.0	0.019502	False	False
syslog_log2metric_humio_metrics	ingress throughput	-20.96KiB/CPU-s	-0.33	100.00%	6.22MiB/CPU-s	136.77KiB/CPU-s	1.68KiB/CPU-s	0.0	0.021466	6.2MiB/CPU-s	127.94KiB/CPU-s	1.57KiB/CPU-s	0.0	0.020145	False	False
file_to_blackhole	egress throughput	-50.61KiB/CPU-s	-0.74	22.33%	6.72MiB/CPU-s	4.15MiB/CPU-s	122.94KiB/CPU-s	4.511278	0.618119	6.67MiB/CPU-s	4.23MiB/CPU-s	129.34KiB/CPU-s	3.925067	0.633966	True	False
datadog_agent_remap_blackhole_acks	ingress throughput	-229.69KiB/CPU-s	-0.74	100.00%	30.52MiB/CPU-s	1.57MiB/CPU-s	19.74KiB/CPU-s	0.0	0.05132	30.29MiB/CPU-s	940.97KiB/CPU-s	11.58KiB/CPU-s	0.0	0.030332	False	False
splunk_hec_route_s3	ingress throughput	-122.79KiB/CPU-s	-1.04	100.00%	11.49MiB/CPU-s	601.96KiB/CPU-s	7.4KiB/CPU-s	0.0	0.051141	11.37MiB/CPU-s	573.21KiB/CPU-s	7.05KiB/CPU-s	0.0	0.049212	False	False
datadog_agent_remap_blackhole	ingress throughput	-336.27KiB/CPU-s	-1.07	100.00%	30.76MiB/CPU-s	1.6MiB/CPU-s	20.19KiB/CPU-s	0.0	0.052121	30.43MiB/CPU-s	1.2MiB/CPU-s	15.13KiB/CPU-s	0.0	0.039469	False	False

vdev/src/commands/meta/install_git_hooks.rs

bruceg

I left a couple of requests for fixes below, and it looks like this needs to be merged with master to pull in spelling checking fixes. Otherwise these should be good to go.

vdev/src/commands/check/component_docs.rs

vdev/src/commands/meta/install_git_hooks.rs

vdev/src/git.rs

github-actions · 2023-03-08T09:23:47Z

Regression Detector Results

Run ID: 110bbab3-e1cd-467c-904d-fefb03192c07
Baseline: 1834ed0
Comparison: e350d3e
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
file_to_blackhole	egress throughput	1.12MiB/CPU-s	19.81	100.00%	5.64MiB/CPU-s	4.36MiB/CPU-s	113.31KiB/CPU-s	0.772905	6.76MiB/CPU-s	4.2MiB/CPU-s	125.81KiB/CPU-s	4.794521	0.620906	True	True
syslog_regex_logs2metric_ddmetrics	ingress throughput	77.45KiB/CPU-s	2.10	100.00%	3.61MiB/CPU-s	335.1KiB/CPU-s	4.12KiB/CPU-s	0.090749	3.68MiB/CPU-s	413.76KiB/CPU-s	5.09KiB/CPU-s	0.0	0.10975	True	True
datadog_agent_remap_datadog_logs_acks	ingress throughput	450.19KiB/CPU-s	1.36	100.00%	32.4MiB/CPU-s	1.33MiB/CPU-s	16.74KiB/CPU-s	0.041002	32.84MiB/CPU-s	1.12MiB/CPU-s	14.11KiB/CPU-s	0.0	0.034096	False	False
syslog_log2metric_humio_metrics	ingress throughput	64.96KiB/CPU-s	1.06	100.00%	5.96MiB/CPU-s	296.93KiB/CPU-s	3.65KiB/CPU-s	0.048646	6.02MiB/CPU-s	138.45KiB/CPU-s	1.7KiB/CPU-s	0.0	0.022444	False	False
datadog_agent_remap_blackhole_acks	ingress throughput	300.0KiB/CPU-s	0.95	100.00%	30.78MiB/CPU-s	1.09MiB/CPU-s	13.69KiB/CPU-s	0.035301	31.08MiB/CPU-s	843.64KiB/CPU-s	10.38KiB/CPU-s	0.0	0.026508	False	False
http_to_http_noack	ingress throughput	3.82KiB/CPU-s	0.03	46.07%	13.61MiB/CPU-s	373.65KiB/CPU-s	4.6KiB/CPU-s	0.026813	13.61MiB/CPU-s	340.05KiB/CPU-s	4.18KiB/CPU-s	0.0	0.024395	False	False
enterprise_http_to_http	ingress throughput	1.91KiB/CPU-s	0.01	47.64%	13.62MiB/CPU-s	190.04KiB/CPU-s	2.34KiB/CPU-s	0.013623	13.62MiB/CPU-s	151.97KiB/CPU-s	1.87KiB/CPU-s	0.0	0.010892	False	False
http_to_http_json	ingress throughput	-667.17B/CPU-s	-0.00	13.51%	13.62MiB/CPU-s	218.79KiB/CPU-s	2.69KiB/CPU-s	0.015686	13.62MiB/CPU-s	221.5KiB/CPU-s	2.72KiB/CPU-s	0.0	0.015881	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	-371.79B/CPU-s	-0.00	6.51%	13.62MiB/CPU-s	254.4KiB/CPU-s	3.13KiB/CPU-s	0.018244	13.62MiB/CPU-s	256.68KiB/CPU-s	3.16KiB/CPU-s	0.0	0.018408	False	False
fluent_elasticsearch	ingress throughput	282.43B/CPU-s	0.00	40.51%	45.41MiB/CPU-s	30.0KiB/CPU-s	373.99B/CPU-s	0.000645	45.41MiB/CPU-s	30.26KiB/CPU-s	377.2B/CPU-s	0.0	0.000651	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-896.67B/CPU-s	-0.01	11.08%	13.61MiB/CPU-s	358.37KiB/CPU-s	4.41KiB/CPU-s	0.025703	13.61MiB/CPU-s	364.39KiB/CPU-s	4.48KiB/CPU-s	0.0	0.026137	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-1008.99B/CPU-s	-0.01	16.47%	13.61MiB/CPU-s	270.32KiB/CPU-s	3.32KiB/CPU-s	0.019388	13.61MiB/CPU-s	274.55KiB/CPU-s	3.38KiB/CPU-s	0.0	0.019693	False	False
otlp_grpc_to_blackhole	ingress throughput	-189.79B/CPU-s	-0.02	17.37%	1.03MiB/CPU-s	50.32KiB/CPU-s	633.9B/CPU-s	0.047534	1.03MiB/CPU-s	46.71KiB/CPU-s	588.47B/CPU-s	0.0	0.044128	False	False
otlp_http_to_blackhole	ingress throughput	-833.6B/CPU-s	-0.05	30.63%	1.53MiB/CPU-s	123.89KiB/CPU-s	1.52KiB/CPU-s	0.078871	1.53MiB/CPU-s	113.54KiB/CPU-s	1.4KiB/CPU-s	0.0	0.072318	False	False
syslog_splunk_hec_logs	ingress throughput	-8.16KiB/CPU-s	-0.09	98.47%	8.8MiB/CPU-s	203.96KiB/CPU-s	2.51KiB/CPU-s	0.022643	8.79MiB/CPU-s	182.41KiB/CPU-s	2.24KiB/CPU-s	0.0	0.020269	False	False
http_to_http_acks	ingress throughput	-9.97KiB/CPU-s	-0.18	15.96%	5.29MiB/CPU-s	2.75MiB/CPU-s	34.64KiB/CPU-s	0.519478	5.28MiB/CPU-s	2.8MiB/CPU-s	35.35KiB/CPU-s	0.0	0.530903	True	False
datadog_agent_remap_datadog_logs	ingress throughput	-123.5KiB/CPU-s	-0.36	100.00%	33.67MiB/CPU-s	988.27KiB/CPU-s	12.16KiB/CPU-s	0.028661	33.55MiB/CPU-s	965.35KiB/CPU-s	11.88KiB/CPU-s	0.0	0.028097	False	False
socket_to_socket_blackhole	ingress throughput	-77.69KiB/CPU-s	-0.57	100.00%	13.24MiB/CPU-s	301.58KiB/CPU-s	3.71KiB/CPU-s	0.022245	13.16MiB/CPU-s	268.46KiB/CPU-s	3.3KiB/CPU-s	0.0	0.019916	False	False
syslog_loki	ingress throughput	-60.5KiB/CPU-s	-0.69	100.00%	8.56MiB/CPU-s	252.73KiB/CPU-s	3.11KiB/CPU-s	0.028841	8.5MiB/CPU-s	218.65KiB/CPU-s	2.69KiB/CPU-s	0.0	0.025126	False	False
http_text_to_http_json	ingress throughput	-219.7KiB/CPU-s	-0.88	100.00%	24.47MiB/CPU-s	605.71KiB/CPU-s	7.45KiB/CPU-s	0.024175	24.25MiB/CPU-s	605.98KiB/CPU-s	7.46KiB/CPU-s	0.0	0.0244	False	False
splunk_hec_route_s3	ingress throughput	-105.13KiB/CPU-s	-0.89	100.00%	11.51MiB/CPU-s	547.3KiB/CPU-s	6.73KiB/CPU-s	0.046423	11.41MiB/CPU-s	698.32KiB/CPU-s	8.59KiB/CPU-s	0.0	0.059766	False	False
datadog_agent_remap_blackhole	ingress throughput	-321.08KiB/CPU-s	-1.01	100.00%	31.12MiB/CPU-s	1.1MiB/CPU-s	13.87KiB/CPU-s	0.035376	30.81MiB/CPU-s	1.27MiB/CPU-s	16.03KiB/CPU-s	0.0	0.04128	False	False
syslog_humio_logs	ingress throughput	-92.89KiB/CPU-s	-1.03	100.00%	8.82MiB/CPU-s	157.42KiB/CPU-s	1.94KiB/CPU-s	0.01743	8.73MiB/CPU-s	277.23KiB/CPU-s	3.41KiB/CPU-s	0.0	0.031015	False	False
syslog_log2metric_splunk_hec_metrics	ingress throughput	-344.38KiB/CPU-s	-3.68	100.00%	9.15MiB/CPU-s	209.49KiB/CPU-s	2.58KiB/CPU-s	0.022363	8.81MiB/CPU-s	386.3KiB/CPU-s	4.75KiB/CPU-s	0.0	0.042814	False	False

vdev/src/commands/meta/install_git_hooks.rs

rustfmt nightly options aren't enforced yet and was discussed it's not super important to have this resolved for now

github-actions · 2023-03-08T23:07:13Z

Regression Detector Results

Run ID: 00116545-f3b4-4e64-8c01-1fb24d98705f
Baseline: 152e729
Comparison: da4337a
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
syslog_log2metric_splunk_hec_metrics	ingress throughput	143.4KiB/CPU-s	1.55	100.00%	9.01MiB/CPU-s	312.43KiB/CPU-s	3.85KiB/CPU-s	0.033846	9.15MiB/CPU-s	270.73KiB/CPU-s	3.33KiB/CPU-s	0.0	0.028879	False	False
syslog_regex_logs2metric_ddmetrics	ingress throughput	44.62KiB/CPU-s	1.18	100.00%	3.68MiB/CPU-s	384.97KiB/CPU-s	4.74KiB/CPU-s	0.102183	3.72MiB/CPU-s	418.81KiB/CPU-s	5.15KiB/CPU-s	0.0	0.109864	True	True
http_to_http_acks	ingress throughput	34.14KiB/CPU-s	0.64	51.53%	5.25MiB/CPU-s	2.74MiB/CPU-s	34.55KiB/CPU-s	0.522529	5.28MiB/CPU-s	2.74MiB/CPU-s	34.55KiB/CPU-s	0.0	0.519219	True	False
otlp_http_to_blackhole	ingress throughput	5.35KiB/CPU-s	0.34	98.95%	1.53MiB/CPU-s	124.8KiB/CPU-s	1.53KiB/CPU-s	0.079504	1.54MiB/CPU-s	115.33KiB/CPU-s	1.42KiB/CPU-s	0.0	0.073224	False	False
enterprise_http_to_http	ingress throughput	3.67KiB/CPU-s	0.03	73.87%	13.62MiB/CPU-s	217.84KiB/CPU-s	2.68KiB/CPU-s	0.015618	13.62MiB/CPU-s	151.7KiB/CPU-s	1.87KiB/CPU-s	0.0	0.010873	False	False
http_to_http_noack	ingress throughput	1.81KiB/CPU-s	0.01	21.79%	13.61MiB/CPU-s	386.0KiB/CPU-s	4.75KiB/CPU-s	0.027703	13.61MiB/CPU-s	367.4KiB/CPU-s	4.52KiB/CPU-s	0.0	0.026365	False	False
splunk_hec_indexer_ack_blackhole	ingress throughput	-632.18B/CPU-s	-0.00	10.92%	13.62MiB/CPU-s	255.73KiB/CPU-s	3.15KiB/CPU-s	0.018341	13.61MiB/CPU-s	261.47KiB/CPU-s	3.22KiB/CPU-s	0.0	0.018753	False	False
fluent_elasticsearch	ingress throughput	295.28B/CPU-s	0.00	41.95%	45.41MiB/CPU-s	30.16KiB/CPU-s	375.78B/CPU-s	0.000649	45.41MiB/CPU-s	30.47KiB/CPU-s	379.7B/CPU-s	0.0	0.000655	False	False
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-819.88B/CPU-s	-0.01	10.01%	13.61MiB/CPU-s	362.93KiB/CPU-s	4.46KiB/CPU-s	0.026031	13.61MiB/CPU-s	368.81KiB/CPU-s	4.54KiB/CPU-s	0.0	0.026455	False	False
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-4.46KiB/CPU-s	-0.03	68.27%	13.62MiB/CPU-s	237.84KiB/CPU-s	2.93KiB/CPU-s	0.017054	13.61MiB/CPU-s	273.62KiB/CPU-s	3.37KiB/CPU-s	0.0	0.019626	False	False
syslog_splunk_hec_logs	ingress throughput	-17.85KiB/CPU-s	-0.20	99.86%	8.62MiB/CPU-s	254.84KiB/CPU-s	3.13KiB/CPU-s	0.028857	8.61MiB/CPU-s	376.37KiB/CPU-s	4.63KiB/CPU-s	0.0	0.042704	False	False
http_to_http_json	ingress throughput	-62.26KiB/CPU-s	-0.45	100.00%	13.62MiB/CPU-s	214.73KiB/CPU-s	2.64KiB/CPU-s	0.015393	13.56MiB/CPU-s	314.42KiB/CPU-s	3.87KiB/CPU-s	0.0	0.02264	False	False
syslog_log2metric_humio_metrics	ingress throughput	-29.73KiB/CPU-s	-0.49	100.00%	5.91MiB/CPU-s	196.18KiB/CPU-s	2.41KiB/CPU-s	0.032386	5.89MiB/CPU-s	197.28KiB/CPU-s	2.43KiB/CPU-s	0.0	0.032729	False	False
syslog_humio_logs	ingress throughput	-48.64KiB/CPU-s	-0.54	100.00%	8.84MiB/CPU-s	180.42KiB/CPU-s	2.22KiB/CPU-s	0.019927	8.79MiB/CPU-s	225.0KiB/CPU-s	2.77KiB/CPU-s	0.0	0.024985	False	False
datadog_agent_remap_datadog_logs	ingress throughput	-196.25KiB/CPU-s	-0.57	100.00%	33.7MiB/CPU-s	957.07KiB/CPU-s	11.78KiB/CPU-s	0.027729	33.51MiB/CPU-s	1.21MiB/CPU-s	15.19KiB/CPU-s	0.0	0.035975	False	False
datadog_agent_remap_datadog_logs_acks	ingress throughput	-311.2KiB/CPU-s	-0.93	100.00%	32.8MiB/CPU-s	1.04MiB/CPU-s	13.13KiB/CPU-s	0.031761	32.5MiB/CPU-s	1.4MiB/CPU-s	17.66KiB/CPU-s	0.0	0.043134	False	False
datadog_agent_remap_blackhole	ingress throughput	-346.48KiB/CPU-s	-1.09	100.00%	31.06MiB/CPU-s	1.12MiB/CPU-s	14.16KiB/CPU-s	0.036199	30.72MiB/CPU-s	1.23MiB/CPU-s	15.52KiB/CPU-s	0.0	0.040106	False	False
splunk_hec_route_s3	ingress throughput	-154.12KiB/CPU-s	-1.28	100.00%	11.75MiB/CPU-s	537.81KiB/CPU-s	6.62KiB/CPU-s	0.044684	11.6MiB/CPU-s	524.66KiB/CPU-s	6.46KiB/CPU-s	0.0	0.044157	False	False
syslog_loki	ingress throughput	-131.83KiB/CPU-s	-1.51	100.00%	8.54MiB/CPU-s	266.27KiB/CPU-s	3.27KiB/CPU-s	0.03046	8.41MiB/CPU-s	277.28KiB/CPU-s	3.41KiB/CPU-s	0.0	0.032206	False	False
otlp_grpc_to_blackhole	ingress throughput	-18.91KiB/CPU-s	-1.77	100.00%	1.05MiB/CPU-s	45.8KiB/CPU-s	577.08B/CPU-s	0.042776	1.03MiB/CPU-s	45.99KiB/CPU-s	579.67B/CPU-s	0.0	0.043731	False	False
datadog_agent_remap_blackhole_acks	ingress throughput	-690.14KiB/CPU-s	-2.21	100.00%	30.49MiB/CPU-s	1.83MiB/CPU-s	23.06KiB/CPU-s	0.060015	29.82MiB/CPU-s	1.63MiB/CPU-s	20.55KiB/CPU-s	0.0	0.054689	False	False
http_text_to_http_json	ingress throughput	-906.75KiB/CPU-s	-3.52	100.00%	25.12MiB/CPU-s	1009.6KiB/CPU-s	12.42KiB/CPU-s	0.039244	24.24MiB/CPU-s	907.97KiB/CPU-s	11.17KiB/CPU-s	0.0	0.036583	False	False
socket_to_socket_blackhole	ingress throughput	-508.63KiB/CPU-s	-3.66	100.00%	13.58MiB/CPU-s	286.75KiB/CPU-s	3.53KiB/CPU-s	0.020625	13.08MiB/CPU-s	317.51KiB/CPU-s	3.91KiB/CPU-s	0.0	0.023705	False	False
file_to_blackhole	egress throughput	-391.65KiB/CPU-s	-5.86	97.82%	6.53MiB/CPU-s	4.01MiB/CPU-s	125.18KiB/CPU-s	0.613648	6.14MiB/CPU-s	4.14MiB/CPU-s	116.02KiB/CPU-s	2.76739	0.674136	True	True

jonathanpv added 2 commits March 2, 2023 02:40

Add test-vrl command to vdev

230ebe6

Rewrite test-vrl.sh and install_git_hooks.sh in vdev

6aab34a

Signed-off-by: jonathanpv <jonpad512@gmail.com>

jonathanpv added ci-condition: integration tests enable Run integration tests on this PR domain: vdev Anything related to the vdev tooling labels Mar 2, 2023

jonathanpv self-assigned this Mar 2, 2023

jonathanpv marked this pull request as draft March 2, 2023 08:31

github-actions bot added the domain: ci Anything related to Vector's CI environment label Mar 2, 2023

jonathanpv added 3 commits March 2, 2023 03:58

clippy check

79d1aee

Signed-off-by: jonathanpv <jonpad512@gmail.com>

Remove make test-vrl and use cargo vdev test-vrl

97b16b0

Signed-off-by: jonathanpv <jonpad512@gmail.com>

clippy checks

9c82815

Signed-off-by: jonathanpv <jonpad512@gmail.com>

Rewrite release-github.rs into vdev

278fd74

Signed-off-by: jonathanpv <jonpad512@gmail.com>

jszwedko reviewed Mar 2, 2023

View reviewed changes

Rewrite check-component-docs to vdev

c08f74c

Signed-off-by: jonathanpv <jonpad512@gmail.com>

github-actions bot added the domain: releasing Anything related to releasing Vector label Mar 2, 2023

jonathanpv requested a review from bruceg March 2, 2023 14:15

jonathanpv marked this pull request as ready for review March 2, 2023 15:10

jonathanpv requested a review from spencergilbert March 2, 2023 16:11

spencergilbert previously requested changes Mar 2, 2023

View reviewed changes

bruceg reviewed Mar 2, 2023

View reviewed changes

vdev/src/commands/install_git_hooks.rs Outdated Show resolved Hide resolved

bruceg reviewed Mar 2, 2023

View reviewed changes

Embed signoff script to install_git_hooks.rs, delete signoff-git-hook.sh

8285d29

Move install-git-hooks to meta subcommand, remove argument from comma…

edb551d

…ndline

jonathanpv added 2 commits March 2, 2023 17:09

Create git helper function and clean up component_docs.rs

b8a97dd

Remove runtime ast for test-vrl, remove clippy::case_sensitive allow

24c3e91

jonathanpv requested review from bruceg and spencergilbert March 3, 2023 14:37

spencergilbert reviewed Mar 3, 2023

View reviewed changes

vdev/src/commands/meta/install_git_hooks.rs Outdated Show resolved Hide resolved

bruceg requested changes Mar 4, 2023

View reviewed changes

vdev/src/commands/check/component_docs.rs Outdated Show resolved Hide resolved

vdev/src/commands/meta/install_git_hooks.rs Outdated Show resolved Hide resolved

vdev/src/git.rs Outdated Show resolved Hide resolved

jonathanpv added 3 commits March 8, 2023 02:59

use into_iter() for component_docs.rs

9124f4f

remove get_git_dir() and use create_dir_all() instead of mkdir

52df619

Merge branch 'master' into jonathanpv/vdev-script-wrappers

e350d3e

jonathanpv requested review from spencergilbert and bruceg March 8, 2023 08:40

bruceg approved these changes Mar 8, 2023

View reviewed changes

jonathanpv commented Mar 8, 2023

View reviewed changes

vdev/src/commands/meta/install_git_hooks.rs Show resolved Hide resolved

jonathanpv commented Mar 8, 2023

View reviewed changes

vdev/src/commands/meta/install_git_hooks.rs Outdated Show resolved Hide resolved

add configuration conditional on execute bit procedure

da4337a

spencergilbert approved these changes Mar 9, 2023

View reviewed changes

jonathanpv merged commit 3e79d63 into master Mar 9, 2023

jonathanpv deleted the jonathanpv/vdev-script-wrappers branch March 9, 2023 18:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(vdev): Rewrite scripts in vdev #16661

chore(vdev): Rewrite scripts in vdev #16661

jonathanpv commented Mar 2, 2023 •

edited

Loading

netlify bot commented Mar 2, 2023 •

edited

Loading

netlify bot commented Mar 2, 2023 •

edited

Loading

github-actions bot commented Mar 2, 2023

github-actions bot commented Mar 2, 2023

jszwedko left a comment

jonathanpv commented Mar 2, 2023 •

edited

Loading

jonathanpv commented Mar 2, 2023

github-actions bot commented Mar 2, 2023

spencergilbert left a comment

bruceg left a comment

jsoref commented Mar 2, 2023

github-actions bot commented Mar 2, 2023

bruceg left a comment

github-actions bot commented Mar 8, 2023

github-actions bot commented Mar 8, 2023

chore(vdev): Rewrite scripts in vdev #16661

chore(vdev): Rewrite scripts in vdev #16661

Conversation

jonathanpv commented Mar 2, 2023 • edited Loading

Problem

Solution

Testing

cargo vdev test-vrl

cargo vdev meta install-git-hooks

cargo vdev release github

cargo vdev check component-docs

Related Issues

Checklist

netlify bot commented Mar 2, 2023 • edited Loading

✅ Deploy Preview for vector-project canceled.

netlify bot commented Mar 2, 2023 • edited Loading

✅ Deploy Preview for vrl-playground canceled.

github-actions bot commented Mar 2, 2023

Regression Detector Results

github-actions bot commented Mar 2, 2023

Regression Detector Results

jszwedko left a comment

Choose a reason for hiding this comment

jonathanpv commented Mar 2, 2023 • edited Loading

jonathanpv commented Mar 2, 2023

github-actions bot commented Mar 2, 2023

Regression Detector Results

spencergilbert left a comment

Choose a reason for hiding this comment

bruceg left a comment

Choose a reason for hiding this comment

jsoref commented Mar 2, 2023

github-actions bot commented Mar 2, 2023

Regression Detector Results

bruceg left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 8, 2023

Regression Detector Results

github-actions bot commented Mar 8, 2023

Regression Detector Results

jonathanpv commented Mar 2, 2023 •

edited

Loading

`cargo vdev test-vrl`

`cargo vdev meta install-git-hooks`

`cargo vdev release github`

`cargo vdev check component-docs`

netlify bot commented Mar 2, 2023 •

edited

Loading

netlify bot commented Mar 2, 2023 •

edited

Loading

jonathanpv commented Mar 2, 2023 •

edited

Loading