Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(vdev): Rewrite release-homebrew into vdev #16772

Merged
merged 10 commits into from
Apr 3, 2023

Conversation

jonathanpv
Copy link
Contributor

@jonathanpv jonathanpv commented Mar 12, 2023

Problem

We would like to rewrite the release-homebrew.sh script to native Rust in vdev

Solution

  • Rewrote release-homebrew.sh into vdev release subcommand
  • Deleted release-homebrew.sh

Testing

cargo vdev release homebrew

Related Issues

Checklist

  • Add the ci-condition integration tests enable
  • Delete any rewritten scripts
  • Modify makefile to point to the new cargo vdev
  • Modify workflow if need be
  • Compare the output of the local script to the newly created cargo vdev Rust version
  • All CI integration tests passing
  • Test newly created cargo vdev commands

@jonathanpv jonathanpv added platform: homebrew Anything `homebrew` platform related ci-condition: integration tests enable Run integration tests on this PR domain: vdev Anything related to the vdev tooling labels Mar 12, 2023
@jonathanpv jonathanpv requested a review from bruceg March 12, 2023 03:43
@jonathanpv jonathanpv self-assigned this Mar 12, 2023
@netlify
Copy link

netlify bot commented Mar 12, 2023

Deploy Preview for vector-project canceled.

Name Link
🔨 Latest commit c8f8178
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/6428dda0c6387e00081d1561

@netlify
Copy link

netlify bot commented Mar 12, 2023

Deploy Preview for vrl-playground ready!

Name Link
🔨 Latest commit 195dd5e
🔍 Latest deploy log https://app.netlify.com/sites/vrl-playground/deploys/6419ef2a207039000781aa89
😎 Deploy Preview https://deploy-preview-16772--vrl-playground.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions github-actions bot added domain: ci Anything related to Vector's CI environment domain: releasing Anything related to releasing Vector labels Mar 12, 2023
@jonathanpv
Copy link
Contributor Author

jonathanpv commented Mar 12, 2023

I feel like this PR is the closest I've come to nailing it on the first try. I think my experience with vdev has helped me get better at writing code that meets the project's standards, and I've been able to respond more quickly to feedback from previous PRs. Definitely feeling more confident in my abilities and excited to keep improving

@github-actions
Copy link

Regression Detector Results

Run ID: 6887ace7-f7d2-4932-ae6d-e1b8efc71a74
Baseline: a01ed48
Comparison: dbfba06
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

Changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%:

experiment goal Δ mean Δ mean % confidence
http_text_to_http_json ingress throughput 1.83MiB/CPU-s 7.64 100.00%
Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_text_to_http_json ingress throughput 1.83MiB/CPU-s 7.64 100.00% 24.01MiB/CPU-s 570.02KiB/CPU-s 7.01KiB/CPU-s 0.0 0.023181 25.85MiB/CPU-s 586.6KiB/CPU-s 7.22KiB/CPU-s 0.0 0.022162 False False
socket_to_socket_blackhole ingress throughput 300.21KiB/CPU-s 2.23 100.00% 13.12MiB/CPU-s 252.46KiB/CPU-s 3.11KiB/CPU-s 0.0 0.018793 13.41MiB/CPU-s 280.21KiB/CPU-s 3.45KiB/CPU-s 0.0 0.020403 False False
syslog_regex_logs2metric_ddmetrics ingress throughput 70.19KiB/CPU-s 1.95 100.00% 3.51MiB/CPU-s 326.13KiB/CPU-s 4.01KiB/CPU-s 0.0 0.09062 3.58MiB/CPU-s 321.0KiB/CPU-s 3.95KiB/CPU-s 0.0 0.087489 False True
file_to_blackhole egress throughput 87.36KiB/CPU-s 1.49 41.30% 5.74MiB/CPU-s 4.28MiB/CPU-s 113.88KiB/CPU-s 0.0 0.745709 5.83MiB/CPU-s 4.47MiB/CPU-s 113.57KiB/CPU-s 0.0 0.766858 True True
datadog_agent_remap_datadog_logs ingress throughput 376.96KiB/CPU-s 1.13 100.00% 32.66MiB/CPU-s 1.03MiB/CPU-s 13.0KiB/CPU-s 0.0 0.031606 33.03MiB/CPU-s 1.14MiB/CPU-s 14.38KiB/CPU-s 0.0 0.034561 False False
otlp_http_to_blackhole ingress throughput 12.2KiB/CPU-s 0.78 100.00% 1.52MiB/CPU-s 122.06KiB/CPU-s 1.5KiB/CPU-s 0.0 0.078273 1.53MiB/CPU-s 112.61KiB/CPU-s 1.39KiB/CPU-s 0.0 0.07165 False False
datadog_agent_remap_datadog_logs_acks ingress throughput 250.52KiB/CPU-s 0.75 100.00% 32.48MiB/CPU-s 1.1MiB/CPU-s 13.88KiB/CPU-s 0.0 0.033925 32.73MiB/CPU-s 1.27MiB/CPU-s 16.06KiB/CPU-s 0.0 0.038955 False False
syslog_log2metric_humio_metrics ingress throughput 10.96KiB/CPU-s 0.18 100.00% 6.01MiB/CPU-s 140.58KiB/CPU-s 1.73KiB/CPU-s 0.0 0.022831 6.02MiB/CPU-s 161.47KiB/CPU-s 1.99KiB/CPU-s 0.0 0.026178 False False
syslog_humio_logs ingress throughput 6.8KiB/CPU-s 0.08 95.61% 8.8MiB/CPU-s 175.79KiB/CPU-s 2.16KiB/CPU-s 0.0 0.019514 8.8MiB/CPU-s 210.53KiB/CPU-s 2.59KiB/CPU-s 0.0 0.023353 False False
enterprise_http_to_http ingress throughput 5.37KiB/CPU-s 0.04 81.07% 13.62MiB/CPU-s 270.41KiB/CPU-s 3.33KiB/CPU-s 0.0 0.019392 13.62MiB/CPU-s 193.34KiB/CPU-s 2.38KiB/CPU-s 0.0 0.013859 False False
http_to_http_noack ingress throughput 3.69KiB/CPU-s 0.03 43.17% 13.61MiB/CPU-s 388.57KiB/CPU-s 4.78KiB/CPU-s 0.0 0.027887 13.61MiB/CPU-s 354.39KiB/CPU-s 4.36KiB/CPU-s 0.0 0.025428 False False
splunk_hec_indexer_ack_blackhole ingress throughput 225.19B/CPU-s 0.00 3.91% 13.62MiB/CPU-s 257.02KiB/CPU-s 3.16KiB/CPU-s 0.0 0.018433 13.62MiB/CPU-s 258.37KiB/CPU-s 3.18KiB/CPU-s 0.0 0.01853 False False
fluent_elasticsearch ingress throughput 249.2B/CPU-s 0.00 36.22% 45.41MiB/CPU-s 29.87KiB/CPU-s 372.16B/CPU-s 0.0 0.000642 45.41MiB/CPU-s 30.21KiB/CPU-s 376.48B/CPU-s 0.0 0.000649 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -1005.16B/CPU-s -0.01 16.65% 13.62MiB/CPU-s 265.94KiB/CPU-s 3.27KiB/CPU-s 0.0 0.019073 13.61MiB/CPU-s 270.78KiB/CPU-s 3.33KiB/CPU-s 0.0 0.019421 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput -1.38KiB/CPU-s -0.01 17.26% 13.61MiB/CPU-s 363.09KiB/CPU-s 4.47KiB/CPU-s 0.0 0.026042 13.61MiB/CPU-s 363.36KiB/CPU-s 4.47KiB/CPU-s 0.0 0.026064 False False
splunk_hec_route_s3 ingress throughput -21.3KiB/CPU-s -0.18 96.84% 11.6MiB/CPU-s 558.75KiB/CPU-s 6.87KiB/CPU-s 0.0 0.047051 11.58MiB/CPU-s 579.86KiB/CPU-s 7.13KiB/CPU-s 0.0 0.048917 False False
otlp_grpc_to_blackhole ingress throughput -2.71KiB/CPU-s -0.26 99.94% 1.03MiB/CPU-s 44.14KiB/CPU-s 556.3B/CPU-s 0.0 0.041741 1.03MiB/CPU-s 46.17KiB/CPU-s 581.77B/CPU-s 0.0 0.043768 False False
http_to_http_json ingress throughput -77.67KiB/CPU-s -0.56 100.00% 13.59MiB/CPU-s 265.17KiB/CPU-s 3.26KiB/CPU-s 0.0 0.019057 13.51MiB/CPU-s 374.85KiB/CPU-s 4.61KiB/CPU-s 0.0 0.027092 False False
syslog_splunk_hec_logs ingress throughput -57.33KiB/CPU-s -0.64 100.00% 8.71MiB/CPU-s 300.03KiB/CPU-s 3.69KiB/CPU-s 0.0 0.033646 8.65MiB/CPU-s 288.88KiB/CPU-s 3.55KiB/CPU-s 0.0 0.032605 False False
http_to_http_acks ingress throughput -35.22KiB/CPU-s -0.65 53.43% 5.3MiB/CPU-s 2.68MiB/CPU-s 33.81KiB/CPU-s 0.0 0.505806 5.27MiB/CPU-s 2.73MiB/CPU-s 34.46KiB/CPU-s 0.0 0.518919 True False
datadog_agent_remap_blackhole ingress throughput -414.77KiB/CPU-s -1.29 100.00% 31.48MiB/CPU-s 1.13MiB/CPU-s 14.25KiB/CPU-s 0.0 0.035949 31.07MiB/CPU-s 782.05KiB/CPU-s 9.62KiB/CPU-s 0.0 0.024578 False False
syslog_log2metric_splunk_hec_metrics ingress throughput -122.42KiB/CPU-s -1.32 100.00% 9.09MiB/CPU-s 223.62KiB/CPU-s 2.75KiB/CPU-s 0.0 0.024024 8.97MiB/CPU-s 327.5KiB/CPU-s 4.03KiB/CPU-s 0.0 0.035653 False False
syslog_loki ingress throughput -192.35KiB/CPU-s -2.22 100.00% 8.46MiB/CPU-s 222.41KiB/CPU-s 2.74KiB/CPU-s 0.0 0.025676 8.27MiB/CPU-s 323.39KiB/CPU-s 3.98KiB/CPU-s 0.0 0.038181 False False
datadog_agent_remap_blackhole_acks ingress throughput -1013.29KiB/CPU-s -3.16 100.00% 31.27MiB/CPU-s 1.34MiB/CPU-s 16.82KiB/CPU-s 0.0 0.042694 30.28MiB/CPU-s 1.31MiB/CPU-s 16.55KiB/CPU-s 0.0 0.043376 False False

Copy link
Member

@bruceg bruceg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty good. I mostly have style type comments on the new code, but there is a blocking issue with catching the exit codes from the git commands.

vdev/src/git.rs Outdated Show resolved Hide resolved
vdev/src/git.rs Outdated Show resolved Hide resolved
vdev/src/commands/release/homebrew.rs Outdated Show resolved Hide resolved
vdev/src/commands/release/homebrew.rs Outdated Show resolved Hide resolved
vdev/src/commands/release/homebrew.rs Outdated Show resolved Hide resolved
vdev/src/commands/release/homebrew.rs Outdated Show resolved Hide resolved
vdev/src/git.rs Outdated Show resolved Hide resolved
vdev/src/git.rs Outdated Show resolved Hide resolved
vdev/src/git.rs Outdated Show resolved Hide resolved
vdev/src/git.rs Outdated Show resolved Hide resolved
@jonathanpv jonathanpv requested a review from bruceg March 15, 2023 13:41
vdev/src/git.rs Outdated Show resolved Hide resolved
vdev/src/git.rs Outdated Show resolved Hide resolved
vdev/src/commands/release/homebrew.rs Outdated Show resolved Hide resolved
vdev/src/commands/release/homebrew.rs Outdated Show resolved Hide resolved
vdev/src/commands/release/homebrew.rs Outdated Show resolved Hide resolved
@jonathanpv jonathanpv requested a review from bruceg March 16, 2023 03:05
@github-actions
Copy link

Regression Detector Results

Run ID: 49c38410-b764-4c55-ab17-9ff0cb038b8c
Baseline: be9e2c4
Comparison: 742aa7f
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
file_to_blackhole egress throughput 659.93KiB/CPU-s 11.19 99.99% 5.76MiB/CPU-s 4.29MiB/CPU-s 110.81KiB/CPU-s 0.0 0.744059 6.4MiB/CPU-s 4.12MiB/CPU-s 124.83KiB/CPU-s 0.0 0.642783 True True
syslog_loki ingress throughput 93.9KiB/CPU-s 1.11 100.00% 8.26MiB/CPU-s 283.58KiB/CPU-s 3.49KiB/CPU-s 0.0 0.033533 8.35MiB/CPU-s 231.38KiB/CPU-s 2.85KiB/CPU-s 0.0 0.02706 False False
syslog_splunk_hec_logs ingress throughput 86.69KiB/CPU-s 1.00 100.00% 8.49MiB/CPU-s 275.45KiB/CPU-s 3.39KiB/CPU-s 0.0 0.031694 8.57MiB/CPU-s 242.07KiB/CPU-s 2.98KiB/CPU-s 0.0 0.027577 False False
syslog_log2metric_splunk_hec_metrics ingress throughput 87.69KiB/CPU-s 0.96 100.00% 8.88MiB/CPU-s 290.51KiB/CPU-s 3.58KiB/CPU-s 0.0 0.031942 8.97MiB/CPU-s 302.31KiB/CPU-s 3.72KiB/CPU-s 0.0 0.032921 False False
http_to_http_acks ingress throughput 50.52KiB/CPU-s 0.94 69.67% 5.27MiB/CPU-s 2.72MiB/CPU-s 34.32KiB/CPU-s 0.0 0.516659 5.32MiB/CPU-s 2.78MiB/CPU-s 35.09KiB/CPU-s 0.0 0.523122 True False
otlp_http_to_blackhole ingress throughput 10.97KiB/CPU-s 0.71 100.00% 1.5MiB/CPU-s 125.93KiB/CPU-s 1.55KiB/CPU-s 0.0 0.081895 1.51MiB/CPU-s 116.67KiB/CPU-s 1.44KiB/CPU-s 0.0 0.075335 False False
http_text_to_http_json ingress throughput 74.86KiB/CPU-s 0.30 100.00% 24.48MiB/CPU-s 556.82KiB/CPU-s 6.85KiB/CPU-s 0.0 0.022209 24.55MiB/CPU-s 626.69KiB/CPU-s 7.71KiB/CPU-s 0.0 0.024922 False False
socket_to_socket_blackhole ingress throughput 7.66KiB/CPU-s 0.06 72.63% 12.92MiB/CPU-s 394.22KiB/CPU-s 4.85KiB/CPU-s 0.0 0.029788 12.93MiB/CPU-s 409.43KiB/CPU-s 5.04KiB/CPU-s 0.0 0.030919 False False
http_to_http_noack ingress throughput 7.47KiB/CPU-s 0.05 77.70% 13.61MiB/CPU-s 387.48KiB/CPU-s 4.77KiB/CPU-s 0.0 0.02781 13.61MiB/CPU-s 313.48KiB/CPU-s 3.86KiB/CPU-s 0.0 0.022486 False False
enterprise_http_to_http ingress throughput 5.93KiB/CPU-s 0.04 90.21% 13.62MiB/CPU-s 247.61KiB/CPU-s 3.05KiB/CPU-s 0.0 0.017755 13.62MiB/CPU-s 152.87KiB/CPU-s 1.88KiB/CPU-s 0.0 0.010956 False False
splunk_hec_indexer_ack_blackhole ingress throughput 1.18KiB/CPU-s 0.01 20.90% 13.62MiB/CPU-s 259.82KiB/CPU-s 3.2KiB/CPU-s 0.0 0.018635 13.62MiB/CPU-s 251.34KiB/CPU-s 3.09KiB/CPU-s 0.0 0.018025 False False
fluent_elasticsearch ingress throughput -45.11B/CPU-s -0.00 6.79% 45.41MiB/CPU-s 30.0KiB/CPU-s 373.87B/CPU-s 0.0 0.000645 45.41MiB/CPU-s 30.11KiB/CPU-s 375.17B/CPU-s 0.0 0.000647 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput 553.53B/CPU-s 0.00 6.53% 13.61MiB/CPU-s 385.17KiB/CPU-s 4.74KiB/CPU-s 0.0 0.027628 13.61MiB/CPU-s 373.62KiB/CPU-s 4.6KiB/CPU-s 0.0 0.026798 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -2.11KiB/CPU-s -0.02 35.39% 13.62MiB/CPU-s 258.55KiB/CPU-s 3.18KiB/CPU-s 0.0 0.018541 13.61MiB/CPU-s 269.56KiB/CPU-s 3.32KiB/CPU-s 0.0 0.019334 False False
http_to_http_json ingress throughput -19.96KiB/CPU-s -0.14 100.00% 13.61MiB/CPU-s 247.88KiB/CPU-s 3.05KiB/CPU-s 0.0 0.017782 13.59MiB/CPU-s 257.69KiB/CPU-s 3.17KiB/CPU-s 0.0 0.018513 False False
datadog_agent_remap_datadog_logs_acks ingress throughput -117.31KiB/CPU-s -0.35 100.00% 32.28MiB/CPU-s 1.13MiB/CPU-s 14.27KiB/CPU-s 0.0 0.035091 32.16MiB/CPU-s 1.42MiB/CPU-s 17.94KiB/CPU-s 0.0 0.044271 False False
syslog_log2metric_humio_metrics ingress throughput -32.32KiB/CPU-s -0.52 100.00% 6.1MiB/CPU-s 164.68KiB/CPU-s 2.03KiB/CPU-s 0.0 0.026376 6.07MiB/CPU-s 176.79KiB/CPU-s 2.18KiB/CPU-s 0.0 0.028463 False False
otlp_grpc_to_blackhole ingress throughput -7.51KiB/CPU-s -0.72 100.00% 1.02MiB/CPU-s 48.2KiB/CPU-s 607.33B/CPU-s 0.0 0.046248 1.01MiB/CPU-s 46.82KiB/CPU-s 589.97B/CPU-s 0.0 0.045249 False False
syslog_humio_logs ingress throughput -69.4KiB/CPU-s -0.78 100.00% 8.74MiB/CPU-s 272.5KiB/CPU-s 3.35KiB/CPU-s 0.0 0.030449 8.67MiB/CPU-s 334.22KiB/CPU-s 4.11KiB/CPU-s 0.0 0.037638 False False
datadog_agent_remap_datadog_logs ingress throughput -324.08KiB/CPU-s -0.96 100.00% 32.96MiB/CPU-s 1.17MiB/CPU-s 14.76KiB/CPU-s 0.0 0.035555 32.64MiB/CPU-s 1.0MiB/CPU-s 12.64KiB/CPU-s 0.0 0.030723 False False
datadog_agent_remap_blackhole ingress throughput -303.41KiB/CPU-s -0.96 100.00% 30.88MiB/CPU-s 1.16MiB/CPU-s 14.63KiB/CPU-s 0.0 0.037609 30.58MiB/CPU-s 1.3MiB/CPU-s 16.42KiB/CPU-s 0.0 0.04261 False False
syslog_regex_logs2metric_ddmetrics ingress throughput -44.02KiB/CPU-s -1.22 100.00% 3.53MiB/CPU-s 317.62KiB/CPU-s 3.91KiB/CPU-s 0.0 0.087785 3.49MiB/CPU-s 303.23KiB/CPU-s 3.73KiB/CPU-s 0.0 0.084842 False True
datadog_agent_remap_blackhole_acks ingress throughput -443.89KiB/CPU-s -1.39 100.00% 31.08MiB/CPU-s 953.05KiB/CPU-s 11.73KiB/CPU-s 0.0 0.029942 30.65MiB/CPU-s 983.87KiB/CPU-s 12.11KiB/CPU-s 0.0 0.031348 False False
splunk_hec_route_s3 ingress throughput -188.34KiB/CPU-s -1.59 100.00% 11.56MiB/CPU-s 555.99KiB/CPU-s 6.84KiB/CPU-s 0.0 0.046976 11.37MiB/CPU-s 616.89KiB/CPU-s 7.59KiB/CPU-s 0.0 0.052965 False False

Copy link
Contributor

@spencergilbert spencergilbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks straightforward to me, looks like a direct port of the existing script but I'm not overly familiar with homebrew

Copy link
Member

@bruceg bruceg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggested some simplifications in git.rs but otherwise this LGTM

vdev/src/git.rs Outdated Show resolved Hide resolved
vdev/src/git.rs Outdated Show resolved Hide resolved
vdev/src/git.rs Outdated Show resolved Hide resolved
@jszwedko
Copy link
Member

@jonathanpv is this still waiting for anything? Or can we merge it in?

@jonathanpv jonathanpv enabled auto-merge (squash) March 30, 2023 20:42
@jonathanpv
Copy link
Contributor Author

@jonathanpv is this still waiting for anything? Or can we merge it in?

whoops sorry didn't know there was a reply waiting here. Nothing to wait on was very focused on the windows environment and neglected this at the moment. Is there a better way to parse through github comments and notifications? I get spammed on my email for failing jobs and its quite cumbersome to go through it all when theres only actionable things to do on a few notifications.

@jszwedko
Copy link
Member

@jonathanpv is this still waiting for anything? Or can we merge it in?

whoops sorry didn't know there was a reply waiting here. Nothing to wait on was very focused on the windows environment and neglected this at the moment. Is there a better way to parse through github comments and notifications? I get spammed on my email for failing jobs and its quite cumbersome to go through it all when theres only actionable things to do on a few notifications.

No worries :D Dealing with GitHub notifications effectively is challenging. One thing you might find useful would be to look at GitHub's notification center: https://github.com/notifications?query=is%3Aunread . There there are some useful filters to find actionable notifications.

@jonathanpv jonathanpv disabled auto-merge March 31, 2023 04:15
@jonathanpv
Copy link
Contributor Author

This needed to be modified to include the recent changes of capture_output to check_output.

It now prints the exit codes of the failing git functions.

Error: Command: "git" "config" "user.name" "vic" "user.email" "vector@datadoghq.com"
failed with exit code: 129

output:

I made this fail on the git::push() function in homebrew.rs by setting an invalid GITHUB_TOKEN
The error code 129 should help users find out what went wrong, but I could also add a specific debug messages stating the git env variables that may be relevant to these functions.

@datadog-vectordotdev
Copy link

datadog-vectordotdev bot commented Mar 31, 2023

Datadog Report

Branch report: jonathanpv/release-homebrew
Commit report: 9e6f915

vector: 1 Failed (1 Known Flaky), 0 New Flaky, 2129 Passed, 0 Skipped, 8m 34.34s Wall Time

❌ Failed Tests (1)

  • sources::aws_s3::integration_tests::handles_errored_status - vector - ❄️ Known flaky - Details

    Expand for error
     Test has failed
    

@github-actions
Copy link

Regression Detector Results

Run ID: 9371de44-9d6b-4381-9412-77121232913a
Baseline: 018637a
Comparison: f79185d
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
file_to_blackhole egress throughput 537.74KiB/CPU-s 8.30 99.64% 6.32MiB/CPU-s 4.34MiB/CPU-s 123.42KiB/CPU-s 0.0 0.686563 6.85MiB/CPU-s 4.31MiB/CPU-s 137.38KiB/CPU-s 0.0 0.6283 True True
datadog_agent_remap_blackhole ingress throughput 397.91KiB/CPU-s 1.27 100.00% 30.54MiB/CPU-s 1.29MiB/CPU-s 16.28KiB/CPU-s 0.0 0.042316 30.93MiB/CPU-s 1.03MiB/CPU-s 12.95KiB/CPU-s 0.0 0.033244 False False
datadog_agent_remap_blackhole_acks ingress throughput 357.87KiB/CPU-s 1.14 100.00% 30.53MiB/CPU-s 1.22MiB/CPU-s 15.43KiB/CPU-s 0.0 0.040093 30.88MiB/CPU-s 851.48KiB/CPU-s 10.48KiB/CPU-s 0.0 0.026924 False False
splunk_hec_route_s3 ingress throughput 83.72KiB/CPU-s 0.71 100.00% 11.55MiB/CPU-s 606.21KiB/CPU-s 7.46KiB/CPU-s 0.0 0.051262 11.63MiB/CPU-s 589.42KiB/CPU-s 7.25KiB/CPU-s 0.0 0.049491 False False
syslog_log2metric_humio_metrics ingress throughput 25.53KiB/CPU-s 0.41 100.00% 6.12MiB/CPU-s 156.06KiB/CPU-s 1.92KiB/CPU-s 0.0 0.024891 6.15MiB/CPU-s 295.14KiB/CPU-s 3.63KiB/CPU-s 0.0 0.046881 False False
syslog_loki ingress throughput 32.06KiB/CPU-s 0.37 100.00% 8.4MiB/CPU-s 242.99KiB/CPU-s 2.99KiB/CPU-s 0.0 0.028244 8.43MiB/CPU-s 261.61KiB/CPU-s 3.22KiB/CPU-s 0.0 0.030295 False False
http_to_http_acks ingress throughput 13.2KiB/CPU-s 0.25 21.62% 5.22MiB/CPU-s 2.74MiB/CPU-s 34.5KiB/CPU-s 0.0 0.524228 5.24MiB/CPU-s 2.66MiB/CPU-s 33.51KiB/CPU-s 0.0 0.507947 True False
syslog_splunk_hec_logs ingress throughput 9.15KiB/CPU-s 0.10 98.12% 8.84MiB/CPU-s 248.83KiB/CPU-s 3.06KiB/CPU-s 0.0 0.02749 8.85MiB/CPU-s 195.81KiB/CPU-s 2.41KiB/CPU-s 0.0 0.021612 False False
socket_to_socket_blackhole ingress throughput 4.81KiB/CPU-s 0.03 72.59% 13.48MiB/CPU-s 224.25KiB/CPU-s 2.76KiB/CPU-s 0.0 0.01624 13.49MiB/CPU-s 278.02KiB/CPU-s 3.42KiB/CPU-s 0.0 0.020127 False False
http_to_http_json ingress throughput 1.43KiB/CPU-s 0.01 27.99% 13.62MiB/CPU-s 231.38KiB/CPU-s 2.85KiB/CPU-s 0.0 0.016589 13.62MiB/CPU-s 226.91KiB/CPU-s 2.79KiB/CPU-s 0.0 0.016267 False False
splunk_hec_indexer_ack_blackhole ingress throughput 1.5KiB/CPU-s 0.01 26.05% 13.61MiB/CPU-s 264.0KiB/CPU-s 3.25KiB/CPU-s 0.0 0.018934 13.62MiB/CPU-s 253.99KiB/CPU-s 3.12KiB/CPU-s 0.0 0.018215 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput 947.97B/CPU-s 0.01 11.94% 13.61MiB/CPU-s 349.36KiB/CPU-s 4.3KiB/CPU-s 0.0 0.02506 13.61MiB/CPU-s 359.2KiB/CPU-s 4.42KiB/CPU-s 0.0 0.025764 False False
fluent_elasticsearch ingress throughput -955.11B/CPU-s -0.00 68.04% 45.41MiB/CPU-s 30.13KiB/CPU-s 375.49B/CPU-s 0.0 0.000648 45.41MiB/CPU-s 70.85KiB/CPU-s 883.0B/CPU-s 0.0 0.001523 False False
enterprise_http_to_http ingress throughput 202.73B/CPU-s 0.00 4.73% 13.62MiB/CPU-s 192.12KiB/CPU-s 2.36KiB/CPU-s 0.0 0.013772 13.62MiB/CPU-s 191.62KiB/CPU-s 2.36KiB/CPU-s 0.0 0.013735 False False
http_to_http_noack ingress throughput -1.89KiB/CPU-s -0.01 23.25% 13.61MiB/CPU-s 357.78KiB/CPU-s 4.4KiB/CPU-s 0.0 0.025671 13.61MiB/CPU-s 376.34KiB/CPU-s 4.63KiB/CPU-s 0.0 0.027006 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -1.3KiB/CPU-s -0.01 22.34% 13.62MiB/CPU-s 259.14KiB/CPU-s 3.19KiB/CPU-s 0.0 0.018584 13.61MiB/CPU-s 269.4KiB/CPU-s 3.31KiB/CPU-s 0.0 0.019322 False False
http_text_to_http_json ingress throughput -3.04KiB/CPU-s -0.01 22.75% 25.05MiB/CPU-s 648.14KiB/CPU-s 7.98KiB/CPU-s 0.0 0.025265 25.05MiB/CPU-s 555.85KiB/CPU-s 6.84KiB/CPU-s 0.0 0.02167 False False
otlp_grpc_to_blackhole ingress throughput -368.03B/CPU-s -0.03 34.09% 1.02MiB/CPU-s 49.18KiB/CPU-s 619.71B/CPU-s 0.0 0.047158 1.02MiB/CPU-s 44.32KiB/CPU-s 558.57B/CPU-s 0.0 0.042514 False False
datadog_agent_remap_datadog_logs_acks ingress throughput -79.59KiB/CPU-s -0.24 100.00% 32.82MiB/CPU-s 850.95KiB/CPU-s 10.47KiB/CPU-s 0.0 0.025316 32.74MiB/CPU-s 950.19KiB/CPU-s 11.69KiB/CPU-s 0.0 0.028336 False False
syslog_regex_logs2metric_ddmetrics ingress throughput -22.99KiB/CPU-s -0.64 99.99% 3.53MiB/CPU-s 336.1KiB/CPU-s 4.14KiB/CPU-s 0.0 0.093093 3.5MiB/CPU-s 329.34KiB/CPU-s 4.05KiB/CPU-s 0.0 0.091804 False True
otlp_http_to_blackhole ingress throughput -11.58KiB/CPU-s -0.75 100.00% 1.51MiB/CPU-s 127.7KiB/CPU-s 1.57KiB/CPU-s 0.0 0.082785 1.49MiB/CPU-s 126.15KiB/CPU-s 1.55KiB/CPU-s 0.0 0.082395 False False
syslog_log2metric_splunk_hec_metrics ingress throughput -92.65KiB/CPU-s -0.98 100.00% 9.19MiB/CPU-s 259.29KiB/CPU-s 3.19KiB/CPU-s 0.0 0.02754 9.1MiB/CPU-s 249.95KiB/CPU-s 3.08KiB/CPU-s 0.0 0.026812 False False
datadog_agent_remap_datadog_logs ingress throughput -384.02KiB/CPU-s -1.14 100.00% 32.97MiB/CPU-s 877.93KiB/CPU-s 10.8KiB/CPU-s 0.0 0.026006 32.59MiB/CPU-s 879.98KiB/CPU-s 10.83KiB/CPU-s 0.0 0.026366 False False
syslog_humio_logs ingress throughput -135.91KiB/CPU-s -1.47 100.00% 9.05MiB/CPU-s 148.58KiB/CPU-s 1.83KiB/CPU-s 0.0 0.016036 8.91MiB/CPU-s 231.69KiB/CPU-s 2.85KiB/CPU-s 0.0 0.02538 False False

@github-actions
Copy link

github-actions bot commented Apr 2, 2023

Regression Detector Results

Run ID: 19ae9499-f339-4afc-973a-2a08caccdf09
Baseline: 00c0316
Comparison: c8f8178
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_text_to_http_json ingress throughput 1.14MiB/CPU-s 4.81 100.00% 23.65MiB/CPU-s 608.94KiB/CPU-s 7.49KiB/CPU-s 0.0 0.025146 24.79MiB/CPU-s 648.23KiB/CPU-s 7.98KiB/CPU-s 0.0 0.025539 False False
datadog_agent_remap_datadog_logs ingress throughput 1.02MiB/CPU-s 3.18 100.00% 32.04MiB/CPU-s 1006.75KiB/CPU-s 12.39KiB/CPU-s 0.0 0.030683 33.06MiB/CPU-s 952.51KiB/CPU-s 11.72KiB/CPU-s 0.0 0.028135 False False
socket_to_socket_blackhole ingress throughput 174.04KiB/CPU-s 1.29 100.00% 13.2MiB/CPU-s 218.43KiB/CPU-s 2.69KiB/CPU-s 0.0 0.016165 13.36MiB/CPU-s 261.18KiB/CPU-s 3.21KiB/CPU-s 0.0 0.019083 False False
datadog_agent_remap_datadog_logs_acks ingress throughput 262.87KiB/CPU-s 0.79 100.00% 32.45MiB/CPU-s 981.9KiB/CPU-s 12.08KiB/CPU-s 0.0 0.029546 32.71MiB/CPU-s 1.17MiB/CPU-s 15.43KiB/CPU-s 0.0 0.035714 False False
http_to_http_acks ingress throughput 28.34KiB/CPU-s 0.53 44.45% 5.22MiB/CPU-s 2.7MiB/CPU-s 34.08KiB/CPU-s 0.0 0.517516 5.25MiB/CPU-s 2.69MiB/CPU-s 33.92KiB/CPU-s 0.0 0.512478 True False
syslog_log2metric_splunk_hec_metrics ingress throughput 29.28KiB/CPU-s 0.32 100.00% 9.08MiB/CPU-s 336.33KiB/CPU-s 4.14KiB/CPU-s 0.0 0.036184 9.11MiB/CPU-s 236.84KiB/CPU-s 2.91KiB/CPU-s 0.0 0.0254 False False
http_to_http_json ingress throughput 40.24KiB/CPU-s 0.29 100.00% 13.58MiB/CPU-s 286.06KiB/CPU-s 3.52KiB/CPU-s 0.0 0.020567 13.62MiB/CPU-s 228.56KiB/CPU-s 2.81KiB/CPU-s 0.0 0.016386 False False
otlp_grpc_to_blackhole ingress throughput 562.63B/CPU-s 0.05 50.68% 1.01MiB/CPU-s 51.27KiB/CPU-s 645.97B/CPU-s 0.0 0.049599 1.01MiB/CPU-s 40.22KiB/CPU-s 506.92B/CPU-s 0.0 0.038887 False False
enterprise_http_to_http ingress throughput 7.35KiB/CPU-s 0.05 91.29% 13.62MiB/CPU-s 293.01KiB/CPU-s 3.61KiB/CPU-s 0.0 0.021015 13.62MiB/CPU-s 189.47KiB/CPU-s 2.33KiB/CPU-s 0.0 0.013582 False False
splunk_hec_indexer_ack_blackhole ingress throughput -448.69B/CPU-s -0.00 7.70% 13.61MiB/CPU-s 258.78KiB/CPU-s 3.18KiB/CPU-s 0.0 0.01856 13.61MiB/CPU-s 262.67KiB/CPU-s 3.23KiB/CPU-s 0.0 0.01884 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput -29.64B/CPU-s -0.00 0.36% 13.61MiB/CPU-s 363.23KiB/CPU-s 4.47KiB/CPU-s 0.0 0.026054 13.61MiB/CPU-s 365.04KiB/CPU-s 4.49KiB/CPU-s 0.0 0.026184 False False
fluent_elasticsearch ingress throughput -970.1B/CPU-s -0.00 93.24% 45.41MiB/CPU-s 30.41KiB/CPU-s 378.91B/CPU-s 0.0 0.000654 45.41MiB/CPU-s 29.82KiB/CPU-s 371.59B/CPU-s 0.0 0.000641 False False
http_to_http_noack ingress throughput -49.85B/CPU-s -0.00 0.63% 13.61MiB/CPU-s 355.07KiB/CPU-s 4.37KiB/CPU-s 0.0 0.025476 13.61MiB/CPU-s 355.25KiB/CPU-s 4.37KiB/CPU-s 0.0 0.025489 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -2.06KiB/CPU-s -0.01 35.45% 13.62MiB/CPU-s 250.8KiB/CPU-s 3.09KiB/CPU-s 0.0 0.017985 13.62MiB/CPU-s 263.61KiB/CPU-s 3.24KiB/CPU-s 0.0 0.018906 False False
syslog_loki ingress throughput -55.47KiB/CPU-s -0.65 100.00% 8.33MiB/CPU-s 255.57KiB/CPU-s 3.14KiB/CPU-s 0.0 0.029951 8.28MiB/CPU-s 267.5KiB/CPU-s 3.29KiB/CPU-s 0.0 0.031554 False False
otlp_http_to_blackhole ingress throughput -14.65KiB/CPU-s -0.93 100.00% 1.53MiB/CPU-s 118.58KiB/CPU-s 1.46KiB/CPU-s 0.0 0.075547 1.52MiB/CPU-s 115.43KiB/CPU-s 1.42KiB/CPU-s 0.0 0.074233 False False
splunk_hec_route_s3 ingress throughput -127.36KiB/CPU-s -1.06 100.00% 11.69MiB/CPU-s 533.8KiB/CPU-s 6.57KiB/CPU-s 0.0 0.044587 11.57MiB/CPU-s 538.79KiB/CPU-s 6.63KiB/CPU-s 0.0 0.045488 False False
datadog_agent_remap_blackhole_acks ingress throughput -368.8KiB/CPU-s -1.16 100.00% 31.04MiB/CPU-s 1.04MiB/CPU-s 13.07KiB/CPU-s 0.0 0.033434 30.68MiB/CPU-s 868.35KiB/CPU-s 10.69KiB/CPU-s 0.0 0.027641 False False
syslog_humio_logs ingress throughput -116.44KiB/CPU-s -1.27 100.00% 8.97MiB/CPU-s 235.52KiB/CPU-s 2.9KiB/CPU-s 0.0 0.025628 8.86MiB/CPU-s 291.06KiB/CPU-s 3.58KiB/CPU-s 0.0 0.032077 False False
datadog_agent_remap_blackhole ingress throughput -469.82KiB/CPU-s -1.49 100.00% 30.72MiB/CPU-s 1.1MiB/CPU-s 13.92KiB/CPU-s 0.0 0.035965 30.26MiB/CPU-s 862.71KiB/CPU-s 10.62KiB/CPU-s 0.0 0.027843 False False
syslog_log2metric_humio_metrics ingress throughput -109.62KiB/CPU-s -1.71 100.00% 6.26MiB/CPU-s 174.16KiB/CPU-s 2.14KiB/CPU-s 0.0 0.027165 6.15MiB/CPU-s 175.36KiB/CPU-s 2.16KiB/CPU-s 0.0 0.027827 False False
syslog_splunk_hec_logs ingress throughput -199.76KiB/CPU-s -2.17 100.00% 8.99MiB/CPU-s 223.26KiB/CPU-s 2.75KiB/CPU-s 0.0 0.024243 8.8MiB/CPU-s 242.07KiB/CPU-s 2.98KiB/CPU-s 0.0 0.026868 False False
file_to_blackhole egress throughput -197.22KiB/CPU-s -2.88 72.35% 6.69MiB/CPU-s 4.34MiB/CPU-s 130.26KiB/CPU-s 4.635193 0.648517 6.5MiB/CPU-s 4.17MiB/CPU-s 125.96KiB/CPU-s 5.831158 0.641246 True True
syslog_regex_logs2metric_ddmetrics ingress throughput -217.89KiB/CPU-s -5.95 100.00% 3.57MiB/CPU-s 409.75KiB/CPU-s 5.04KiB/CPU-s 0.0 0.111925 3.36MiB/CPU-s 382.82KiB/CPU-s 4.71KiB/CPU-s 0.0 0.111185 True True

@jonathanpv jonathanpv merged commit 21fa240 into master Apr 3, 2023
@jonathanpv jonathanpv deleted the jonathanpv/release-homebrew branch April 3, 2023 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-condition: integration tests enable Run integration tests on this PR domain: ci Anything related to Vector's CI environment domain: releasing Anything related to releasing Vector domain: vdev Anything related to the vdev tooling platform: homebrew Anything `homebrew` platform related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants