Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SDTEST-437] Auto test retries for minitest #214

Merged
merged 7 commits into from
Aug 9, 2024

Conversation

anmarchenko
Copy link
Member

@anmarchenko anmarchenko commented Aug 8, 2024

What does this PR do?
Adds auto test retries support for minitest

How to test the change?
Feature is covered by unit tests in minitest/instrumentation_spec.rb

Tested using Sidekiq in this commit: anmarchenko/sidekiq@020ca88

Local test run ends successfully:
image

New flaky test is detected by Datadog:
image

Correct test runs are traced:
image

Retries can be filtered using test.is_retry tag:
image

Test session is reported as passed:
image

@codecov-commenter
Copy link

codecov-commenter commented Aug 9, 2024

Codecov Report

Attention: Patch coverage is 98.70130% with 2 lines in your changes missing coverage. Please review.

Project coverage is 98.88%. Comparing base (b891aaf) to head (067c735).

Files Patch % Lines
...atadog/ci/contrib/minitest/instrumentation_spec.rb 97.97% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #214      +/-   ##
==========================================
- Coverage   98.89%   98.88%   -0.01%     
==========================================
  Files         261      261              
  Lines       12096    12243     +147     
  Branches      544      552       +8     
==========================================
+ Hits        11962    12107     +145     
- Misses        134      136       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@anmarchenko anmarchenko marked this pull request as ready for review August 9, 2024 11:39
@anmarchenko anmarchenko requested review from a team as code owners August 9, 2024 11:39
@anmarchenko anmarchenko merged commit 96c1fe6 into main Aug 9, 2024
28 checks passed
@anmarchenko anmarchenko deleted the anmarchenko/auto_test_retries_minitest branch August 9, 2024 12:49
@github-actions github-actions bot added this to the 1.4.0 milestone Aug 9, 2024

class FlakyTestSuite2 < Minitest::Test
@@max_flaky_test_failures = 4
@@flaky_test_failures = 0

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Code Quality Violation

Do not use class variables (...read more)

The rule "Avoid class variables" refers to the practice of refraining from using class variables (variables starting with '@@') in Ruby. Class variables are shared between a class and all of its descendants, which can lead to unexpected behavior and bugs that are difficult to trace. This is because if a class variable is changed in a subclass, that change will also affect the superclass and all other subclasses.

This rule is crucial for maintaining clean, predictable, and easy-to-debug code. It also helps to prevent unintentional side effects that can occur when class variables are manipulated in different parts of a program.

To adhere to this rule, consider using class instance variables or constants instead. Class instance variables belong solely to the class they are defined in, and their value does not get shared with subclasses. Constants, on the other hand, are a good option when the value is not meant to change. For example, in the given non-compliant code, the class variable @@class_var could be replaced with a class instance variable @class_var or a constant CLASS_VAR, depending on the intended use.

View in Datadog  Leave us feedback  Documentation

i_suck_and_my_tests_are_order_dependent!

@@max_flaky_test_failures = 4
@@flaky_test_failures = 0

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Code Quality Violation

Do not use class variables (...read more)

The rule "Avoid class variables" refers to the practice of refraining from using class variables (variables starting with '@@') in Ruby. Class variables are shared between a class and all of its descendants, which can lead to unexpected behavior and bugs that are difficult to trace. This is because if a class variable is changed in a subclass, that change will also affect the superclass and all other subclasses.

This rule is crucial for maintaining clean, predictable, and easy-to-debug code. It also helps to prevent unintentional side effects that can occur when class variables are manipulated in different parts of a program.

To adhere to this rule, consider using class instance variables or constants instead. Class instance variables belong solely to the class they are defined in, and their value does not get shared with subclasses. Constants, on the other hand, are a good option when the value is not meant to change. For example, in the given non-compliant code, the class variable @@class_var could be replaced with a class instance variable @class_var or a constant CLASS_VAR, depending on the intended use.

View in Datadog  Leave us feedback  Documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants