Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Reduce flaky integration tests triggered by test_get_tag #25958

Merged
merged 2 commits into from
Nov 13, 2023

Conversation

sebastianliebscher
Copy link
Contributor

@sebastianliebscher sebastianliebscher commented Nov 11, 2023

SUMMARY

Local tests and GitHub CI runs sometimes fail because of

FAILED tests/integration_tests/tags/api_tests.py::TestTagApi::test_get_tag - AssertionError: 'now' != 'a second ago'

Examples from the last two weeks (note: the links will expire)

The test now additionally checks for a second ago and two seconds ago as seen in the CI examples above.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

  • scripts/tests/run.sh --module tests/integration_tests/tags/api_tests.py::TestTagApi::test_get_tag

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@john-bodley
Copy link
Member

@cccs-RyanK it seems like you might have authored the original test. Do you have any insights into why said test may be somewhat flakey?

@villebro
Copy link
Member

villebro commented Nov 12, 2023

I believe the flakiness is caused by test execution times being variable, causing it to sometimes take longer than usual, which in turn causes the gap between the operations to extend beyond 1 second. I think freezegun could be a good tool here to ensure the test execution time doesn't affect the test result.

@sebastianliebscher
Copy link
Contributor Author

@villebro thanks for the hint to use freezegun. That is a much cleaner approach now.

Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@john-bodley john-bodley merged commit c3249dd into apache:master Nov 13, 2023
26 checks passed
@sebastianliebscher sebastianliebscher deleted the test/test_get_tag branch November 13, 2023 17:29
josedev-union pushed a commit to Ortege-xyz/studio that referenced this pull request Jan 22, 2024
cccs-rc pushed a commit to CybercentreCanada/superset that referenced this pull request Mar 6, 2024
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.1.0 labels Mar 8, 2024
sfirke pushed a commit to sfirke/superset that referenced this pull request Mar 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/S 🚢 3.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants