Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Instrument unit tests with CI Test Visibility 2.5.0 #1999

Merged
merged 14 commits into from
Aug 21, 2024

Conversation

ncreated
Copy link
Member

@ncreated ncreated commented Aug 19, 2024

What and why?

📦 This PR enables (back*) the Test Visibility instrumentation for our unit tests. With our tests being more stable in GitLab, we can leverage this instrumentation for more visibility into their performance and runs.

* we disabled Test Visibility a while ago, due to increased number of flaky test runs in Bitrise CI.

How?

Following the documentation, config ENVs are injected to every xcscheme and their values are exported in test.sh. Each scheme is configured to gather code coverage data, so we can later enable ITR.

🐶 This PR only integrates dd-sdk-swift-testing to an extra CI job meant for shadowing regular iOS unit tests runs. If we find our tests stable, we will integrate it to regular "Unit Test" jobs.

Unfortunately, there is no simpler way than applying the same configuration to all 14 schemes. The fact that we centralize tests execution through test.sh helps to control values, but the actual ENV injection must include forwarding it from Environment Variables list in xscheme definition. As explained in docs, the basic DD_TEST_RUNNER=1 ... xcodebuild test -scheme ... only works for Package.swift and not for Xcode project.

ENVs configuration

Review checklist

  • Feature or bugfix MUST have appropriate tests (unit, integration)
  • Make sure each commit and the PR mention the Issue number or JIRA reference
  • Add CHANGELOG entry for user facing changes

@ncreated ncreated self-assigned this Aug 19, 2024
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Aug 19, 2024

Datadog Report

Branch report: ncreated/chore/ci-visibility-shadow-build
Commit report: 5fa648b
Test service: dd-sdk-ios

✅ 0 Failed, 1848 Passed, 0 Skipped, 1m 15.14s Total Time

@ncreated ncreated marked this pull request as ready for review August 19, 2024 12:32
@ncreated ncreated requested review from a team as code owners August 19, 2024 12:32
@ncreated ncreated requested a review from ypopovych August 19, 2024 12:32
@ncreated
Copy link
Member Author

blocking/ We also need to pass CI Provider variables to all tests.

maxep
maxep previously approved these changes Aug 20, 2024
Copy link
Member

@maxep maxep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good 👍 Hope to remove the xcodeproj in a near future 🤞

@ncreated ncreated changed the title chore: Instrument unit tests with CI Test Visibility 2.5.0-alpha3 chore: Instrument unit tests with CI Test Visibility 2.5.0 Aug 21, 2024
@ncreated ncreated merged commit 93fd18c into develop Aug 21, 2024
17 checks passed
@ncreated ncreated deleted the ncreated/chore/ci-visibility-shadow-build branch August 21, 2024 09:41
@mariedm mariedm mentioned this pull request Sep 11, 2024
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants