internal/civisibility: intelligent test runner support #2943

tonyredondo · 2024-10-23T21:58:12Z

What does this PR do?

This PR adds the Test Visibility Intelligent Test Runner feature: https://datadoghq.atlassian.net/wiki/spaces/SDTEST/pages/3123348305/Intelligent+Test+Runner+in+CI+Visibility+Libraries

Motivation

This is one of the most important feature for Test Visibility.

Reviewer's Checklist

Changed code has unit tests for its functionality at or near 100% coverage.
System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
There is a benchmark for any new code, or changes to existing code.
If this interacts with the agent in a new way, a system test has been added.
Add an appropriate team label so this PR gets put in the right place for the release notes.
Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

pr-commenter · 2024-10-24T11:19:24Z

Benchmarks

Benchmark execution time: 2024-10-29 10:39:35

Comparing candidate commit 17e359b in PR branch tony/civisibility-test-code-coverage-research with baseline commit 4b56ee5 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 0 unstable metrics.

ManuelPalenzuelaDD

LGTM. I lack context in a lot of the implementation details of this, but the go impl looks ok. I've left some minor Qs/nits

internal/civisibility/integrations/civisibility_features.go

ManuelPalenzuelaDD · 2024-10-28T15:07:28Z

internal/civisibility/utils/net/coverage.go

+	}
+
+	// Create a dummy event to send with the coverage payload.
+	dummyEvent := FormFile{


Q: Why do we need to upload a dummy event?

yeah, that's required, Why? you tell me 😉 documented here: https://datadoghq.atlassian.net/wiki/spaces/SDTEST/pages/3123054226/Per+test+code+coverage#JSON-attachment

ManuelPalenzuelaDD · 2024-10-28T15:16:47Z

internal/civisibility/integrations/gotesting/coverage/test_coverage.go

+	}
+
+	restore := setStdOutToTemp()
+	t.postCoverageFilename = filepath.Join(temporaryDir, fmt.Sprintf("%d-%d-%d-post.out", t.moduleID, t.suiteID, t.testID))


why is this line inside the restore := setStdOutToTemp() block? isn't it only necessary for the tearDown fn to be in that block?

True, I did a refactor to remove that method and just create a tearDown wrapper in 1ba2dda

…ge by test

…t coverage

…client library. Also adds a new method to send coverage data.

…o start sending payloads to the backend

…interface to get the coverage writer

…pend on the linker (messes with tests running inside the IDE)

tonyredondo force-pushed the tony/civisibility-test-code-coverage-research branch 2 times, most recently from 672e27b to 6ccdc75 Compare October 24, 2024 10:35

DataDog deleted a comment from pr-commenter bot Oct 24, 2024

tonyredondo marked this pull request as ready for review October 24, 2024 15:23

tonyredondo requested a review from a team as a code owner October 24, 2024 15:23

tonyredondo requested a review from ManuelPalenzuelaDD October 24, 2024 15:23

tonyredondo force-pushed the tony/civisibility-test-code-coverage-research branch from 93d3a97 to 2dcc3fc Compare October 24, 2024 16:15

ManuelPalenzuelaDD approved these changes Oct 28, 2024

View reviewed changes

tonyredondo force-pushed the tony/civisibility-test-code-coverage-research branch from 1ba2dda to 396f778 Compare October 28, 2024 22:11

tonyredondo added 20 commits October 29, 2024 12:41

internal/civisibility: initial research on how to extract code covera…

66682f1

…ge by test

internal/civisibility: changes and continue research

b12f816

internal/civisibility: refactor and create a dedicate package for tes…

254dd1e

…t coverage

internal/civisibility: adds the citestcov format

7e4d1cc

internal/civisibility: changes

6f1ef95

internal/civisibility: add custom code coverage payload

2f08d3e

internal/civisibility: add support for messagepack to the rapid http …

7e961b0

…client library. Also adds a new method to send coverage data.

internal/civisibility: add a test for the new coverage api

c98080d

internal/civisibility: add a coverage writer and connect everything t…

bc5360b

…o start sending payloads to the backend

internal/civisibility: stdout coverage payload on debug for now.

3cdbcbe

internal/civisibility: changes

5624320

internal/civisibility: fixes

35c05f8

internal/civisibility: fixes issue with the payload buffer

d885e26

internal/civisibility: add the skippable request

acfa600

internal/civisibility: initial ITR implementation (test skipping)

0b50455

internal/civisibility: comments and add global code coverage support

00d21b6

internal/civisibility: change the condition to TestsSkipping

d3b3ed4

internal/civisibility: adds skippable_test.go

2458504

internal/civisibility: fixes and refactor. Also disables parallel tests

c86a9f0

internal/civisibility: add missing session tags for ITR

68b55c9

tonyredondo added 17 commits October 29, 2024 12:41

internal/civisibility: changes to debug logs and add some null checks

212ab3f

internal/civisibility: add safety checks for extracting the testDeps …

cbe63dd

…interface to get the coverage writer

internal/civisibility: remove the linkname between packages to not de…

74454b6

…pend on the linker (messes with tests running inside the IDE)

internal/civisibility: add reflections_test.go and fixes

f324875

internal/civisibility: more tests

83f2c8a

internal/civisibility: more tests

4769d74

internal/civisibility: renaming and more tests

ebf0220

internal/civisibility: fixes

5fbeb3b

internal/civisibility: linter fixes

e8ce39e

internal/civisibility: add a test suite for Intelligent Test Runner

995c0da

internal/civisibility: fix tag

a1cbced

internal/civisibility: addressing review comments

ff643e3

internal/civisibility: changes

3125f4a

internal/civisibility: fix tests in CI

cac4555

internal/civisibility: trying to make the linter happy

597324a

internal/civisibility: fix possible data race

29ff6be

internal/civisibility: fix possible data race

3ccf36e

tonyredondo force-pushed the tony/civisibility-test-code-coverage-research branch from 17e359b to 3ccf36e Compare October 29, 2024 11:41

tonyredondo merged commit fcda5e2 into main Oct 29, 2024
149 of 150 checks passed

tonyredondo deleted the tony/civisibility-test-code-coverage-research branch October 29, 2024 12:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

internal/civisibility: intelligent test runner support #2943

internal/civisibility: intelligent test runner support #2943

tonyredondo commented Oct 23, 2024 •

edited

Loading

pr-commenter bot commented Oct 24, 2024 •

edited

Loading

ManuelPalenzuelaDD left a comment

ManuelPalenzuelaDD Oct 28, 2024

tonyredondo Oct 28, 2024

ManuelPalenzuelaDD Oct 28, 2024

tonyredondo Oct 28, 2024

internal/civisibility: intelligent test runner support #2943

internal/civisibility: intelligent test runner support #2943

Conversation

tonyredondo commented Oct 23, 2024 • edited Loading

What does this PR do?

Motivation

Reviewer's Checklist

pr-commenter bot commented Oct 24, 2024 • edited Loading

Benchmarks

ManuelPalenzuelaDD left a comment

Choose a reason for hiding this comment

ManuelPalenzuelaDD Oct 28, 2024

Choose a reason for hiding this comment

tonyredondo Oct 28, 2024

Choose a reason for hiding this comment

ManuelPalenzuelaDD Oct 28, 2024

Choose a reason for hiding this comment

tonyredondo Oct 28, 2024

Choose a reason for hiding this comment

tonyredondo commented Oct 23, 2024 •

edited

Loading

pr-commenter bot commented Oct 24, 2024 •

edited

Loading