Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: define test session semantics #1513

Merged
merged 60 commits into from
Sep 18, 2023

Conversation

mdelapenya
Copy link
Member

@mdelapenya mdelapenya commented Aug 21, 2023

What does this PR do?

This PR refactors how the session ID is obtained. Instead of simply getting an UUID from the Google's library, which we will still use as default, we are going to get the current process' parent ID (parent pid) in order to make sure that any test package in the project shares the same unique identifier.

This is needed to synchronise how multiple test packages in Go are seen from the test execution standpoint: because each Go package will be compiled, packaged and tested in a separate binary, and because "go test" will call all of those test binaries for running tests, we need a way to aggregate the execution into one single hash representing that test execution. By test execution we mean:

  • a single "go test" invocation (including flags)
  • a single "go test ./..." invocation (including flags)
  • the execution of a single test or a set of tests using the IDE

As a consequence, with the sole goal of aggregating test execution across multiple packages, the new code will use the parent process ID (pid) of the current process and use it to generate a unique session ID. We are using the parent pid because the current process will be a child process of:

  • the process that is running the tests, e.g.: "go test";
  • the process that is running the application in development mode, e.g. "go run main.go -tags dev";
  • the process that is running the tests in the IDE, e.g.: "go test ./...".

For getting an unique representation of the test session, we are creating the following hash:

sessionID := hash("testcontainers-go:${parentPid}:${parentPidCreationTime})

As a consequence, the reaper (Ryuk) has to be started just once across the entire test session, so we have implemented a way to look up the Ryuk container, and if it exists, create our own reaper instance from the raw representation of the underlying container. With this, Ryuk should be started just once, killing all containers in the test session. The diagram for getting Ryuk:

graph TD
    classDef red fill:#ff7e6b, color: #fff;
    classDef green fill:#6fd890;
    classDef yellow fill:#fff68f;
    classDef blue fill:#668aff, color: #fff;

    G0[Goroutine 0] --> Mutex((sync.Mutex))
    G1[Goroutine 1] --> Mutex
    G2[Goroutine 2] --> Mutex
    G3[Goroutine 3] --> Mutex
    Gn[Goroutine n] --> Mutex
    Mutex --> InstanceNil{"reaperInstance == nil"}
    InstanceNil -- yes --> LookupDocker{"reaper in Docker"}
    InstanceNil -- no --> ReaperInstance(Reaper Instance)
    LookupDocker -- no --> Once((sync.Once))
    LookupDocker -- yes --> ReadFromDocker["Read instance from Docker"]
    ReadFromDocker --> ReaperInstance
    Once --> CreateReaper["Create Reaper Container"]
    CreateReaper --> ReaperInstance

    class Mutex,Once red
    class CreateReaper green
    class InstanceNil,LookupDocker yellow
    class ReadFromDocker blue
Loading

Then, Ryuk has to label all the containers in the session properly, including the sessionID, which will be the label used to kill them, among others that we use, such as the testcontainers language. For that reason, we have simplified how the container labels are spread across the codebase, keeping an internal DefaultLabels function, which will set the default labels for a given sessionID (helpful for tests too).

Why is it important?

A perfect use case to consume this unique identifier for a test execution would be Ryuk: avoid multiple instances per Go package. Therefore, Ryuk will label all containers in a test session to kill them. As a consequence, Ryuk has to be started just once across the entire test session.

@mdelapenya mdelapenya added the chore Changes that do not impact the existing functionality label Aug 21, 2023
@mdelapenya mdelapenya self-assigned this Aug 21, 2023
@netlify
Copy link

netlify bot commented Aug 21, 2023

Deploy Preview for testcontainers-go ready!

Name Link
🔨 Latest commit 765221e
🔍 Latest deploy log https://app.netlify.com/sites/testcontainers-go/deploys/6508632231fc6f0008371ec2
😎 Deploy Preview https://deploy-preview-1513--testcontainers-go.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@mdelapenya mdelapenya closed this Aug 24, 2023
@mdelapenya mdelapenya deleted the session-ids branch August 24, 2023 06:57
@mdelapenya mdelapenya restored the session-ids branch August 24, 2023 06:58
@mdelapenya mdelapenya reopened this Aug 24, 2023
* main: (32 commits)
  fix: remove wrong example from workspace (testcontainers#1556)
  chore(deps): bump the all group in /modules/localstack with 1 update (testcontainers#1552)
  modulegen: generate code-workspace with json marshal (testcontainers#1551)
  chore(deps): bump the all group in /modules/compose with 2 updates (testcontainers#1553)
  feat: add mariadb module (testcontainers#1548)
  feat(modulegen): print out VSCode workspace file if needed (testcontainers#1549)
  modulegen: generate md file inside internal/mkdocs (testcontainers#1543)
  modulegen: create internal/module and internal/modfile (testcontainers#1539)
  [Enhancement]: add ability to set repo:tag for ContainerRequest FromDockerfile (testcontainers#1508)
  Fix module generator for examples (testcontainers#1545)
  Update pulsar.md (testcontainers#1542)
  modulegen: create internal/make (testcontainers#1537)
  chore: fix workflow (testcontainers#1538)
  chore(deps): bump the all group in /examples/cockroachdb with 1 update (testcontainers#1522)
  chore(deps): bump the all group in /examples/bigtable with 1 update (testcontainers#1534)
  chore(deps): bump the all group in /modules/localstack with 4 updates (testcontainers#1535)
  chore(deps): bump the all group in /modules/k3s with 2 updates (testcontainers#1526)
  chore(deps): bump the all group in /examples/spanner with 2 updates (testcontainers#1532)
  chore(deps): bump the all group in /examples/firestore with 1 update (testcontainers#1523)
  chore(deps): bump the all group in /modules/redis with 1 update (testcontainers#1524)
  ...
* main:
  chore: refine fail-fast strategy on CI (testcontainers#1555)
@mdelapenya mdelapenya changed the title feat: aggregate test executions at the sessionID level feat: define test session semantics Sep 18, 2023
@mdelapenya mdelapenya added enhancement New feature or request and removed chore Changes that do not impact the existing functionality labels Sep 18, 2023
@mdelapenya mdelapenya merged commit aa3be2d into testcontainers:main Sep 18, 2023
104 checks passed
mdelapenya added a commit to mdelapenya/testcontainers-go that referenced this pull request Sep 20, 2023
* main:
  fix: proper next version
  chore: prepare for next minor development cycle ()
  chore: use new version (v0.24.0) in modules and examples
  fix: include sonarcloud file into the release commit
  modulegen: generate sonar configuration (testcontainers#1644)
  feat: define test session semantics (testcontainers#1513)
  chore(deps): bump actions/checkout from 3 to 4 (testcontainers#1623)
@mdelapenya mdelapenya deleted the session-ids branch September 26, 2023 14:23
@mdelapenya mdelapenya mentioned this pull request Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant