Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: add tests for v1.7.0 builders #638

Merged
merged 3 commits into from
Jun 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 14 additions & 18 deletions cli/slsa-verifier/main_regression_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,10 @@ func getBuildersAndVersions(t *testing.T,
}

func Test_runVerifyGHAArtifactPath(t *testing.T) {
// We cannot use t.Setenv due to parallelized tests.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we just not run the test in parallel?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one in particular takes a very long time since it scales across the builders and vers and makes online requests to Rekor - I'll try it out, but I think the best solution is just of get rid of the env var by regenerating the test.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how to specify it, but if we could run these tests (the test cases run by t.Run) in parallel but not concurrently with other top-level tests then it should work.

// TODO(639): Remove this by regenerating multiple subjects test.
os.Setenv("SLSA_VERIFIER_TESTING", "1")

t.Parallel()
goBuilder := "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/builder_go_slsa3.yml"
genericBuilder := "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml"
Expand Down Expand Up @@ -1298,11 +1302,7 @@ func Test_runVerifyGCBArtifactImage(t *testing.T) {
}
}

// TODO(#485): Version the test-cases when a version for the builder is released.
func Test_runVerifyGHAContainerBased(t *testing.T) {
// We cannot use t.Setenv due to parallelized tests.
os.Setenv("SLSA_VERIFIER_TESTING", "1")

t.Parallel()

tests := []struct {
Expand All @@ -1318,58 +1318,58 @@ func Test_runVerifyGHAContainerBased(t *testing.T) {
}{
{
name: "valid main branch default",
artifacts: []string{"workflow_dispatch.main.default"},
artifacts: []string{"binary-linux-amd64-workflow_dispatch"},
source: "github.com/slsa-framework/example-package",
},
{
name: "versioned tag no match empty tag workflow_dispatch",
artifacts: []string{"workflow_dispatch.main.default"},
artifacts: []string{"binary-linux-amd64-workflow_dispatch"},
source: "github.com/slsa-framework/example-package",
pversiontag: pString("v1"),
err: serrors.ErrorInvalidRef,
},
{
name: "tag no match empty tag workflow_dispatch",
artifacts: []string{"workflow_dispatch.main.default"},
artifacts: []string{"binary-linux-amd64-workflow_dispatch"},
source: "github.com/slsa-framework/example-package",
ptag: pString("v1.2.3"),
err: serrors.ErrorInvalidRef,
},
{
name: "wrong branch master",
artifacts: []string{"workflow_dispatch.main.default"},
artifacts: []string{"binary-linux-amd64-workflow_dispatch"},
source: "github.com/slsa-framework/example-package",
pbranch: pString("master"),
err: serrors.ErrorMismatchBranch,
},
{
name: "valid main branch set",
artifacts: []string{"workflow_dispatch.main.default"},
artifacts: []string{"binary-linux-amd64-workflow_dispatch"},
source: "github.com/slsa-framework/example-package",
pbranch: pString("main"),
},
{
name: "valid main branch default - invalid builderID",
artifacts: []string{"workflow_dispatch.main.default"},
artifacts: []string{"binary-linux-amd64-workflow_dispatch"},
source: "github.com/slsa-framework/example-package",
pBuilderID: pString("https://github.com/slsa-framework/slsa-github-generator/.github/workflows/not-trusted.yml"),
err: serrors.ErrorUntrustedReusableWorkflow,
},
{
name: "wrong source append A",
artifacts: []string{"workflow_dispatch.main.default"},
artifacts: []string{"binary-linux-amd64-workflow_dispatch"},
source: "github.com/slsa-framework/example-packageA",
err: serrors.ErrorMismatchSource,
},
{
name: "wrong source prepend A",
artifacts: []string{"workflow_dispatch.main.default"},
artifacts: []string{"binary-linux-amd64-workflow_dispatch"},
source: "Agit.luolix.top/slsa-framework/example-package",
err: serrors.ErrorMismatchSource,
},
{
name: "wrong source middle A",
artifacts: []string{"workflow_dispatch.main.default"},
artifacts: []string{"binary-linux-amd64-workflow_dispatch"},
source: "github.com/Aslsa-framework/example-package",
err: serrors.ErrorMismatchSource,
},
Expand Down Expand Up @@ -1399,14 +1399,10 @@ func Test_runVerifyGHAContainerBased(t *testing.T) {
builder := "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/builder_container-based_slsa3.yml"

refName := "@refs/tags/"
if sv == "main" {
refName = "@refs/heads/"
}
// TODO(#485): Add pString(builder + "@" + sv) when migrating to tagged builders
// and remove main builder test.
builderIDs := []*string{
pString(builder + refName + sv),
pString(builder),
pString(builder + "@" + sv),
nil,
}

Expand Down

This file was deleted.

This file was deleted.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Binary file not shown.
Loading