e2e: create a v3/ set of packages for creating Nomad e2e tests #17620

shoenig · 2023-06-20T15:44:38Z

This PR creates an experimental set of packages under e2e/v3/ for crafting
Nomad e2e tests. This is an attempt at providing a way to create tests in a
declarative (ish) pattern, with a focus on being easy to use, easy to cleanup,
and easy to debug.

There's enough in place for the examples package, and for running a docker registry
and running tests against that (in another PR).

Lots of features need to be implemented.
Many more docs need to be written.
Breaking changes are to be expected.
There are known and unknown bugs.
No warranty.

A revised example suite

package example

import (
    "fmt"
    "testing"
    "time"

    "github.com/hashicorp/nomad/e2e/v3/cluster3"
    "github.com/hashicorp/nomad/e2e/v3/jobs3"
    "github.com/hashicorp/nomad/e2e/v3/namespaces3"
    "github.com/hashicorp/nomad/e2e/v3/util3"
    "github.com/shoenig/test/must"
)

func TestExample(t *testing.T) {
    cluster3.Establish(t,
        cluster3.Leader(),
        cluster3.LinuxClients(1),
        cluster3.Timeout(3*time.Second),
    )

    t.Run("testSleep", testSleep)
    t.Run("testNamespace", testNamespace)
    t.Run("testEnv", testEnv)
}

func testSleep(t *testing.T) {
    _, cleanup := jobs3.Submit(t, "./input/sleep.hcl")
    t.Cleanup(cleanup)
}

func testNamespace(t *testing.T) {
    name := util3.ShortID("example")

    nsCleanup := namespaces3.Create(t, name)
    t.Cleanup(nsCleanup)

    _, jobCleanup := jobs3.Submit(t, "./input/sleep.hcl", jobs3.Namespace(name))
    t.Cleanup(jobCleanup)
}

func testEnv(t *testing.T) {
    job, cleanup := jobs3.Submit(t, "./input/env.hcl")
    t.Cleanup(cleanup)

    expect := fmt.Sprintf("NOMAD_JOB_ID=%s", job.JobID())
    logs := job.TaskLogs("group", "task")
    must.StrContains(t, logs.Stdout, expect)
}

Quick run of example with verbose logging.

➜ NOMAD_E2E_VERBOSE=1 go test -v
=== RUN   TestExample
=== RUN   TestExample/testSleep
    util3.go:25: register (service) job: "sleep-809"
    util3.go:25: checking eval: 9f0ae04d-7259-9333-3763-44d0592d03a1, status: pending
    util3.go:25: checking eval: 9f0ae04d-7259-9333-3763-44d0592d03a1, status: complete
    util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running
    util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running
    util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running
    util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running
    util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: successful
    util3.go:25: deployment a85ad2f8-269c-6620-d390-8eac7a9c397d was a success
    util3.go:25: deregister job "sleep-809"
    util3.go:25: system gc
=== RUN   TestExample/testNamespace
    util3.go:25: apply namespace "example-291"
    util3.go:25: register (service) job: "sleep-967"
    util3.go:25: checking eval: a2a2303a-adf1-2621-042e-a9654292e569, status: pending
    util3.go:25: checking eval: a2a2303a-adf1-2621-042e-a9654292e569, status: complete
    util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: running
    util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: running
    util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: running
    util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: successful
    util3.go:25: deployment 3395e9a8-3ffc-8990-d5b8-cc0ce311f302 was a success
    util3.go:25: deregister job "sleep-967"
    util3.go:25: system gc
    util3.go:25: cleanup namespace "example-291"
=== RUN   TestExample/testEnv
    util3.go:25: register (batch) job: "env-582"
    util3.go:25: checking eval: 600f3bce-ea17-6d13-9d20-9d9eb2a784f7, status: pending
    util3.go:25: checking eval: 600f3bce-ea17-6d13-9d20-9d9eb2a784f7, status: complete
    util3.go:25: deregister job "env-582"
    util3.go:25: system gc
--- PASS: TestExample (10.08s)
    --- PASS: TestExample/testSleep (5.02s)
    --- PASS: TestExample/testNamespace (4.02s)
    --- PASS: TestExample/testEnv (1.03s)
PASS
ok      github.com/hashicorp/nomad/e2e/example  10.079s

@shoenig

This PR creates an experimental set of packages under `e2e/v3/` for crafting Nomad e2e tests. Unlike previous generations, this is an attempt at providing a way to create tests in a declarative (ish) pattern, with a focus on being easy to use, easy to cleanup, and easy to debug. @shoenig is just trying this out to see how it goes. Lots of features need to be implemented. Many more docs need to be written. Breaking changes are to be expected. There are known and unknown bugs. No warranty. Quick run of `example` with verbose logging. ```shell ➜ NOMAD_E2E_VERBOSE=1 go test -v === RUN TestExample === RUN TestExample/testSleep util3.go:25: register (service) job: "sleep-809" util3.go:25: checking eval: 9f0ae04d-7259-9333-3763-44d0592d03a1, status: pending util3.go:25: checking eval: 9f0ae04d-7259-9333-3763-44d0592d03a1, status: complete util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: successful util3.go:25: deployment a85ad2f8-269c-6620-d390-8eac7a9c397d was a success util3.go:25: deregister job "sleep-809" util3.go:25: system gc === RUN TestExample/testNamespace util3.go:25: apply namespace "example-291" util3.go:25: register (service) job: "sleep-967" util3.go:25: checking eval: a2a2303a-adf1-2621-042e-a9654292e569, status: pending util3.go:25: checking eval: a2a2303a-adf1-2621-042e-a9654292e569, status: complete util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: running util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: running util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: running util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: successful util3.go:25: deployment 3395e9a8-3ffc-8990-d5b8-cc0ce311f302 was a success util3.go:25: deregister job "sleep-967" util3.go:25: system gc util3.go:25: cleanup namespace "example-291" === RUN TestExample/testEnv util3.go:25: register (batch) job: "env-582" util3.go:25: checking eval: 600f3bce-ea17-6d13-9d20-9d9eb2a784f7, status: pending util3.go:25: checking eval: 600f3bce-ea17-6d13-9d20-9d9eb2a784f7, status: complete util3.go:25: deregister job "env-582" util3.go:25: system gc --- PASS: TestExample (10.08s) --- PASS: TestExample/testSleep (5.02s) --- PASS: TestExample/testNamespace (4.02s) --- PASS: TestExample/testEnv (1.03s) PASS ok github.com/hashicorp/nomad/e2e/example 10.079s ```

tgross

This looks great @shoenig! I'd be happy to have this merged as-is and then we can keep iterating on it

tgross · 2023-06-22T15:27:00Z

e2e/v3/cluster3/cluster3.go

+			eligible := 0
+			for _, node := range nodes {
+				if node.Status == "ready" {
+					info, _, err := nodesAPI.Info(node.ID, nil)
+					if err != nil {
+						return err
+					}
+					if info.Attributes["kernel.name"] == "linux" {
+						eligible++
+					}
+				}
+			}


Code golf: we could probably add a filter on the List API call for ready node with the appropriate attribute.

tgross · 2023-06-22T15:30:28Z

e2e/v3/jobs3/jobs3.go

+	// we need to go through evals until we find the deployment
+	evalAPI := sub.nomadClient.Evaluations()


The initial eval ID comes back in the job registration response.

tgross · 2023-06-22T15:34:02Z

e2e/v3/jobs3/jobs3.go

+func Namespace(name string) Option {
+	return func(sub *Submission) {
+		sub.inNamespace = name
+	}
+}
+func AuthToken(token string) Option {
+	return func(sub *Submission) {
+		sub.authToken = token
+	}
+}


Another sort of option we may want to consider here in future work is handling intentionally-failing or otherwise degraded job deployments. There's a pattern we use a bunch where we submit a job that's broken and then change something in the cluster to cause it to get re-evaluated and then allocs get created, or the allocs get placed but then are stuck pending on the client until we add something their template block needs.

Yeah definitely - I was just going to keep iterating on features as I (or others) need them. And this pattern is probably going to come up soon.

lgfa29

Looks like a great API design! If node pools didn't require infrastructure change I would be writing some tests with this already 😄

lgfa29 · 2023-06-22T20:52:21Z

e2e/example/example_test.go

Do we expect to write all new tests using v3 or does it need to bake a bit more? If we should wait perhaps this could be a new example_3_test.go file?

I plan on fixing up the failing tests during the 1.6-1.7 gap using these new packages. I wouldn't go so far as to say "all new tests should use this" yet, because there's still so many missing features I haven't even thought of.

e2e/example/input/sleep.hcl

lgfa29 · 2023-06-22T20:54:52Z

e2e/v3/README.md

+- `v3/cluster3` - establish and verify the state of the cluster
+- `v3/jobs3` - manage nomad jobs and wait for deployments, etc.
+- `v3/namespaces3` - manage nomad namespaces
+- `v3/util3` - helper methods specific to the `v3` utilities


Is it a Go thing that the packages need to be suffixed with 3 or is it just to make it more clear which version is being used?

I couldn't think of a good naming convention, so <package>3 is what it is for now 😅 We can certainly change this if anyone thinks of better names

lgfa29 · 2023-06-22T21:13:26Z

e2e/v3/jobs3/jobs3.go

+		case nomadapi.AllocClientStatusLost:
+			must.Unreachable(sub.t, must.Sprintf("group %q allocation %s lost", group, id))
+		case nomadapi.AllocClientStatusFailed:
+			must.Unreachable(sub.t, must.Sprintf("group %q allocation %s failed", group, id))
+		case nomadapi.AllocClientStatusPending:
+			break
+		case nomadapi.AllocClientStatusRunning:
+			break
+		case nomadapi.AllocClientStatusComplete:
+			break ALLOCATION


There's also unknown which seem to be missing from api 😬

yikes! filed #17688

@shoenig

* e2e: create a v3/ set of packages for creating Nomad e2e tests This PR creates an experimental set of packages under `e2e/v3/` for crafting Nomad e2e tests. Unlike previous generations, this is an attempt at providing a way to create tests in a declarative (ish) pattern, with a focus on being easy to use, easy to cleanup, and easy to debug. @shoenig is just trying this out to see how it goes. Lots of features need to be implemented. Many more docs need to be written. Breaking changes are to be expected. There are known and unknown bugs. No warranty. Quick run of `example` with verbose logging. ```shell ➜ NOMAD_E2E_VERBOSE=1 go test -v === RUN TestExample === RUN TestExample/testSleep util3.go:25: register (service) job: "sleep-809" util3.go:25: checking eval: 9f0ae04d-7259-9333-3763-44d0592d03a1, status: pending util3.go:25: checking eval: 9f0ae04d-7259-9333-3763-44d0592d03a1, status: complete util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: running util3.go:25: checking deployment: a85ad2f8-269c-6620-d390-8eac7a9c397d, status: successful util3.go:25: deployment a85ad2f8-269c-6620-d390-8eac7a9c397d was a success util3.go:25: deregister job "sleep-809" util3.go:25: system gc === RUN TestExample/testNamespace util3.go:25: apply namespace "example-291" util3.go:25: register (service) job: "sleep-967" util3.go:25: checking eval: a2a2303a-adf1-2621-042e-a9654292e569, status: pending util3.go:25: checking eval: a2a2303a-adf1-2621-042e-a9654292e569, status: complete util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: running util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: running util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: running util3.go:25: checking deployment: 3395e9a8-3ffc-8990-d5b8-cc0ce311f302, status: successful util3.go:25: deployment 3395e9a8-3ffc-8990-d5b8-cc0ce311f302 was a success util3.go:25: deregister job "sleep-967" util3.go:25: system gc util3.go:25: cleanup namespace "example-291" === RUN TestExample/testEnv util3.go:25: register (batch) job: "env-582" util3.go:25: checking eval: 600f3bce-ea17-6d13-9d20-9d9eb2a784f7, status: pending util3.go:25: checking eval: 600f3bce-ea17-6d13-9d20-9d9eb2a784f7, status: complete util3.go:25: deregister job "env-582" util3.go:25: system gc --- PASS: TestExample (10.08s) --- PASS: TestExample/testSleep (5.02s) --- PASS: TestExample/testNamespace (4.02s) --- PASS: TestExample/testEnv (1.03s) PASS ok github.com/hashicorp/nomad/e2e/example 10.079s ``` * cluster3: use filter for kernel.name instead of filtering manually

shoenig force-pushed the e2e-v3 branch from f3091df to 57a65d5 Compare June 21, 2023 13:47

vercel bot deployed to Preview – nomad-storybook-and-ui June 21, 2023 13:50 View deployment

shoenig force-pushed the e2e-v3 branch 2 times, most recently from b430042 to bfb1ada Compare June 21, 2023 18:49

vercel bot deployed to Preview – nomad-storybook-and-ui June 21, 2023 18:53 View deployment

shoenig force-pushed the e2e-v3 branch from bfb1ada to 0fc620c Compare June 21, 2023 19:07

vercel bot deployed to Preview – nomad-storybook-and-ui June 21, 2023 19:10 View deployment

shoenig mentioned this pull request Jun 21, 2023

e2e podman private registry #17642

Merged

shoenig force-pushed the e2e-v3 branch from 0fc620c to bba1870 Compare June 21, 2023 19:29

vercel bot deployed to Preview – nomad-storybook-and-ui June 21, 2023 19:31 View deployment

shoenig marked this pull request as ready for review June 21, 2023 20:56

shoenig requested review from tgross and lgfa29 June 21, 2023 20:56

tgross approved these changes Jun 22, 2023

View reviewed changes

lgfa29 approved these changes Jun 22, 2023

View reviewed changes

cluster3: use filter for kernel.name instead of filtering manually

c259553

vercel bot deployed to Preview – nomad-storybook-and-ui June 23, 2023 13:55 View deployment

shoenig merged commit 5b5fbc0 into main Jun 23, 2023
13 checks passed

shoenig deleted the e2e-v3 branch June 23, 2023 14:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e: create a v3/ set of packages for creating Nomad e2e tests #17620

e2e: create a v3/ set of packages for creating Nomad e2e tests #17620

shoenig commented Jun 20, 2023 •

edited

Loading

tgross left a comment

tgross Jun 22, 2023

shoenig Jun 23, 2023

tgross Jun 22, 2023

tgross Jun 22, 2023

shoenig Jun 23, 2023

lgfa29 left a comment

lgfa29 Jun 22, 2023

shoenig Jun 23, 2023

lgfa29 Jun 22, 2023

shoenig Jun 23, 2023

lgfa29 Jun 22, 2023

shoenig Jun 22, 2023

		// we need to go through evals until we find the deployment
		evalAPI := sub.nomadClient.Evaluations()

e2e: create a v3/ set of packages for creating Nomad e2e tests #17620

e2e: create a v3/ set of packages for creating Nomad e2e tests #17620

Conversation

shoenig commented Jun 20, 2023 • edited Loading

tgross left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lgfa29 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shoenig commented Jun 20, 2023 •

edited

Loading