Fix client restart behavior for Connect

[tgross]

Fixes #6310

I've pulled the test infrastructure work this requires out into #6313 and #6314 to keep the diff size reasonable. (Unit tests / linting won't pass till those are merged.) Done.

This fixes the client restart bug by:

• making sure we don't run AllocRunner post-run hooks during shutdown
• making the CreateNetwork call safe to re-run on start without recreating existing networks
• making sure the Consul proxy socket is recreated

cc @nickethier @schmichael

[tgross]

Note for reviewers: I refactored this big chunk of logic in createContainer out to containerByName because I wanted to reuse it in CreateNetwork. We want callers to be able to determine what to do if the container already existed.

[tgross]

@nickethier and @schmichael it looks like I've got this working, so this should be ready for review. I've tested out both client restarts and job stops by hand, and run the included e2e test:

▶ go test -v ./e2e/ -run 'TestE2E/Connect/connect\.ConnectClientStateE2ETest/TestClientRestart'
=== RUN   TestE2E
=== RUN   TestE2E/Connect
=== RUN   TestE2E/Connect/*connect.ConnectClientStateE2ETest
=== RUN   TestE2E/Connect/*connect.ConnectClientStateE2ETest/TestClientRestart
--- PASS: TestE2E (18.62s)
    --- PASS: TestE2E/Connect (18.62s)
        --- PASS: TestE2E/Connect/*connect.ConnectClientStateE2ETest (18.24s)
            --- PASS: TestE2E/Connect/*connect.ConnectClientStateE2ETest/TestClientRestart (18.11s)
PASS
ok      github.com/hashicorp/nomad/e2e  18.881s

[nickethier]

Only blocker is the question about os.Remove handling in consulsock hook.

[tgross]

@nickethier this should be ready for re-review.

[schmichael]

You could probably just create container here and skip the func, but this is a neat way to skip memory allocations for structs that may not be used.

[tgross]

I was a little wary of creating a partial NetworkIsolationSpec and then populating the fields later just because it's the sort of thing that's easy for someone to come along later and return it early and expose a nil pointer exception in a field access. If you think the indirection of the function here isn't idiomatic I'd be happy to change it though.

[nickethier]

The appveyor failure looks like a flake

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.