nomad exec part 3: executor based drivers #5634

notnoop · 2019-04-30T18:37:31Z

Sequel to #5632 and #5633 .

Extend driver implementation for executor based drivers: raw_exec, exec, and java.

Here, the drivers use the low-level interface drivers.ExecTaskStreamingRawDriver, so drivers can delegate to the executor. Executors handle the exec stream, and construct the tty session.

We used kr/pty for terminal creation in the universal executor, because I found issues on macOS when using http://github.com/containerd/console (the pty library used by docker and imported here).

schmichael

Approving because I don't think any of my comments are blockers. Do the conformance tests cover a good set or error conditions or would it be possible to add some unit tests to internal funcs? I just want to make sure we don't have deadlocks or other unintended behavior when things stray off the happy path in production.

schmichael · 2019-05-10T18:16:40Z

drivers/shared/executor/client.go

+		}
+
+		if err := execStream.Send(m); err != nil {
+			return err


Do all returns need grpcutils.HandleGrpcErr? I'm a little fuzzy on where that needs to be called.

If all returns do need that check maybe just wrap the whole method in a shim to do the check?

func (d *grpcExecutorClient) ExecStreaming(...) error { if err := d.execStreaming(...); err != nil { return grpcutils.HandleGrpcErr(err, d.doneCtx) } } // actual impl in execStreaming method

we don't call grpcutils.HandleGrpcErr anywhere here. It seems mostly for detecting shutdown and being able to handle it (by restarting, etc), in this case, we don't want to handle executor shutdown any specially.

Ah! execStream is "this" (the driver) side whereas the HandleGrpcErr calls above are for remote calls to executors? I guess I'm still confused why Recv wouldn't call HandleGrpcErr as its the executor side...

Oh I see - I missed it - i'll add it.

schmichael · 2019-05-10T18:23:03Z

drivers/shared/executor/exec_utils.go

+
+	// newTerminal function creates a tty appropriate for the command
+	// The returned master end of tty function is to be called after process start.
+	newTerminal func() (master func() (*os.File, error), slave *os.File, err error)


Let's use parent / child terminology instead like https://godoc.org/github.com/opencontainers/runc/libcontainer/utils#NewSockPair

This is a bit strange of a pattern as well. What if you made all of these funcs methods on an interface like:

type terminal interface{ newTerminal() (...) setTTY(...) error ... }

...and then created structs that implemented them? Might be a little more code, but I think it might be more idiomatic and unittestable?

It's a bit confusing - libcontainer uses Parent/child for the socket that's controlling the tty - but the tty documentation uses master/slave e.g. https://linux.die.net/man/4/pts .

libcontainer avoids using master/slave by using RecvFd[1] and SendFd[2] but they are going against pretty much all tty and tty libraries documentation. Recognize the history of terminology, but I would rather not confuse future devs who need to consult publicly available documentations for tty.

[1] https://godoc.org/github.com/opencontainers/runc/libcontainer/utils#RecvFd
[2] https://godoc.org/github.com/opencontainers/runc/libcontainer/utils#SendFd

Let's break from the historical terminology on this one and use libcontainer's terminology. The kr/pty library recently fixed their terminology as well: kr/pty@7dc38fb

Umm.. actually looks like libcontainer still calls them master/slave in the docs[1][2] and in its implementation [3], and so does containerd/console[4] (the library used by runc to create console and created by ~the same maintainers) when referring to the console fields. I'll use kr/pty terminology instead.

[1] https://github.com/opencontainers/runc/blob/eb4aeed24ffbf8e2d740fafea39d91faa0ee84d0/docs/terminals.md#-new-terminal
[2] https://godoc.org/github.com/opencontainers/runc/libcontainer#Process
[3] https://github.com/opencontainers/runc/blob/master/libcontainer/console_linux.go#L9-L41
[4] https://github.com/containerd/console/blob/0650fd9eeb50bab4fc99dceb9f2e14cf58f36e7f/console_unix.go#L27-L30

schmichael · 2019-05-10T18:25:22Z

drivers/shared/executor/exec_utils.go

+}
+
+func handleStdin(logger hclog.Logger, stdin io.WriteCloser, stream drivers.ExecTaskStream, errCh chan<- error) {
+	go func() {


In general for funcs which are intended to run concurrently with the caller I prefer to let the caller call go. This makes it explicit when reading the caller's code that this func does not block and it can ease unit testing.

schmichael · 2019-05-10T18:27:50Z

drivers/shared/executor/exec_utils.go

+	wg.Add(1)
+
+	go func() {
+		defer wg.Done()


Same as above, and I think the caller should manage the waitgroup as well.

schmichael · 2019-05-10T18:28:00Z

drivers/shared/executor/exec_utils.go

+	wg.Add(1)
+
+	go func() {
+		defer wg.Done()


Same as above.

schmichael · 2019-05-10T18:28:57Z

drivers/shared/executor/exec_utils.go

+		defer wg.Done()
+
+		for {
+			buf := make([]byte, 4096)


I thought in #5632 you often reused buffers because you were careful to make send() not hold onto the buffer after returning? Is that not the case in this code? It would be nice to try to keep the approaches consistent or at least comment carefully when they diverge.

Good catch - I missed these during the refactor - will make buf outside the for loop

schmichael · 2019-05-10T18:35:08Z

drivers/shared/executor/executor.go

+func (e *UniversalExecutor) ExecStreaming(ctx context.Context, command []string, tty bool,
+	stream drivers.ExecTaskStream) error {
+
+	cmd := exec.CommandContext(ctx, command[0], command[1:]...)


Do we need a len check here to make sure we don't panic if command is empty?

schmichael · 2019-05-10T18:42:34Z

drivers/shared/executor/pty_unix.go

+		return fmt.Errorf("attempted to resize a non-tty session")
+	}
+
+	pty.Setsize(f, &pty.Winsize{


Comment as to why we ignore Setsize errors.

schmichael · 2019-05-10T18:44:40Z

drivers/shared/executor/pty_windows.go

+}
+
+func setTTYSize(w io.Writer, height, width int32) error {
+	return fmt.Errorf("unsupported")


Should this be an error or a noop?

schmichael · 2019-05-10T18:45:38Z

drivers/shared/executor/server.go

+		return fmt.Errorf("first message should always be setup")
+	}
+
+	return s.impl.ExecStreaming(server.Context(),


Actually is there where we could wrap errors in that HandleGrpcErr?

Prepare executor to handle streaming exec API calls that reuse drivers protobuf structs.

Implements streamign exec handling in both executors (i.e. universal and libcontainer). For creation of TTY, some incidental complexity leaked in. The universal executor uses github.com/kr/pty for creation of TTYs. On the other hand, libcontainer expects a console socket and for libcontainer to create the underlying console object on process start. The caller can then use `libcontainer.utils.RecvFd()` to get tty master end. I chose github.com/kr/pty for managing TTYs here. I tried `github.com/containerd/console` package (which is already imported), but the package did not work as expected on macOS.

These simply delegate call to backend executor.

github-actions · 2023-02-10T02:18:51Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

notnoop requested a review from schmichael April 30, 2019 18:38

notnoop force-pushed the f-nomad-exec-parts-02-cli branch from f6969b3 to 96ac203 Compare May 9, 2019 23:42

notnoop changed the base branch from f-nomad-exec-parts-02-cli to master May 9, 2019 23:43

notnoop force-pushed the f-nomad-exec-parts-03-executors branch from 6592b7f to c302f15 Compare May 9, 2019 23:44

schmichael approved these changes May 10, 2019

View reviewed changes

notnoop force-pushed the f-nomad-exec-parts-03-executors branch from e6bd183 to 90386e4 Compare May 10, 2019 22:45

Mahmood Ali added 5 commits May 10, 2019 19:17

executor: scaffolding for executor grpc handling

efc4249

Prepare executor to handle streaming exec API calls that reuse drivers protobuf structs.

drivers: implement streaming exec for executor based drivers

74e5e20

These simply delegate call to backend executor.

vendor github.com/kr/pty

f6a4b88

use pty/tty terminology similar to github.com/kr/pty

7f76aed

notnoop force-pushed the f-nomad-exec-parts-03-executors branch from fc4f92b to 7f76aed Compare May 10, 2019 23:17

notnoop merged commit c25b247 into master May 11, 2019

notnoop deleted the f-nomad-exec-parts-03-executors branch May 11, 2019 01:24

notnoop mentioned this pull request May 12, 2019

implement nomad exec for rkt #5690

Merged

github-actions bot locked as resolved and limited conversation to collaborators Feb 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nomad exec part 3: executor based drivers #5634

nomad exec part 3: executor based drivers #5634

notnoop commented Apr 30, 2019

schmichael left a comment

schmichael May 10, 2019

notnoop May 10, 2019

schmichael May 10, 2019

notnoop May 10, 2019

schmichael May 10, 2019

schmichael May 10, 2019

notnoop May 10, 2019

schmichael May 10, 2019 •

edited

Loading

notnoop May 10, 2019 •

edited

Loading

schmichael May 10, 2019

schmichael May 10, 2019

schmichael May 10, 2019

schmichael May 10, 2019

notnoop May 10, 2019

schmichael May 10, 2019

schmichael May 10, 2019

schmichael May 10, 2019

schmichael May 10, 2019

github-actions bot commented Feb 10, 2023

nomad exec part 3: executor based drivers #5634

nomad exec part 3: executor based drivers #5634

Conversation

notnoop commented Apr 30, 2019

schmichael left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schmichael May 10, 2019 • edited Loading

Choose a reason for hiding this comment

notnoop May 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 10, 2023

schmichael May 10, 2019 •

edited

Loading

notnoop May 10, 2019 •

edited

Loading