Move chroot building into TaskRunner #2132

schmichael · 2016-12-21T00:39:30Z

The largest part of this change is refactoring a allocdir.TaskDir struct out of the allocdir.AllocDir struct. AllocRunner now builds only the alloc dir and defers task dir building to TaskRunner.

This allows TaskRunner to consult the new Driver.FSIsolation() method to inform TaskDir.Build() what kind of filesystem layout the task driver expects.

The executor no longer has access to the AllocDir (or even the new TaskDir) struct. Instead it just knows whether fs isolation is used (chroot) and what its task and log paths are.

Feel free to bikeshed names and API details. I think the basic approach is sound, but I'm sure lots can be tweaked.

Test Changes

Most test changes are mechanical to fix API differences.

I did switch a couple test helper functions to return test-only structs instead of having a number of return values: testDriverContext

It made for an obnoxious number of small changes, but it does make it easier to tweak what test helpers setup in the future.

Future Work

GetTaskEnv could be refactored to not take both Node and Config
Add a Driver.Cleanup method for cases like where Docker.Prestart downloads an image but is then killed. Cleanup would remove the leaked image.

diptanu · 2017-01-03T20:28:54Z

client/alloc_runner.go

-	if r.ctx == nil {
-		snapshotErrors.Errors = append(snapshotErrors.Errors, fmt.Errorf("alloc_runner snapshot includes a nil context"))
+	if r.allocDir == nil {
+		//FIXME Upgrade path?


You could probably keep the context for now, and if the r.allocDir is still nil, create it. That should take care of the upgrade scenario. And remove context in the next release.

diptanu · 2017-01-03T21:30:15Z

client/alloc_runner.go

+			err := fmt.Errorf("failed to find task dir metadata for alloc %q task %q",
+				r.alloc.ID, name)
+			r.logger.Printf("[ERR] client: %v", err)
+			mErr.Errors = append(mErr.Errors, err)


We should error out and not do anything else if we can't find the task directory from the alloc dir struct. You will be passing nil otherwise to the TaskRunner.

Good call. Fixed.

diptanu · 2017-01-03T21:54:09Z

client/alloc_runner.go

@@ -448,7 +454,11 @@ func (r *AllocRunner) Run() {
 			continue
 		}

-		tr := NewTaskRunner(r.logger, r.config, r.setTaskState, r.ctx, r.Alloc(), task.Copy(), r.vaultClient)
+		r.allocDirLock.Lock()


It might be cleaner if AllocDir maintains it's own locks internally so that the consumers of the API doesn't have to ensure they are locked appropriately.

Sadly we'd still need this lock because it guards the setting and checking the allocDir field on the AllocRunner.

We might be able to skip acquiring it on this particular line if AllocDir had an internal lock, but I'd rather always guard accesses to a field than try to figure out safe places to skip synchronization and confuse future readers of the code.

dadgar

Very nice!

dadgar · 2017-01-04T23:48:38Z

client/allocdir/task_dir.go

+	cstructs "github.com/hashicorp/nomad/client/structs"
+)
+
+type TaskDir struct {


Comment on the struct

dadgar · 2017-01-04T23:52:30Z

client/allocdir/task_dir.go

+// create paths on disk.
+//
+// Call AllocDir.NewTaskDir to create new TaskDirs
+func newTaskDir(allocDir, taskName string) *TaskDir {


Can you make these take a logger. Will be helpful in the future

I did that during development and removed it... that probably should have been a hint to keep it!

Fixed.

dadgar · 2017-01-04T23:54:08Z

client/driver/docker.go

-		return nil, fmt.Errorf("Could not find task directory for task: %v", d.DriverContext.taskName)
-	}
-	pluginLogFile := filepath.Join(taskDir, fmt.Sprintf("%s-executor.out", task.Name))
+	pluginLogFile := filepath.Join(ctx.TaskDir.Dir, fmt.Sprintf("%s-executor.out", task.Name))


Can you drop the task name as part of the file. It is within the directory anyways

Gladly. Fixed.

dadgar · 2017-01-05T00:02:21Z

client/driver/executor/executor.go

@@ -277,30 +272,41 @@ func (e *UniversalExecutor) LaunchCmd(command *ExecCommand) (*ProcessState, erro
 		return nil, err
 	}

+	e.logger.Printf("[DEBUG] executor: XXX 1")


Oh my. Fixed all of these.

dadgar · 2017-01-05T00:05:47Z

client/driver/raw_exec.go


 	// Get the command to be ran
 	command := driverConfig.Command
 	if err := validateCommand(command, "args"); err != nil {
+		d.logger.Printf("[WARN] driver.raw_exec: XXX error validating command")


May wanna scan through the whole PR and clean some of these up

* Refactor AllocDir to have a TaskDir struct per task. * Drivers expose filesystem isolation preference * Fix lxc mounting of `secrets/`

AllocRunner's state dropped the Context struct which needs to be converted to the new AllocDir+TaskDir structs in RestoreState. TaskRunner added a TaskDirBuilt flag, but it's safe to just let that default to `false` and rebuild all task dirs once on upgrade.

dadgar · 2017-01-06T18:09:34Z

client/alloc_runner.go

 	AllocClientStatus      string
 	AllocClientDescription string
-	Context                *driver.ExecContext
+
+	// Context is deprecated and only used to migrate from older releases.


We use the following format for these: // COMPAT: Remove in 0.X.0: <reason for compatibility code and when it can be removed

Consolidate task environment building in GetTaskEnv since it can determine what kind of filesystem isolation is used. This means drivers no longer have to manipulate task environment paths.

github-actions · 2023-04-12T02:10:18Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

diptanu reviewed Jan 3, 2017

View reviewed changes

dadgar requested changes Jan 5, 2017

View reviewed changes

schmichael added 8 commits January 5, 2017 16:31

Move chroot building into TaskRunner

de7351b

* Refactor AllocDir to have a TaskDir struct per task. * Drivers expose filesystem isolation preference * Fix lxc mounting of `secrets/`

Test tasks now require driver name

e7f6986

Fail fast on taskdir errors

d87dcce

Fix upgrade path for #2132

3011df1

AllocRunner's state dropped the Context struct which needs to be converted to the new AllocDir+TaskDir structs in RestoreState. TaskRunner added a TaskDirBuilt flag, but it's safe to just let that default to `false` and rebuild all task dirs once on upgrade.

Add comments to TaskDir

957adf2

Put a logger in AllocDir/TaskDir

e25274b

Remove task name prefix from executor logs

86db194

Remove debug logging

12915c3

schmichael force-pushed the f-chrootectomy-4 branch from dc13f87 to 452825f Compare January 6, 2017 00:36

schmichael force-pushed the f-chrootectomy-4 branch 2 times, most recently from a2b1f3b to 0073dd6 Compare January 6, 2017 00:44

dadgar approved these changes Jan 6, 2017

View reviewed changes

schmichael added 4 commits January 6, 2017 11:39

Fix tests post rebase

6235122

Driver is now required in test tasks

59c501f

Add COMPAT comment

9bfa8dc

Fix executor tests

ed8b3cd

schmichael force-pushed the f-chrootectomy-4 branch from f465db3 to ed8b3cd Compare January 6, 2017 19:39

Fix inconsistent task env setting

a087a8b

Consolidate task environment building in GetTaskEnv since it can determine what kind of filesystem isolation is used. This means drivers no longer have to manipulate task environment paths.

schmichael merged commit 9541290 into master Jan 6, 2017

schmichael deleted the f-chrootectomy-4 branch January 6, 2017 21:03

This was referenced Jan 10, 2017

Don't require root for Docker #2178

Closed

Adding support for task specific Chroot #2062

Closed

schmichael mentioned this pull request Mar 2, 2017

Ensure dev/, proc/, and alloc/ are mounted every time a task is run #2394

Merged

github-actions bot locked as resolved and limited conversation to collaborators Apr 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move chroot building into TaskRunner #2132

Move chroot building into TaskRunner #2132

schmichael commented Dec 21, 2016 •

edited

Loading

diptanu Jan 3, 2017

diptanu Jan 3, 2017

schmichael Jan 4, 2017

diptanu Jan 3, 2017

schmichael Jan 4, 2017 •

edited

Loading

dadgar left a comment

dadgar Jan 4, 2017

schmichael Jan 5, 2017

dadgar Jan 4, 2017

schmichael Jan 5, 2017

dadgar Jan 4, 2017

schmichael Jan 6, 2017

dadgar Jan 5, 2017

schmichael Jan 6, 2017

dadgar Jan 5, 2017

dadgar Jan 6, 2017

github-actions bot commented Apr 12, 2023

Move chroot building into TaskRunner #2132

Move chroot building into TaskRunner #2132

Conversation

schmichael commented Dec 21, 2016 • edited Loading

Test Changes

Future Work

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schmichael Jan 4, 2017 • edited Loading

Choose a reason for hiding this comment

dadgar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Apr 12, 2023

schmichael commented Dec 21, 2016 •

edited

Loading

schmichael Jan 4, 2017 •

edited

Loading