Add --chown flag to ADD/COPY commands #34263

estesp · 2017-07-26T16:28:49Z

Carry #28499 (which was a carry of #27303)

This adds --chown function to the ADD and COPY commands in Dockerfile. The format of the --chown flag allows any uid/username and gid/groupname combination separated by a colon, e.g.:

--chown=someuser:555
--chown=anyuser:anygroup
--chown=1001:1002
--chown=333:agroupname

Lookups (to translate name -> ID integer) will use the /etc/passwd and /etc/group files found in the container filesystem. Currently if they do not exist, then the docker build will fail. This PR probably needs a way to fail gracefully (or ignore failures) on Windows as those containers will not have these files. If only numeric IDs are used, no lookups will be performed.

User namespaces are supported and IDs will be shifted (after translated names to integers) based on the user namespace mappings set in the daemon.

I also think a few more tests are necessary to fully test the behavior, but wanted to get the rebased code available for review initially.

@estesp

Rebased by @estesp Signed-off-by: Kara Alexandra <kalexandra@us.ibm.com> Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>

shouze · 2017-07-26T16:38:53Z

builder/dockerfile/copy.go

@@ -56,6 +56,7 @@ type copyInstruction struct {
 	cmdName                 string
 	infos                   []copyInfo
 	dest                    string
+	flChown                 string


does fl stands for flag?

yes, used commonly in the codebase; although in this struct the flag value is being stored, so toss up whether I should be using the fl prefix at this point :)

Ya, maybe this should be user or something like that?

@dnephin could be a string serialized uid:gid id pair, maybe it worth to parse the chown option earluer & split into uid & gid in the struct in place of a single string?

If yes could be named destUid & destGid? Or simply uid & gid of course.

We have an object for that, idtools.IDPair

CWSpear · 2017-07-27T20:25:09Z

This was quite a journey. I Googled my issue (the whole "having to chown" stuff) and found an issue from 2 years ago.

Then I had to keep reading through massive threads, with both Docker contributors and the community supporting this, but kept having to pick out "continuing this thread [here]," and about 7 or 8 threads later, I find this, which is obviously a very recent PR.

So... where are we with all this? :)

estesp · 2017-07-27T20:30:50Z

Hopefully as close as it has ever been to reality! With the maintainers discussion and comments (like this one from issue #30110 I would expect it is just a matter of review and cleaning up a few things with my implementation.

Given @justincormack wants it, I assume he will be a huge supporter and reviewer of my PR, right Justin! =)

justincormack · 2017-07-28T10:27:01Z

builder/dockerfile/internals.go

+		return false
+	})
+	if err != nil {
+		return -1, err


Minor thing, but it is kind of odd returning -1 here, as -1 in chown specifically means "do not chown this part" eg so you can chown a user but not a group, but you are not actually using that, so returning 0 ie the empty int is probably clearer. I had to double check you were not using the value regardless of the error code...

Sure; was somewhat of a weird paranoia considering some future caller assuming that the translation was a valid uid/gid (0) by ignoring the err, but I agree that there is no actual valid reason to pass -1 given the err.

justincormack · 2017-07-28T10:29:15Z

integration-cli/docker_cli_build_test.go

+RUN echo 'test2:x:1002:' >> /etc/group
+ADD --chown=test1:1002 . /new_dir
+RUN ls -l /
+RUN [ $(ls -l / | grep new_dir | awk '{print $3":"$4}') = 'test1:test2' ]`)),


I would be inclined to add a test that does ls -ln to check the numeric values too as well as the names...

sounds good

justincormack · 2017-07-28T10:34:42Z

Some minor comments but otherwise looks good. I think it is ok to fail on named users on Windows for now, maybe have a test for platform rather than trying to look for /etc/passwd.

dnephin · 2017-07-28T17:07:28Z

builder/dockerfile/copy.go

+		// to the requested --chown flags as provided to the original command
+		if err := os.Lchown(dest, chownPair.UID, chownPair.GID); err != nil {
+			return errors.Wrapf(err, "failed to chown target directory")
+		}


This (and the isExistingDirectory() call above) is unnnecessary. It's already handled by fixPermissions()

except that it doesn't work for the chown case. I took the longest time trying to decide it if was worth adding complexity to fixPermissions() or not. The case that doesn't work is when the new directory is actually created by archive.CopyWithTar, so when checked for existence in fixPermissions, it does exist, so the wrong codepath is taken for the chown case. What I wanted was to be able to make an archiver with ChownOpts just for this ADD or COPY command, but after spending time in that code, it isn't possible without significant changes to add TarOptions cascading through a bunch of functions that don't take it today.

Ah that makes sense, which means this is probably a bug introduced when this was moved out of daemon.

I think the logic should still be in fixPermission(). We can pass in the argument as a param, which is what the old code did I believe.

dnephin · 2017-07-28T17:10:15Z

builder/dockerfile/copy.go

@@ -56,6 +56,7 @@ type copyInstruction struct {
 	cmdName                 string
 	infos                   []copyInfo
 	dest                    string
+	flChown                 string


s/flChown/idPair/ or something like that. At this point I don't think it's relevant that this came from a flag.

dnephin · 2017-07-28T17:12:38Z

builder/dockerfile/dispatchers.go

@@ -157,6 +158,7 @@ func add(req dispatchRequest) error {
 	if err != nil {
 		return err
 	}
+	copyInstruction.flChown = flChown.Value


Are we sure we want to add this to ADD as well? We only have --from on COPY, and I think that's by design.

I thought ADD was kind of being deprecated/frozen.

Both prior PRs were adding the same feature to both so I simply followed suit. I didn't see anything in recent discussions narrowing the focus, but I'm willing to hear arguments either way.

I think we should add it to both (even though we don't want to make ADD more magical); if we don't add it to ADD, people using it for adding remote files won't be helped by this PR.

Shouldn't people using ADD for remote files be converting things to RUN curl + COPY anyway? So by not adding it to ADD we're helping nudge people in the right direction. It doesn't really make sense to have two almost identical instructions.

dnephin · 2017-07-28T17:13:23Z

builder/dockerfile/dispatchers.go

@@ -187,6 +190,7 @@ func dispatchCopy(req dispatchRequest) error {
 	if err != nil {
 		return err
 	}
+	copyInstruction.flChown = flChown.Value


I think we should parse the Value here so we can fail fast when it's wrong. That way we can store an IDPair in copyInstruction as well.

we have to be at a spot where we have the mount information for the container fs.. hence where it happens in this PR. Otherwise there is no source of truth to translate name -> IDs. I can look again and make sure I do that translation as early as we have that info.

dnephin · 2017-07-28T17:14:17Z

builder/dockerfile/internals.go

+	if inst.flChown != "" {
+		commentStr = fmt.Sprintf("%s --chown=%s %s in %s ", inst.cmdName, inst.flChown, srcHash, inst.dest)
+	} else {
+		commentStr = fmt.Sprintf("%s %s in %s ", inst.cmdName, srcHash, inst.dest)


minor nit, these lines are mostly the same so this could just be a variable for chownComment and use %s %s%s %s in %s as the format.

Could even be a separate function to keep it from cluttering the logic in performCopy()

I should have known this comment was coming! :) Even though I factored it out further per your comments on prior PRs for this, there is more trimming possible for sure :)

dnephin · 2017-07-28T17:15:41Z

builder/dockerfile/internals.go

+		if err != nil {
+			return errors.Wrapf(err, "unable to convert uid/gid to host mapping")
+		}
+	}


Please extract this whole block (parsing the uid/gid pair) into a new function, and maybe call that function from dispatchCopy() instead of here.

dnephin · 2017-07-28T17:17:58Z

builder/dockerfile/internals.go

+	return parts[0], parts[1]
+}
+
+func lookupUser(userStr, filepath string) (int, error) {


I think this already exists as idtools.LookupUser() / LookupUID()? Can we re-use either of those?

No, per my prior comment

dnephin · 2017-07-28T17:18:23Z

builder/dockerfile/internals.go

+	return users[0].Uid, nil
+}
+
+func lookupGroup(groupStr, filepath string) (int, error) {


I think this already exists as idtools.LookupGroup() / LookupGID()? Can we re-use either of those?

dnephin · 2017-07-28T17:20:44Z

builder/dockerfile/internals.go

@@ -137,6 +171,53 @@ func (b *Builder) performCopy(state *dispatchState, inst copyInstruction) error
 	return b.exportImage(state, imageMount, runConfigWithCommentCmd)
 }

+func parseChownFlag(flChown string) (string, string) {


I'm surprised this doesn't exist in pkg/idtools. I think it should probably move to idtools.ParseIDPair(string) (IDPair, error)

because the context is quite different and requires different knowledge in each situation where it is used. pkg/idtools only considers the case where the host /etc/{passwd,group} files are used for translation, as well as possible use of getent. That would be totally invalid for this case, where we need the container files to be used.

Could we refactor it further to share more of the code?

would need to introduce some context yup, but at the price of complexity?

I don't think the complexity changes much, it's just being moved somewhere more appropriate, so there's less duplicate code to contain.

dnephin · 2017-07-28T17:23:15Z

Looks like windows failures are real because of lchown. I think maybe fixed by removing the code I suggested is not necessary.

estesp · 2017-07-28T17:40:54Z

Yes, the Windows failures are real--my delay in properly handling here yet is on the design point of whether to fail or silently ignore; seems ignore (by using some sort of _windows.go variation of the lookups/chown codepaths) is probably better if you can see a case for Dockerfile sharing, but that seems unlikely in lots of cases anyway. Do we want to fail on a --chown flag on ADD/COPY on Windows?

shouze · 2017-07-31T10:45:39Z

@estesp @dnephin can/will this PR fix #32816?

estesp · 2017-08-10T17:55:00Z

@dnephin and others; PR has been updated. A bit of vacation delay, sorry :)

At this point I believe the only review comments not handled is the recommended consolidation of pkg/idtools parsing of host /etc/{passwd,group} files and the new parsing here of container specific /etc/ files.

After looking through the idtools package use of the opencontainers/runc/libcontainer/user package and this new code's use, I don't see any effective way of combining these without simply adding complexity. In essence, the user package offers two paths:

the "easy" default path of letting that package read host default files; and
the more complex path of providing that code filepaths or open readers along with a custom filter function to find a specific entry.

The idtools package uses the "easy" path, because that fits the need of that package, and this new PR code uses the other path, because we want to reference specific paths inside the container we're using during build. Unless we force idtools to take the more complex path, and basically rewrite the "easy path" as a consumer of the functions already in that user package. Either way we end up duplicating code by copying functions that currently exist in user. I believe the way it is now, we have two consumers of the user package from libcontainer; they just use two different layers of that functionality.

g13013 · 2017-08-13T16:37:37Z

Any news on that ?

dnephin · 2017-08-15T16:04:23Z

builder/dockerfile/internals.go

+		// if no group specified, use the user spec as group as well
+		userStr, grpStr = parts[0], parts[0]
+	}
+	userStr, grpStr = parts[0], parts[1]


This line will will panic when len(parts) == 1 , it's not in an else.

This function should have unit tests.

dnephin · 2017-08-15T16:07:37Z

integration-cli/docker_cli_build_test.go

@@ -602,6 +602,36 @@ RUN [ $(cat "/test dir/test_file6") = 'test6' ]`, command, command, command, com
 	}
 }

+func (s *DockerSuite) TestBuildChownFlagAdd(c *check.C) {


I'm still not convinced we want to add this to ADD.

We definitely don't need two slow integration tests for this. The code path is 99% the same. Let's just have a single one for COPY. Even better would be to add it to an existing integration test that does COPY, but I'm not sure if we have one in the API suite yet.

integration-cli is deprecated, so these should be API tests. We just merged integration/ so there isn't a lot of work there yet, but at least this should be in docker_api_build_test.go

dnephin · 2017-08-15T16:09:03Z

builder/dockerfile/internals.go

+	return chownPair, nil
+}
+
+func lookupUser(userStr, filepath string) (int, error) {


This (and lookupGroup) should have unit tests for each case

dnephin · 2017-08-15T16:10:36Z

I believe the way it is now, we have two consumers of the user package from libcontainer; they just use two different layers of that functionality.

Thanks for looking into it

estesp · 2017-08-16T19:01:25Z

@dnephin bugs created during the re-work of those functions are fixed and unit tests added that test all formats of the input string and with/without userns mappings. This tests the parseChownFlag and also the lookupUser/lookupGroup functions. Let me know if you see any holes there.

I understand the comments re: testing, but as you note integration is barely started, and I don't see any movement of cli_* content to api_* yet. I could see potentially adding this to build API, but that will take some extra work as there is very poor scaffolding for standard use of test files, etc. Before I do that can we resolve the ADD and COPY discussion? I wasn't in the maintainer's meeting where it was recommended this work proceed, so I haven't been given any clue as to whether there as a debate around just COPY vs. COPY + ADD. If you mean only one test given the code paths are very similar, that makes sense to me.

dnephin · 2017-08-17T15:26:17Z

here is very poor scaffolding for standard use of test files

Not true. All the same scaffolding works for the api tests. See https://github.com/moby/moby/blob/master/integration-cli/docker_api_build_test.go#L288-L306

I don't see any movement of cli_* content to api_*

There should be. Every PR I have seen has called out to change the cli test to an api test

vipseixas · 2017-11-13T15:54:37Z

--chown does not work for compressed files (eg: tgz), it seems to only change the owner of the compressed file itself and then ignore the extracted content.

Is it the intended behavior?

thaJeztah · 2017-11-13T16:09:40Z

@vipseixas I'd say; yes, because it'll only change permissions on the file you add. Archives themselves can have permissions set as well, which are preserved (if I'm not mistaken).

Given that automatically extracting archives only works for local files; can you explain your use-case?
i.e., instead of compressing local files, adding them to the image, and then decompressing, why not add the uncompressed files directly?

vipseixas · 2017-11-13T16:22:23Z

@thaJeztah I have a 3rd party installation (PHP code) that comes as a 180MB tar.gz file and I have to open it inside Apache's /var/www. The uncompressed files grows to ~570MB so it would be easier to store the compressed file with my Dockerfile. But if I ADD the compressed file and use a "RUN chown" I end up with 2 huge layers in my image.

derikson · 2017-11-13T16:41:58Z

@vipseixas as a workaround, you could use multi-stage build to extract the files in one build stage, then COPY with chown in the second stage. This would prevent the final image from having the extra layer.

vipseixas · 2017-11-13T17:32:04Z

@derikson Your suggestion worked perfectly! I was not aware of this new feature, thanx.

marius311 · 2017-11-17T14:33:43Z

This is awesome, thanks guys! Probably one of my most desired features for years.

At the risk of being one those never-satisfied Docker users though :) ...

@thaJeztah:

It's not a bug; variable expansion in options is not supported at this moment

Is this planned at all and if so is there an issue we could track this at?

thaJeztah · 2017-11-17T14:43:57Z

Is this planned at all and if so is there an issue we could track this at?

A feature request for that was opened here: #35018 (but not decided on yet 😅)

khatribharat · 2017-12-27T09:42:32Z

This isn't yet reflected in the Dockerfile doc

thaJeztah · 2017-12-27T10:57:35Z

@khatribharat docs took some time to get merged (see docker/cli#467), but will be updated once Docker 17.12 has been released

Ever since moby/moby/pull/34263 became merged, there's no need to shuffle users in order to get the correct ownership on non-root users

moby/moby#34263

Add --chown flag to Dockerfile ADD and COPY

858fad3

Rebased by @estesp Signed-off-by: Kara Alexandra <kalexandra@us.ibm.com> Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>

GordonTheTurtle added the status/0-triage label Jul 26, 2017

estesp force-pushed the chown-flag-add-copy branch from 6836f61 to 6c11df6 Compare July 26, 2017 16:32

estesp mentioned this pull request Jul 26, 2017

Add --chown flag to Dockerfile ADD and COPY #28499

Closed

estesp added status/1-design-review area/builder and removed status/0-triage labels Jul 26, 2017

shouze reviewed Jul 26, 2017

View reviewed changes

justincormack reviewed Jul 28, 2017

View reviewed changes

dnephin suggested changes Jul 28, 2017

View reviewed changes

kaikuchn mentioned this pull request Jul 31, 2017

Dockerfile: ADD does not honor USER: files always owned by root #6119

Closed

mightyiam mentioned this pull request Aug 6, 2017

builder: make ADD and COPY obey the USER directive #21088

Closed

GordonTheTurtle assigned ehazlett Aug 10, 2017

estesp force-pushed the chown-flag-add-copy branch from 6c11df6 to 8f9561c Compare August 10, 2017 17:12

estesp requested a review from tonistiigi as a code owner August 10, 2017 17:12

estesp force-pushed the chown-flag-add-copy branch from 8f9561c to 1996f01 Compare August 10, 2017 17:26

dnephin reviewed Aug 15, 2017

View reviewed changes

estesp force-pushed the chown-flag-add-copy branch from 1996f01 to a0885ff Compare August 16, 2017 17:44

baronfel mentioned this pull request Nov 9, 2017

Please add documentation for the --chown flag to COPY/ADD docker/docs#5258

Closed

salah-khan mentioned this pull request Nov 15, 2017

Add --chown flag support for ADD/COPY commands for Windows #35507

Open

thaJeztah mentioned this pull request Nov 16, 2017

ADD archive file with the --chown flag #35525

Closed

ghost mentioned this pull request Nov 22, 2017

Add chown flag to ADD and COPY commands bmuschko/gradle-docker-plugin#497

Closed

rhatdan mentioned this pull request Nov 27, 2017

Add --chown option to add/copy containers/buildah#336

Closed

marius311 mentioned this pull request Dec 6, 2017

Using 'root' user accounts instead of non-root user within docker containers? marius311/boinc-server-docker#14

Closed

thaJeztah mentioned this pull request Jan 26, 2018

Unknown flag: chown #35731

Closed

aglorei added a commit to aglorei/pokemon_battle_rails that referenced this pull request Feb 2, 2018

Use new COPY --chown flag

ce52764

Ever since moby/moby/pull/34263 became merged, there's no need to shuffle users in order to get the correct ownership on non-root users

thaJeztah mentioned this pull request Feb 19, 2018

uid and gid mismatch when using 'ADD --chown' in Dockerfile #36349

Closed

omercnet mentioned this pull request Feb 26, 2018

Add --chown to WORKDIR #36408

Closed

lsl mentioned this pull request Mar 13, 2018

Build: Exploring Docker multi-stage and best practices Automattic/wp-calypso#19962

Closed

thaJeztah mentioned this pull request Mar 22, 2018

Builds on hub not supporting ADD & chown docker/hub-feedback#1427

Closed

simnalamburt mentioned this pull request Mar 10, 2023

[proposal] Emit warning if ADD --chown flag is used with archive file moby/buildkit#3700

Open

sinhyeok-cp added a commit to EOSYS-io/eoshub.io that referenced this pull request Jun 27, 2018

Add --chown flag to COPY command

62afd9c

moby/moby#34263

gvenzl mentioned this pull request Jul 25, 2018

Add Dockerfile for multi-stage build of oracle/database:18.3.0-se2 oracle/docker-images#946

Closed

gucharbon mentioned this pull request Sep 12, 2018

301 docker sukeesh/Jarvis#318

Merged

erik-wramner mentioned this pull request Oct 3, 2020

Remove write access to group and others to Amavis configuration files docker-mailserver/docker-mailserver#1632

Merged

jamesmstone mentioned this pull request Feb 25, 2021

Docker and docker-compose support OpenBB-finance/OpenBB#13

Closed

janhn mentioned this pull request Apr 8, 2021

Improve Dockerfile romanz/electrs#392

Merged

thaJeztah mentioned this pull request May 24, 2022

[feature request/propsal] add option --keep-ownership for ADD and COPY commands moby/buildkit#2875

Open

andersonpem mentioned this pull request Feb 13, 2024

Docs: Dockerfile: Updated the ADD statement mlocati/docker-php-extension-installer#890

Merged

Add --chown flag to ADD/COPY commands #34263

Add --chown flag to ADD/COPY commands #34263

Conversation

estesp commented Jul 26, 2017 • edited Loading

shouze Jul 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shouze Jul 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CWSpear commented Jul 27, 2017

estesp commented Jul 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justincormack Jul 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justincormack commented Jul 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dnephin commented Jul 28, 2017

estesp commented Jul 28, 2017

shouze commented Jul 31, 2017

estesp commented Aug 10, 2017

g13013 commented Aug 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dnephin commented Aug 15, 2017

estesp commented Aug 16, 2017

dnephin commented Aug 17, 2017

vipseixas commented Nov 13, 2017

thaJeztah commented Nov 13, 2017

vipseixas commented Nov 13, 2017

derikson commented Nov 13, 2017

vipseixas commented Nov 13, 2017

marius311 commented Nov 17, 2017

thaJeztah commented Nov 17, 2017

khatribharat commented Dec 27, 2017

thaJeztah commented Dec 27, 2017

estesp commented Jul 26, 2017 •

edited

Loading

shouze Jul 26, 2017 •

edited

Loading

shouze Jul 26, 2017 •

edited

Loading

justincormack Jul 28, 2017 •

edited

Loading