add,copy: if multiple sources are used in `ADD` or `COPY` then dest must end with a `slash` #4183

flouthoc · 2022-08-11T04:53:26Z

When using ADD or COPY enforce the condition that destination must
look like a directory by making sure that destination ends with a slash
/.

This ensures that we don't hit undefined behaviour for example with
the use case COPY some/* /oncontainer it is not clear that
oncontainer will be a directory or a file, users expect it to be a
directory if * wildcard resolves to multiple files and logically if
wildcard resolves to single file then it should be a directory, since
this condition is undefined and not in parity with docker so lets drop
support for it.

Docker's defined rule:

If multiple <src> resources are specified, either directly or due to the
use of a wildcard, then <dest> must be a directory, and it must end with a slash /.

Reference: https://docs.docker.com/engine/reference/builder/#add

Closes: #4167

openshift-ci · 2022-08-11T04:55:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: flouthoc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [flouthoc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rhatdan · 2022-08-11T19:20:14Z

add.go

+		// Check length before accessing `0`
+		// to avoid `index out of range`.
+		if len(sources) == 1 {
+			if strings.HasSuffix(sources[0], "*") {


Should you check for len(sources)==0 and return an error?

Why is a suffix of "*" the special case here?

Special case is here to match with the current behavior of docker #4167 (comment)

rhatdan · 2022-08-11T19:21:08Z

add.go

+		}
+	}
+	if multipleSources {
+		if !strings.HasSuffix(destination, "/") {


Does docker enforce this? Or does it just check if destination is not a directory then throw an error?

Just chiming in that if the directory already exists, podman/buildah does the expected (?) thing and copies all the matches of the wildcard into the existing directory

Re-using the reproducer from #4167, if you modify the Dockerfile to this

FROM alpine:latest as builder RUN mkdir /spam COPY spam/*.txt /spam

Then it copies all the files

@rhatdan I think docker checks this cause rule is

* If multiple <src> resources are specified, either directly or due to the use of a wildcard, then <dest> must be a directory, and it must end with a slash /.

https://docs.docker.com/engine/reference/builder/#add

I still believe this should be done on in the copier.

Basically if the SRC contains more then one file and the DEST is not a directory, then it should return an error.

The directory should not be checked for ending with /

@rhatdan Yes the function is validateSourceAndDest to validates this using filepath.Glob( there is also a test which verifies the same, i can add few more test cases if this is a concern and move it to copier.

We're already globbing inside a chroot further down, when we call copier.Stat(), and we're probably better off doing this sort of check just after that, when we have all of the facts. Doing this before that means that the current iteration of this patch passes URLs for remote locations to filepath.Glob(), which is a bug.

I guess they do document the following:

If <dest> does not end with a trailing slash, it will be considered a regular file and the contents of <src> will be written at <dest>.

But @nalind question above has not been addressed

Basically if the SRC contains more then one file and the DEST is not a directory, then it should return an error.

The directory should not be checked for ending with /

+1 to this, it sounds reasonable to me that if the destination already exists then the trailing slash doesn't need to be enforced. It wouldn't be 100% consistent with docker, but isn't it likely that some users are already doing this? E.g. copying things to /etc or other standard dirs

@chmeliik Yes that is already addressed I just need to move code around as per @nalind 's suggestion here #4183 (comment) , I'll amend this.

Done moved to just after we split local and remote sources and only validating local sources now.

tests/add.bats

TomSweeneyRedHat · 2022-08-24T19:01:20Z

I know I lost my mind a long time ago, but didn't we enforce an ending slash at one point and then removed the enforcement as it was counter to Docker?

github-actions · 2022-09-26T00:14:38Z

A friendly reminder that this PR had no activity for 30 days.

rhatdan · 2022-09-26T17:10:53Z

@flouthoc any chance you get get back to this one?

flouthoc · 2022-09-27T10:31:25Z

@flouthoc any chance you get get back to this one?

Sure yes, I'll amend this again in this week.

github-actions · 2022-11-15T00:11:57Z

A friendly reminder that this PR had no activity for 30 days.

rhatdan · 2022-11-15T14:57:37Z

@flouthoc Any movement on this?

github-actions · 2022-12-16T00:09:36Z

A friendly reminder that this PR had no activity for 30 days.

rhatdan · 2022-12-17T10:47:25Z

@flouthoc Reminder

github-actions · 2023-01-17T00:10:26Z

A friendly reminder that this PR had no activity for 30 days.

rhatdan · 2023-01-19T00:22:31Z

@flouthoc after the release lets get back to this one.

flouthoc · 2023-01-26T10:58:50Z

@rhatdan Yes getting back on this and closing this one.

When using `ADD` or `COPY` enforce the condition that destination must look like a directory by making sure that destination ends with a slash `/`. This ensures that we don't hit undefined behaviour for example with the use case `COPY some/* /oncontainer` it is not clear that `oncontainer` will be a directory or a file, users expect it to be a directory if `*` wildcard resolves to multiple files and logically if wildcard resolves to single file then it should be a directory, since this condition is undefined and not in parity with docker so lets drop support for it. Docker's defined rule: * If multiple <src> resources are specified, either directly or due to the use of a wildcard, then <dest> must be a directory, and it must end with a slash /. Reference: https://docs.docker.com/engine/reference/builder/#add Closes: containers#4167 Signed-off-by: Aditya R <arajan@redhat.com>

Accoring to new validation in ADD and COPY if there are multiple sources then destination must specify that it is a directory Signed-off-by: Aditya R <arajan@redhat.com>

flouthoc · 2023-01-26T12:54:49Z

@nalind @rhatdan PTAL

github-actions · 2023-02-26T00:11:30Z

A friendly reminder that this PR had no activity for 30 days.

rhatdan · 2023-03-01T14:03:39Z

LGTM
@nalind @vrothberg @giuseppe @umohnani8 PTAL

TomSweeneyRedHat · 2023-03-01T20:20:37Z

LGTM, but would like a head nod from @vrothberg or @nalind to make sure comments were addressed.

github-actions · 2023-04-02T00:11:14Z

A friendly reminder that this PR had no activity for 30 days.

rhatdan · 2023-04-03T20:54:52Z

This still needs to be reworked.

openshift-ci bot added the approved label Aug 11, 2022

flouthoc mentioned this pull request Aug 11, 2022

Inconsistent behavior when COPYing multiple files to /dest without trailing slash #4167

Open

flouthoc force-pushed the add-disallow-dest-dir-without-slash branch from 7274705 to 7ab4388 Compare August 11, 2022 06:00

rhatdan reviewed Aug 11, 2022

View reviewed changes

nalind reviewed Aug 16, 2022

View reviewed changes

tests/add.bats Outdated Show resolved Hide resolved

nalind reviewed Aug 16, 2022

View reviewed changes

tests/add.bats Outdated Show resolved Hide resolved

flouthoc force-pushed the add-disallow-dest-dir-without-slash branch 3 times, most recently from e60a9ef to f1d47cc Compare August 23, 2022 17:31

github-actions bot added the stale-pr label Sep 26, 2022

github-actions bot removed the stale-pr label Oct 15, 2022

github-actions bot added the stale-pr label Nov 15, 2022

rhatdan removed the stale-pr label Nov 15, 2022

github-actions bot added the stale-pr label Dec 16, 2022

github-actions bot removed the stale-pr label Dec 18, 2022

github-actions bot added the stale-pr label Jan 17, 2023

rhatdan removed the stale-pr label Jan 19, 2023

flouthoc force-pushed the add-disallow-dest-dir-without-slash branch from f1d47cc to 33bcb72 Compare January 26, 2023 11:12

flouthoc added 2 commits January 26, 2023 16:44

test: retrofit tests for new validation

86a9ac4

Accoring to new validation in ADD and COPY if there are multiple sources then destination must specify that it is a directory Signed-off-by: Aditya R <arajan@redhat.com>

flouthoc force-pushed the add-disallow-dest-dir-without-slash branch from 33bcb72 to 86a9ac4 Compare January 26, 2023 11:14

flouthoc requested review from nalind, rhatdan and chmeliik January 26, 2023 12:54

github-actions bot added the stale-pr label Feb 26, 2023

github-actions bot removed the stale-pr label Mar 2, 2023

github-actions bot added the stale-pr label Apr 2, 2023

github-actions bot removed the stale-pr label Apr 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add,copy: if multiple sources are used in `ADD` or `COPY` then dest must end with a `slash` #4183

add,copy: if multiple sources are used in `ADD` or `COPY` then dest must end with a `slash` #4183

flouthoc commented Aug 11, 2022

openshift-ci bot commented Aug 11, 2022

rhatdan Aug 11, 2022

nalind Aug 16, 2022

flouthoc Aug 23, 2022

rhatdan Aug 11, 2022

chmeliik Aug 16, 2022

flouthoc Aug 24, 2022 •

edited

Loading

rhatdan Aug 24, 2022

flouthoc Aug 24, 2022

nalind Aug 24, 2022

rhatdan Aug 25, 2022

chmeliik Aug 26, 2022

flouthoc Aug 26, 2022

flouthoc Jan 26, 2023

TomSweeneyRedHat commented Aug 24, 2022

github-actions bot commented Sep 26, 2022

rhatdan commented Sep 26, 2022

flouthoc commented Sep 27, 2022

github-actions bot commented Nov 15, 2022

rhatdan commented Nov 15, 2022

github-actions bot commented Dec 16, 2022

rhatdan commented Dec 17, 2022

github-actions bot commented Jan 17, 2023

rhatdan commented Jan 19, 2023

flouthoc commented Jan 26, 2023

flouthoc commented Jan 26, 2023

github-actions bot commented Feb 26, 2023

rhatdan commented Mar 1, 2023

TomSweeneyRedHat commented Mar 1, 2023

github-actions bot commented Apr 2, 2023

rhatdan commented Apr 3, 2023

add,copy: if multiple sources are used in ADD or COPY then dest must end with a slash #4183

Are you sure you want to change the base?

add,copy: if multiple sources are used in ADD or COPY then dest must end with a slash #4183

Conversation

flouthoc commented Aug 11, 2022

openshift-ci bot commented Aug 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flouthoc Aug 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomSweeneyRedHat commented Aug 24, 2022

github-actions bot commented Sep 26, 2022

rhatdan commented Sep 26, 2022

flouthoc commented Sep 27, 2022

github-actions bot commented Nov 15, 2022

rhatdan commented Nov 15, 2022

github-actions bot commented Dec 16, 2022

rhatdan commented Dec 17, 2022

github-actions bot commented Jan 17, 2023

rhatdan commented Jan 19, 2023

flouthoc commented Jan 26, 2023

flouthoc commented Jan 26, 2023

github-actions bot commented Feb 26, 2023

rhatdan commented Mar 1, 2023

TomSweeneyRedHat commented Mar 1, 2023

github-actions bot commented Apr 2, 2023

rhatdan commented Apr 3, 2023

add,copy: if multiple sources are used in `ADD` or `COPY` then dest must end with a `slash` #4183

add,copy: if multiple sources are used in `ADD` or `COPY` then dest must end with a `slash` #4183

flouthoc Aug 24, 2022 •

edited

Loading