Disallow verify_path() failures from fast-import

[newren]

Since en/fast-import-path-sanitize has already made it to next, this commit is based on that. (See https://lore.kernel.org/git/pull.1831.v2.git.1732561248717.gitgitgadget@gmail.com/ for discussion of that series.)

Changes relative to that commit: this fixes up the error message as suggested by Kristoffer, and makes the checks more encompassing as suggested by Patrick and Peff -- in particular, using verify_path() as suggested by Peff.

Changes since v1:

• Moved the check to a higher level, as suggested by Peff.

cc: Eric Sunshine sunshine@sunshineco.com
cc: Patrick Steinhardt ps@pks.im
cc: "Kristoffer Haugsbakk" kristofferhaugsbakk@fastmail.com
cc: Jeff King peff@peff.net
cc: Elijah Newren newren@gmail.com

[newren]

/submit

[gitgitgadget]

Submitted as pull.1832.git.1732740464398.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-1832/newren/disallow-verify-path-fast-import-v1

To fetch this version to local tag pr-1832/newren/disallow-verify-path-fast-import-v1:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-1832/newren/disallow-verify-path-fast-import-v1

[gitgitgadget]

On the Git mailing list, Junio C Hamano wrote (reply to this):

"Elijah Newren via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Elijah Newren <newren@gmail.com>
>
> Instead of just disallowing '.' and '..', make use of verify_path() to
> ensure that fast-import will disallow anything we wouldn't allow into
> the index, such as anything under .git/, .gitmodules as a symlink, or
> a dos drive prefix on Windows.
>
> Since a few fast-export and fast-import tests that tried to stress-test
> the correct handling of quoting relied on filenames that fail
> is_valid_win32_path(), such as spaces or periods at the end of filenames
> or backslashes within the filename, turn off core.protectNTFS for those
> tests to ensure they keep passing.
>
> Signed-off-by: Elijah Newren <newren@gmail.com>
> ---
>     Disallow verify_path() failures from fast-import
>     
>     Since en/fast-import-path-sanitize has already made it to next, this
>     commit is based on that. (See
>     https://lore.kernel.org/git/pull.1831.v2.git.1732561248717.gitgitgadget@gmail.com/
>     for discussion of that series.)
>     
>     Changes relative to that commit: this fixes up the error message as
>     suggested by Kristoffer, and makes the checks more encompassing as
>     suggested by Patrick and Peff -- in particular, using verify_path() as
>     suggested by Peff.
>
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1832%2Fnewren%2Fdisallow-verify-path-fast-import-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1832/newren/disallow-verify-path-fast-import-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/1832

Thanks, all.  Looking good to me.

Will queue.

[gitgitgadget]

This patch series was integrated into seen via git@89004b9.

[gitgitgadget]

This patch series was integrated into seen via git@da9ae6e.

[gitgitgadget]

On the Git mailing list, Jeff King wrote (reply to this):

On Wed, Nov 27, 2024 at 08:47:44PM +0000, Elijah Newren via GitGitGadget wrote:

> Instead of just disallowing '.' and '..', make use of verify_path() to
> ensure that fast-import will disallow anything we wouldn't allow into
> the index, such as anything under .git/, .gitmodules as a symlink, or
> a dos drive prefix on Windows.
> 
> Since a few fast-export and fast-import tests that tried to stress-test
> the correct handling of quoting relied on filenames that fail
> is_valid_win32_path(), such as spaces or periods at the end of filenames
> or backslashes within the filename, turn off core.protectNTFS for those
> tests to ensure they keep passing.

Great, thanks for following up on this. I think it will work as
advertised, though...

> @@ -1413,6 +1414,8 @@ static int tree_content_set(
>  		die("Empty path component found in input");
>  	if (!*slash1 && !S_ISDIR(mode) && subtree)
>  		die("Non-directories cannot have subtrees");
> +	if (!verify_path(p, mode))
> +		die("invalid path '%s'", p);

This spot is over-verifying, I think, because it's recursive. Given the
path foo/bar/baz, it will see "foo/bar/baz", then "bar/baz", then "baz".
But just the first one should be sufficient to feed to verify_path().

I'd have expected the check when we parse the path in file_change_m().
That would also require touching other callers, too. One option would be
to put a non-recursive wrapper around tree_content_set(), and add the
check there.

But looking at those other callers, I think it's really just
file_change_cr() that maters. The other call in do_change_note_fanout()
is using a notes path that we've constructed ourselves (so it's not
wrong to check, but probably pointless).

The patch below passes your tests (though perhaps it would be worth
adding an explicit check of a copy/rename). Is it worth worrying about
the efficiency of the extra calls? I'm not sure, and I didn't measure.
But this just seems less surprising to me overall.

(Both your patch and what I've shown below do not verify deletions from
the tree, but I think that's fine; such a name would not exist in the
tree in the first place).

-Peff

---
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index bb4b769c7c..265d1b7d52 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -1414,8 +1414,6 @@ static int tree_content_set(
 		die("Empty path component found in input");
 	if (!*slash1 && !S_ISDIR(mode) && subtree)
 		die("Non-directories cannot have subtrees");
-	if (!verify_path(p, mode))
-		die("invalid path '%s'", p);
 
 	if (!root->tree)
 		load_tree(root);
@@ -2417,6 +2415,9 @@ static void file_change_m(const char *p, struct branch *b)
 		tree_content_replace(&b->branch_tree, &oid, mode, NULL);
 		return;
 	}
+
+	if (!verify_path(path.buf, mode))
+		die("invalid path '%s'", path.buf);
 	tree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);
 }
 
@@ -2454,6 +2455,8 @@ static void file_change_cr(const char *p, struct branch *b, int rename)
 			leaf.tree);
 		return;
 	}
+	if (!verify_path(dest.buf, leaf.versions[1].mode))
+		die("invalid path '%s'", dest.buf);
 	tree_content_set(&b->branch_tree, dest.buf,
 		&leaf.versions[1].oid,
 		leaf.versions[1].mode,

[newren]

/submit

[gitgitgadget]

Submitted as pull.1832.v2.git.1732928970059.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-1832/newren/disallow-verify-path-fast-import-v2

To fetch this version to local tag pr-1832/newren/disallow-verify-path-fast-import-v2:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-1832/newren/disallow-verify-path-fast-import-v2

[gitgitgadget]

On the Git mailing list, Jeff King wrote (reply to this):

On Sat, Nov 30, 2024 at 01:09:29AM +0000, Elijah Newren via GitGitGadget wrote:

>     Changes since v1:
>     
>      * Moved the check to a higher level, as suggested by Peff.

Thanks, the code change looks good. Is it worth tweaking one of the
tests to do "R innocent-path .git/evil"? Otherwise I don't think there's
any coverage of the file_change_cr() call at all.

-Peff

[gitgitgadget]

On the Git mailing list, Junio C Hamano wrote (reply to this):

"Elijah Newren via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Elijah Newren <newren@gmail.com>
>
> Instead of just disallowing '.' and '..', make use of verify_path() to
> ensure that fast-import will disallow anything we wouldn't allow into
> the index, such as anything under .git/, .gitmodules as a symlink, or
> a dos drive prefix on Windows.
>
> Since a few fast-export and fast-import tests that tried to stress-test
> the correct handling of quoting relied on filenames that fail
> is_valid_win32_path(), such as spaces or periods at the end of filenames
> or backslashes within the filename, turn off core.protectNTFS for those
> tests to ensure they keep passing.
>
> Helped-by: Jeff King <peff@peff.net>
> Signed-off-by: Elijah Newren <newren@gmail.com>
> ---
>     Disallow verify_path() failures from fast-import
>     
>     Since en/fast-import-path-sanitize has already made it to next, this
>     commit is based on that. (See
>     https://lore.kernel.org/git/pull.1831.v2.git.1732561248717.gitgitgadget@gmail.com/
>     for discussion of that series.)

Ah, sorry and thanks.

Will queue.

[gitgitgadget]

This patch series was integrated into seen via git@2afaea3.

[gitgitgadget]

This patch series was integrated into next via git@2932fa5.

[gitgitgadget]

This patch series was integrated into seen via git@92d94d3.

[gitgitgadget]

On the Git mailing list, Elijah Newren wrote (reply to this):

On Sun, Dec 1, 2024 at 1:40 PM Jeff King <peff@peff.net> wrote:
>
> On Sat, Nov 30, 2024 at 01:09:29AM +0000, Elijah Newren via GitGitGadget wrote:
>
> >     Changes since v1:
> >
> >      * Moved the check to a higher level, as suggested by Peff.
>
> Thanks, the code change looks good. Is it worth tweaking one of the
> tests to do "R innocent-path .git/evil"? Otherwise I don't think there's
> any coverage of the file_change_cr() call at all.
>
> -Peff

I would say yes, but since this patch too has made it to next and is
marked for master, I'm kinda tempted to just leave it as-is...

[gitgitgadget]

User Elijah Newren <newren@gmail.com> has been added to the cc: list.

[gitgitgadget]

On the Git mailing list, Jeff King wrote (reply to this):

On Tue, Dec 03, 2024 at 12:01:51AM -0800, Elijah Newren wrote:

> > On Sat, Nov 30, 2024 at 01:09:29AM +0000, Elijah Newren via GitGitGadget wrote:
> >
> > >     Changes since v1:
> > >
> > >      * Moved the check to a higher level, as suggested by Peff.
> >
> > Thanks, the code change looks good. Is it worth tweaking one of the
> > tests to do "R innocent-path .git/evil"? Otherwise I don't think there's
> > any coverage of the file_change_cr() call at all.
> 
> I would say yes, but since this patch too has made it to next and is
> marked for master, I'm kinda tempted to just leave it as-is...

Is is tempting. :) I wrote this up, though, which can just go on top (of
en/fast-import-verify-path).

-Peff

-- >8 --
Subject: [PATCH] t9300: test verification of renamed paths

Commit da91a90c2f (fast-import: disallow more path components,
2024-11-30) added two separate verify_path() calls (one for
added/modified files, and one for renames/copies). But our tests only
exercise the first one. Let's protect ourselves against regressions by
tweaking one of the tests to rename into the bad path. There are
adjacent tests that will stay as additions, so now both calls are
covered.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t9300-fast-import.sh | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index e2b1db6bc2..fd01a2353c 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -553,9 +553,16 @@ test_expect_success 'B: fail on invalid file path of .' '
 	commit refs/heads/badpath
 	committer Name <email> $GIT_COMMITTER_DATE
 	data <<COMMIT
-	Commit Message
+	Good path
+	COMMIT
+	M 100644 :1 ok-path
+
+	commit refs/heads/badpath
+	committer Name <email> $GIT_COMMITTER_DATE
+	data <<COMMIT
+	Bad path
 	COMMIT
-	M 100644 :1 ./invalid-path
+	R ok-path ./invalid-path
 	INPUT_END
 
 	test_when_finished "git update-ref -d refs/heads/badpath" &&
-- 
2.47.1.707.g92f6f18526

[gitgitgadget]

On the Git mailing list, Junio C Hamano wrote (reply to this):

Elijah Newren <newren@gmail.com> writes:

> On Sun, Dec 1, 2024 at 1:40 PM Jeff King <peff@peff.net> wrote:
>>
>> On Sat, Nov 30, 2024 at 01:09:29AM +0000, Elijah Newren via GitGitGadget wrote:
>>
>> >     Changes since v1:
>> >
>> >      * Moved the check to a higher level, as suggested by Peff.
>>
>> Thanks, the code change looks good. Is it worth tweaking one of the
>> tests to do "R innocent-path .git/evil"? Otherwise I don't think there's
>> any coverage of the file_change_cr() call at all.
>>
>> -Peff
>
> I would say yes, but since this patch too has made it to next and is
> marked for master, I'm kinda tempted to just leave it as-is...

It is perfectly OK to have a follow-up patch that adds a test or two
;-)

[gitgitgadget]

On the Git mailing list, Elijah Newren wrote (reply to this):

On Tue, Dec 3, 2024 at 1:07 PM Jeff King <peff@peff.net> wrote:
>
> On Tue, Dec 03, 2024 at 12:01:51AM -0800, Elijah Newren wrote:
>
> > > On Sat, Nov 30, 2024 at 01:09:29AM +0000, Elijah Newren via GitGitGadget wrote:
> > >
> > > >     Changes since v1:
> > > >
> > > >      * Moved the check to a higher level, as suggested by Peff.
> > >
> > > Thanks, the code change looks good. Is it worth tweaking one of the
> > > tests to do "R innocent-path .git/evil"? Otherwise I don't think there's
> > > any coverage of the file_change_cr() call at all.
> >
> > I would say yes, but since this patch too has made it to next and is
> > marked for master, I'm kinda tempted to just leave it as-is...
>
> Is is tempting. :) I wrote this up, though, which can just go on top (of
> en/fast-import-verify-path).

Thanks!

> -Peff
>
> -- >8 --
> Subject: [PATCH] t9300: test verification of renamed paths
>
> Commit da91a90c2f (fast-import: disallow more path components,
> 2024-11-30) added two separate verify_path() calls (one for
> added/modified files, and one for renames/copies). But our tests only
> exercise the first one. Let's protect ourselves against regressions by
> tweaking one of the tests to rename into the bad path. There are
> adjacent tests that will stay as additions, so now both calls are
> covered.
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  t/t9300-fast-import.sh | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
> index e2b1db6bc2..fd01a2353c 100755
> --- a/t/t9300-fast-import.sh
> +++ b/t/t9300-fast-import.sh
> @@ -553,9 +553,16 @@ test_expect_success 'B: fail on invalid file path of .' '
>         commit refs/heads/badpath
>         committer Name <email> $GIT_COMMITTER_DATE
>         data <<COMMIT
> -       Commit Message
> +       Good path
> +       COMMIT
> +       M 100644 :1 ok-path
> +
> +       commit refs/heads/badpath
> +       committer Name <email> $GIT_COMMITTER_DATE
> +       data <<COMMIT
> +       Bad path
>         COMMIT
> -       M 100644 :1 ./invalid-path
> +       R ok-path ./invalid-path
>         INPUT_END
>
>         test_when_finished "git update-ref -d refs/heads/badpath" &&
> --
> 2.47.1.707.g92f6f18526

Change looks good to me.

[gitgitgadget]

On the Git mailing list, Junio C Hamano wrote (reply to this):

Jeff King <peff@peff.net> writes:

>> I would say yes, but since this patch too has made it to next and is
>> marked for master, I'm kinda tempted to just leave it as-is...
>
> Is is tempting. :) I wrote this up, though, which can just go on top (of
> en/fast-import-verify-path).

Thanks, queued.

> -- >8 --
> Subject: [PATCH] t9300: test verification of renamed paths
>
> Commit da91a90c2f (fast-import: disallow more path components,
> 2024-11-30) added two separate verify_path() calls (one for
> added/modified files, and one for renames/copies). But our tests only
> exercise the first one. Let's protect ourselves against regressions by
> tweaking one of the tests to rename into the bad path. There are
> adjacent tests that will stay as additions, so now both calls are
> covered.
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  t/t9300-fast-import.sh | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
> index e2b1db6bc2..fd01a2353c 100755
> --- a/t/t9300-fast-import.sh
> +++ b/t/t9300-fast-import.sh
> @@ -553,9 +553,16 @@ test_expect_success 'B: fail on invalid file path of .' '
>  	commit refs/heads/badpath
>  	committer Name <email> $GIT_COMMITTER_DATE
>  	data <<COMMIT
> -	Commit Message
> +	Good path
> +	COMMIT
> +	M 100644 :1 ok-path
> +
> +	commit refs/heads/badpath
> +	committer Name <email> $GIT_COMMITTER_DATE
> +	data <<COMMIT
> +	Bad path
>  	COMMIT
> -	M 100644 :1 ./invalid-path
> +	R ok-path ./invalid-path
>  	INPUT_END
>  
>  	test_when_finished "git update-ref -d refs/heads/badpath" &&

[gitgitgadget]

This patch series was integrated into seen via git@10125ad.