git-annex support

[git-annex](https://git-annex.branchable.com/) is a more complicated cousin to
git-lfs, storing large files in an optional-download side content.  Unlike lfs,
it allows mixing and matching storage remotes, so the content remote(s) doesn't
need to be on the same server as the git remote, making it feasible to scatter
a collection across cloud storage, old harddrives, or anywhere else storage can
be scavenged.  Since this can get complicated, fast, it has a content-tracking
database (`git annex whereis`) to help find everything later.

The use-case we imagine for including it in Gitea is just the simple case, where
we're primarily emulating git-lfs: each repo has its large content at the same URL.

Our motivation is so we can self-host https://www.datalad.org/ datasets, which
currently are only hostable by fragilely scrounging together cloud storage --
and having to manage all the credentials associated with all the pieces -- or at
https://openneuro.org which is fragile in its own ways.

Supporting git-annex also allows multiple Gitea instance to be annex remotes for
each other, mirroring the content or otherwise collaborating the split up the
hosting costs.

Enabling
--------

TODO

HTTP
----

TODO

Permission Checking
-------------------

This tweaks the API in routers/private/serv.go to expose the calling user's
computed permission, instead of just returning HTTP 403.

This doesn't fit in super well. It's the opposite from how the git-lfs support is
done, where there's a complete list of possible subcommands and their matching
permission levels, and then the API compares the requested with the actual level
and returns HTTP 403 if the check fails.

But it's necessary. The main git-annex verbs, 'git-annex-shell configlist' and
'git-annex-shell p2pstdio' are both either read-only or read-write operations,
depending on the state on disk on either end of the connection and what the user
asked it to ask for, with no way to know before git-annex examines the situation.
So tell the level via GIT_ANNEX_READONLY and trust it to handle itself.

In the older Gogs version, the permission was directly read in cmd/serv.go:

```
mode, err = db.UserAccessMode(user.ID, repo)
```
- https://github.com/G-Node/gogs/blob/966e925cf320beff768b192276774d9265706df5/internal/cmd/serv.go#L334

but in Gitea permission enforcement has been centralized in the API layer.
(perhaps so the cmd layer can avoid making direct DB connections?)

Deletion
--------

git-annex has this "lockdown" feature where it tries
really quite very hard to prevent you deleting its
data, to the point that even an rm -rf won't do it:
each file in annex/objects/ is nested inside a
folder with read-only permissions.

The recommended workaround is to run chmod -R +w when
you're sure you actually want to delete a repo. See
https://git-annex.branchable.com/internals/lockdown

So we edit util.RemoveAll() to do just that, so now
it's `chmod -R +w && rm -rf` instead of just `rm -rf`.

cmd/serv.go

@@ -37,6 +37,7 @@ import (
 
 const (
 	lfsAuthenticateVerb = "git-lfs-authenticate"
+	gitAnnexShellVerb   = "git-annex-shell"
 )
 
 // CmdServ represents the available serv sub-command.
@@ -89,6 +90,7 @@ var (
 		"git-upload-archive": perm.AccessModeRead,
 		"git-receive-pack":   perm.AccessModeWrite,
 		lfsAuthenticateVerb:  perm.AccessModeNone,
+		gitAnnexShellVerb:    perm.AccessModeNone, // annex permissions are enforced by GIT_ANNEX_SHELL_READONLY, rather than the Gitea API
 	}
 	alphaDashDotPattern = regexp.MustCompile(`[^\w-\.]`)
 )
@@ -201,6 +203,7 @@ func runServ(c *cli.Context) error {
 
 	verb := words[0]
 	repoPath := words[1]
+
 	if repoPath[0] == '/' {
 		repoPath = repoPath[1:]
 	}
@@ -216,9 +219,44 @@ func runServ(c *cli.Context) error {
 		}
 	}
 
+	if verb == gitAnnexShellVerb {
+		// if !setting.Annex.Enabled { // TODO: https://github.com/neuropoly/gitea/issues/8
+		if false {
+			return fail(ctx, "Unknown git command", "git-annex request over SSH denied, git-annex support is disabled")
+		}
+
+		if len(words) < 3 {
+			return fail(ctx, "Too few arguments", "Too few arguments in cmd: %s", cmd)
+		}
+
+		// git-annex always puts the repo in words[2], unlike most other
+		// git subcommands; and it sometimes names repos like /~/, as if
+		// $HOME should get expanded while also being rooted. e.g.:
+		//   git-annex-shell 'configlist' '/~/user/repo'
+		//   git-annex-shell 'sendkey' '/user/repo 'key'
+		repoPath = words[2]
+		repoPath = strings.TrimPrefix(repoPath, "/")
+		repoPath = strings.TrimPrefix(repoPath, "~/")
+	}
+
 	// LowerCase and trim the repoPath as that's how they are stored.
 	repoPath = strings.ToLower(strings.TrimSpace(repoPath))
 
+	// prevent directory traversal attacks
+	repoPath = filepath.Clean("/" + repoPath)[1:]
+
+	// put the sanitized repoPath back into the argument list for later
+	if verb == gitAnnexShellVerb {
+		// git-annex-shell demands an absolute path
+		absRepoPath, err := filepath.Abs(filepath.Join(setting.RepoRootPath, repoPath))
+		if err != nil {
+			return fail(ctx, "Error locating repoPath", "%v", err)
+		}
+		words[2] = absRepoPath
+	} else {
+		words[1] = repoPath
+	}
+
 	rr := strings.SplitN(repoPath, "/", 2)
 	if len(rr) != 2 {
 		return fail(ctx, "Invalid repository path", "Invalid repository path: %v", repoPath)
@@ -305,21 +343,46 @@ func runServ(c *cli.Context) error {
 		return nil
 	}
 
-	var gitcmd *exec.Cmd
 	gitBinPath := filepath.Dir(git.GitExecutable) // e.g. /usr/bin
 	gitBinVerb := filepath.Join(gitBinPath, verb) // e.g. /usr/bin/git-upload-pack
 	if _, err := os.Stat(gitBinVerb); err != nil {
 		// if the command "git-upload-pack" doesn't exist, try to split "git-upload-pack" to use the sub-command with git
 		// ps: Windows only has "git.exe" in the bin path, so Windows always uses this way
+		// ps: git-annex-shell and other extensions may not necessarily be in gitBinPath,
+		//     but '{gitBinPath}/git annex-shell' should be able to find them on $PATH.
 		verbFields := strings.SplitN(verb, "-", 2)
 		if len(verbFields) == 2 {
 			// use git binary with the sub-command part: "C:\...\bin\git.exe", "upload-pack", ...
-			gitcmd = exec.CommandContext(ctx, git.GitExecutable, verbFields[1], repoPath)
+			gitBinVerb = git.GitExecutable
+			words = append([]string{verbFields[1]}, words...)
 		}
 	}
-	if gitcmd == nil {
-		// by default, use the verb (it has been checked above by allowedCommands)
-		gitcmd = exec.CommandContext(ctx, gitBinVerb, repoPath)
+
+	// by default, use the verb (it has been checked above by allowedCommands)
+	gitcmd := exec.CommandContext(ctx, gitBinVerb, words[1:]...)
+
+	if verb == gitAnnexShellVerb {
+		// This doesn't get its own isolated section like LFS does, because LFS
+		// is handled by internal Gitea routines, but git-annex has to be shelled out
+		// to like other git subcommands, so we need to build up gitcmd.
+
+		// TODO: does this work on Windows?
+		gitcmd.Env = append(gitcmd.Env,
+			// "If set, disallows running git-shell to handle unknown commands."
+			// - git-annex-shell(1)
+			"GIT_ANNEX_SHELL_LIMITED=True",
+			// "If set, git-annex-shell will refuse to run commands
+			//  that do not operate on the specified directory."
+			// - git-annex-shell(1)
+			fmt.Sprintf("GIT_ANNEX_SHELL_DIRECTORY=%s", words[2]),
+		)
+		if results.UserMode < perm.AccessModeWrite {
+			// "If set, disallows any action that could modify the git-annex repository."
+			// - git-annex-shell(1)
+			// We set this when the backend API has told us that we don't have write permission to this repo.
+			log.Debug("Setting GIT_ANNEX_SHELL_READONLY=True")
+			gitcmd.Env = append(gitcmd.Env, "GIT_ANNEX_SHELL_READONLY=True")
+		}
 	}
 
 	process.SetSysProcAttribute(gitcmd)

modules/private/serv.go

@@ -40,6 +40,7 @@ type ServCommandResults struct {
 	UserName    string
 	UserEmail   string
 	UserID      int64
+	UserMode    perm.AccessMode
 	OwnerName   string
 	RepoName    string
 	RepoID      int64

modules/util/remove.go

@@ -4,7 +4,9 @@
 package util
 
 import (
+	"io/fs"
 	"os"
+	"path/filepath"
 	"runtime"
 	"syscall"
 	"time"
@@ -41,10 +43,40 @@ func Remove(name string) error {
 	return err
 }
 
-// RemoveAll removes the named file or (empty) directory with at most 5 attempts.
+// RemoveAll removes the named file or directory with at most 5 attempts.
 func RemoveAll(name string) error {
 	var err error
+
 	for i := 0; i < 5; i++ {
+		// Do chmod -R +w to help ensure the removal succeeds.
+		// In particular, in the git-annex case, this handles
+		// https://git-annex.branchable.com/internals/lockdown/ :
+		//
+		// > (The only bad consequence of this is that rm -rf .git
+		// > doesn't work unless you first run chmod -R +w .git)
+
+		err = filepath.WalkDir(name, func(path string, d fs.DirEntry, err error) error {
+			// NB: this is called WalkDir but it works on a single file too
+			if err == nil {
+				info, err := d.Info()
+				if err != nil {
+					return err
+				}
+
+				// 0200 == u+w, in octal unix permission notation
+				err = os.Chmod(path, info.Mode()|0o200)
+				if err != nil {
+					return err
+				}
+			}
+			return nil
+		})
+		if err != nil {
+			// try again
+			<-time.After(100 * time.Millisecond)
+			continue
+		}
+
 		err = os.RemoveAll(name)
 		if err == nil {
 			break

routers/private/serv.go

@@ -81,12 +81,14 @@ func ServCommand(ctx *context.PrivateContext) {
 	ownerName := ctx.Params(":owner")
 	repoName := ctx.Params(":repo")
 	mode := perm.AccessMode(ctx.FormInt("mode"))
+	verbs := ctx.FormStrings("verb")
 
 	// Set the basic parts of the results to return
 	results := private.ServCommandResults{
 		RepoName:  repoName,
 		OwnerName: ownerName,
 		KeyID:     keyID,
+		UserMode:  perm.AccessModeNone,
 	}
 
 	// Now because we're not translating things properly let's just default some English strings here
@@ -287,8 +289,10 @@ func ServCommand(ctx *context.PrivateContext) {
 			repo.IsPrivate ||
 			owner.Visibility.IsPrivate() ||
 			(user != nil && user.IsRestricted) || // user will be nil if the key is a deploykey
+			( /*setting.Annex.Enabled && */ len(verbs) > 0 && verbs[0] == "git-annex-shell") || // git-annex has its own permission enforcement, for which we expose results.UserMode
 			setting.Service.RequireSignInView) {
 		if key.Type == asymkey_model.KeyTypeDeploy {
+			results.UserMode = deployKey.Mode
 			if deployKey.Mode < mode {
 				ctx.JSON(http.StatusUnauthorized, private.Response{
 					UserMsg: fmt.Sprintf("Deploy Key: %d:%s is not authorized to %s %s/%s.", key.ID, key.Name, modeString, results.OwnerName, results.RepoName),
@@ -310,9 +314,9 @@ func ServCommand(ctx *context.PrivateContext) {
 				return
 			}
 
-			userMode := perm.UnitAccessMode(unitType)
+			results.UserMode = perm.UnitAccessMode(unitType)
 
-			if userMode < mode {
+			if results.UserMode < mode {
 				log.Warn("Failed authentication attempt for %s with key %s (not authorized to %s %s/%s) from %s", user.Name, key.Name, modeString, ownerName, repoName, ctx.RemoteAddr())
 				ctx.JSON(http.StatusUnauthorized, private.Response{
 					UserMsg: fmt.Sprintf("User: %d:%s with Key: %d:%s is not authorized to %s %s/%s.", user.ID, user.Name, key.ID, key.Name, modeString, ownerName, repoName),
@@ -353,6 +357,7 @@ func ServCommand(ctx *context.PrivateContext) {
 			})
 			return
 		}
+		results.UserMode = perm.AccessModeWrite
 		results.RepoID = repo.ID
 	}
 
@@ -381,13 +386,14 @@ func ServCommand(ctx *context.PrivateContext) {
 			return
 		}
 	}
-	log.Debug("Serv Results:\nIsWiki: %t\nDeployKeyID: %d\nKeyID: %d\tKeyName: %s\nUserName: %s\nUserID: %d\nOwnerName: %s\nRepoName: %s\nRepoID: %d",
+	log.Debug("Serv Results:\nIsWiki: %t\nDeployKeyID: %d\nKeyID: %d\tKeyName: %s\nUserName: %s\nUserID: %d\nUserMode: %d\nOwnerName: %s\nRepoName: %s\nRepoID: %d",
 		results.IsWiki,
 		results.DeployKeyID,
 		results.KeyID,
 		results.KeyName,
 		results.UserName,
 		results.UserID,
+		results.UserMode,
 		results.OwnerName,
 		results.RepoName,
 		results.RepoID)