Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent double use of git cat-file session. #29298

Merged
merged 2 commits into from
Feb 21, 2024

Conversation

KN4CK3R
Copy link
Member

@KN4CK3R KN4CK3R commented Feb 21, 2024

Fixes the reason why #29101 is hard to replicate.
Related #29297

Create a repo with a file with minimum size 4097 bytes (I use 10000) and execute the following code:

gitRepo, err := gitrepo.OpenRepository(db.DefaultContext, <repo>)
assert.NoError(t, err)

commit, err := gitRepo.GetCommit(<sha>)
assert.NoError(t, err)

entry, err := commit.GetTreeEntryByPath(<file>)
assert.NoError(t, err)

b := entry.Blob()

// Create a reader
r, err := b.DataAsync()
assert.NoError(t, err)
defer r.Close()

// Create a second reader
r2, err := b.DataAsync()
assert.NoError(t, err) // Should be no error but is ErrNotExist
defer r2.Close()

The problem is the check in CatFileBatch:

func (repo *Repository) CatFileBatch(ctx context.Context) (WriteCloserError, *bufio.Reader, func()) {
if repo.batchCancel == nil || repo.batchReader.Buffered() > 0 {
log.Debug("Opening temporary cat file batch for: %s", repo.Path)
return CatFileBatch(ctx, repo.Path)
}
return repo.batchWriter, repo.batchReader, func() {}
}

Buffered() > 0 is used to check if there is a "operation" in progress at the moment. This is a problem because we can't control the internal buffer in the bufio.Reader. The code above demonstrates a sequence which initiates an operation for which the code thinks there is no active processing. The second call to DataAsync() therefore reuses the existing instances instead of creating a new batch reader.

@KN4CK3R KN4CK3R added type/bug backport/v1.21 This PR should be backported to Gitea 1.21 labels Feb 21, 2024
@KN4CK3R KN4CK3R added this to the 1.22.0 milestone Feb 21, 2024
@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Feb 21, 2024
@pull-request-size pull-request-size bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 21, 2024
@@ -27,10 +27,12 @@ type Repository struct {

gpgSettings *GPGSettings

batchInUse bool
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No atomic because this is single thread code.

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Feb 21, 2024
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Feb 21, 2024
@KN4CK3R KN4CK3R merged commit f74c869 into go-gitea:main Feb 21, 2024
26 checks passed
KN4CK3R added a commit to KN4CK3R/gitea that referenced this pull request Feb 21, 2024
Fixes the reason why go-gitea#29101 is hard to replicate.
Related go-gitea#29297

Create a repo with a file with minimum size 4097 bytes (I use 10000) and
execute the following code:
```go
gitRepo, err := gitrepo.OpenRepository(db.DefaultContext, <repo>)
assert.NoError(t, err)

commit, err := gitRepo.GetCommit(<sha>)
assert.NoError(t, err)

entry, err := commit.GetTreeEntryByPath(<file>)
assert.NoError(t, err)

b := entry.Blob()

// Create a reader
r, err := b.DataAsync()
assert.NoError(t, err)
defer r.Close()

// Create a second reader
r2, err := b.DataAsync()
assert.NoError(t, err) // Should be no error but is ErrNotExist
defer r2.Close()
```

The problem is the check in `CatFileBatch`:

https://github.com/go-gitea/gitea/blob/79217ea63c1f77de7ca79813ae45950724e63d02/modules/git/repo_base_nogogit.go#L81-L87
`Buffered() > 0` is used to check if there is a "operation" in progress
at the moment. This is a problem because we can't control the internal
buffer in the `bufio.Reader`. The code above demonstrates a sequence
which initiates an operation for which the code thinks there is no
active processing. The second call to `DataAsync()` therefore reuses the
existing instances instead of creating a new batch reader.
@KN4CK3R KN4CK3R deleted the fix-catfile-unreliable branch February 21, 2024 19:48
zjjhot added a commit to zjjhot/gitea that referenced this pull request Feb 22, 2024
* giteaofficial/main: (32 commits)
  [skip ci] Updated translations via Crowdin
  Prevent double use of `git cat-file` session. (go-gitea#29298)
  Revert go-gitea#28753 because UI broken. (go-gitea#29293)
  Fix error display when merging PRs (go-gitea#29288)
  Refactor markup rendering to accept general "protocol:" prefix (go-gitea#29276)
  Remove jQuery from the installation page (go-gitea#29284)
  Always write proc-receive hook for all git versions (go-gitea#29287)
  Do not use `ctx.Doer` when reset password (go-gitea#29289)
  Update Discord logo (go-gitea#29285)
  [skip ci] Updated translations via Crowdin
  Remove jQuery .map() and enable eslint rules for it (go-gitea#29272)
  Explained where create issue/PR template (go-gitea#29035) (go-gitea#29266)
  Remove jQuery from repo wiki creation page (go-gitea#29271)
  Do not show delete button when time tracker is disabled (go-gitea#29257)
  Left align the input labels for the link account page (go-gitea#29255)
  [skip ci] Updated translations via Crowdin
  Remove jQuery from the repo migration form (go-gitea#29229)
  Fix content size does not match error when uploading lfs file (go-gitea#29259)
  Workaround to clean up old reviews on creating a new one (go-gitea#28554)
  Deduplicate translations for contributors graph (go-gitea#29256)
  ...
silverwind pushed a commit that referenced this pull request Feb 22, 2024
Backport #29298
Fixes the reason why #29101 is hard to replicate.
Related #29297

Create a repo with a file with minimum size 4097 bytes (I use 10000) and
execute the following code:
```go
gitRepo, err := gitrepo.OpenRepository(db.DefaultContext, <repo>)
assert.NoError(t, err)

commit, err := gitRepo.GetCommit(<sha>)
assert.NoError(t, err)

entry, err := commit.GetTreeEntryByPath(<file>)
assert.NoError(t, err)

b := entry.Blob()

// Create a reader
r, err := b.DataAsync()
assert.NoError(t, err)
defer r.Close()

// Create a second reader
r2, err := b.DataAsync()
assert.NoError(t, err) // Should be no error but is ErrNotExist
defer r2.Close()
```

The problem is the check in `CatFileBatch`:


https://github.com/go-gitea/gitea/blob/79217ea63c1f77de7ca79813ae45950724e63d02/modules/git/repo_base_nogogit.go#L81-L87
`Buffered() > 0` is used to check if there is a "operation" in progress
at the moment. This is a problem because we can't control the internal
buffer in the `bufio.Reader`. The code above demonstrates a sequence
which initiates an operation for which the code thinks there is no
active processing. The second call to `DataAsync()` therefore reuses the
existing instances instead of creating a new batch reader.
lunny pushed a commit that referenced this pull request Feb 22, 2024
Fixes #29101
Related #29298

Discard all read data to prevent misinterpreting existing data. Some
discard calls were missing in error cases.

---------

Co-authored-by: yp05327 <576951401@qq.com>
@GiteaBot
Copy link
Contributor

I was unable to create a backport for 1.21. @KN4CK3R, please send one manually. 🍵

go run ./contrib/backport 29298
...  // fix git conflicts if any
go run ./contrib/backport --continue

@GiteaBot GiteaBot added the backport/manual No power to the bots! Create your backport yourself! label Feb 22, 2024
GiteaBot pushed a commit to GiteaBot/gitea that referenced this pull request Feb 22, 2024
Fixes go-gitea#29101
Related go-gitea#29298

Discard all read data to prevent misinterpreting existing data. Some
discard calls were missing in error cases.

---------

Co-authored-by: yp05327 <576951401@qq.com>
lunny pushed a commit that referenced this pull request Feb 22, 2024
Backport #29297 by @KN4CK3R

Fixes #29101
Related #29298

Discard all read data to prevent misinterpreting existing data. Some
discard calls were missing in error cases.

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: yp05327 <576951401@qq.com>
@lunny lunny added the backport/done All backports for this PR have been created label Feb 22, 2024
Copy link

github-actions bot commented Mar 1, 2024

Automatically locked because of our CONTRIBUTING guidelines

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport/done All backports for this PR have been created backport/manual No power to the bots! Create your backport yourself! backport/v1.21 This PR should be backported to Gitea 1.21 lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. type/bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants