Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LFS: unexpected EOF despite good integrity of the file on the server #5182

Closed
2 of 7 tasks
qzmfranklin opened this issue Oct 25, 2018 · 10 comments · Fixed by #5722
Closed
2 of 7 tasks

LFS: unexpected EOF despite good integrity of the file on the server #5182

qzmfranklin opened this issue Oct 25, 2018 · 10 comments · Fixed by #5722
Labels
issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented type/bug
Milestone
1.8.0

Comments

@qzmfranklin
Copy link

  • Gitea version (or commit ref):
    git-lfs/2.5.2
  • Git version:
    git version 2.17.1
  • Operating system:
    Ubuntu 18.04.1
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
    • Not relevant
  • Log gist:
git-lfs/2.5.2 (GitHub; linux amd64; go 1.10.3; git 8e3c5c93)
git version 2.17.1

$ git-lfs filter-process
Error downloading object: it/docs/jenkins/step1.jpg (c5e6043): Smudge error: Error downloading it/docs/jenkins/step1.jpg (c5e60437c9e7596445342c5adc91f6535e7565fab96ff200112ed8b2aabdbf47): cannot write data to tempfile "/home/zhongming/git/logi3/.git/lfs/incomplete/c5e60437c9e7596445342c5adc91f6535e7565fab96ff200112ed8b2aabdbf47.tmp": LFS: unexpected EOF

unexpected EOF
LFS
github.com/git-lfs/git-lfs/errors.newWrappedError
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/errors/types.go:188
github.com/git-lfs/git-lfs/errors.NewRetriableError
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/errors/types.go:363
github.com/git-lfs/git-lfs/tools.(*RetriableReader).Read
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/tools/iotools.go:99
github.com/git-lfs/git-lfs/tools.(*HashingReader).Read
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/tools/iotools.go:66
github.com/git-lfs/git-lfs/tools.(*CallbackReader).Read
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/tools/copycallback.go:73
io.copyBuffer
        /usr/local/Cellar/go/1.10.3/libexec/src/io/io.go:400
io.Copy
        /usr/local/Cellar/go/1.10.3/libexec/src/io/io.go:362
github.com/git-lfs/git-lfs/tools.CopyWithCallback
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/tools/iotools.go:39
github.com/git-lfs/git-lfs/tq.(*basicDownloadAdapter).download
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/tq/basic_download.go:205
github.com/git-lfs/git-lfs/tq.(*basicDownloadAdapter).DoTransfer
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/tq/basic_download.go:48
github.com/git-lfs/git-lfs/tq.(*adapterBase).worker
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/tq/adapterbase.go:176
runtime.goexit
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/asm_amd64.s:2361
cannot write data to tempfile "/home/zhongming/git/logi3/.git/lfs/incomplete/c5e60437c9e7596445342c5adc91f6535e7565fab96ff200112ed8b2aabdbf47.tmp"
github.com/git-lfs/git-lfs/errors.newWrappedError
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/errors/types.go:184
github.com/git-lfs/git-lfs/errors.Wrapf
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/errors/errors.go:85
github.com/git-lfs/git-lfs/tq.(*basicDownloadAdapter).download
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/tq/basic_download.go:207
github.com/git-lfs/git-lfs/tq.(*basicDownloadAdapter).DoTransfer
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/tq/basic_download.go:48
github.com/git-lfs/git-lfs/tq.(*adapterBase).worker
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/tq/adapterbase.go:176
runtime.goexit
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/asm_amd64.s:2361
Error downloading it/docs/jenkins/step1.jpg (c5e60437c9e7596445342c5adc91f6535e7565fab96ff200112ed8b2aabdbf47)
github.com/git-lfs/git-lfs/errors.newWrappedError
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/errors/types.go:184
github.com/git-lfs/git-lfs/errors.Wrapf
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/errors/errors.go:85
github.com/git-lfs/git-lfs/lfs.(*GitFilter).downloadFile
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:110
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:71
github.com/git-lfs/git-lfs/commands.smudge
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.filterCommand
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:118
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551
github.com/git-lfs/git-lfs/commands.Run
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/commands/run.go:77
main.main
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/git-lfs.go:35
runtime.main
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/proc.go:198
runtime.goexit
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/asm_amd64.s:2361
Smudge error
github.com/git-lfs/git-lfs/errors.newWrappedError
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/errors/types.go:184
github.com/git-lfs/git-lfs/errors.NewSmudgeError
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/errors/types.go:270
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:80
github.com/git-lfs/git-lfs/commands.smudge
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.filterCommand
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:118
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:477
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:551
github.com/git-lfs/git-lfs/commands.Run
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/commands/run.go:77
main.main
        /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/git-lfs.go:35
runtime.main
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/proc.go:198
runtime.goexit
        /usr/local/Cellar/go/1.10.3/libexec/src/runtime/asm_amd64.s:2361

Current time in UTC: 
2018-10-25 14:11:01

ENV:
LocalWorkingDir=/home/zhongming/git/logi3
LocalGitDir=/home/zhongming/git/logi3/.git
LocalGitStorageDir=/home/zhongming/git/logi3/.git
LocalMediaDir=/home/zhongming/git/logi3/.git/lfs/objects
LocalReferenceDirs=
TempDir=/home/zhongming/git/logi3/.git/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/zhongming/git/logi3/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_DIR=.git
GIT_PROMPT_ONLY_IN_REPO=1
GIT_BRANCH=master
__GIT_PROMPT_IGNORE_SUBMODULES__=1
__GIT_PROMPT_SHOW_CHANGED_FILES_COUNT=1
__GIT_PROMPT_SHOW_UPSTREAM=
__GIT_PROMPT_IGNORE_SUBMODULES=
__GIT_PROMPT_SHOW_UNTRACKED_FILES=normal
__GIT_PROMPT_IGNORE_STASH=

Description

I have a local gitea server set up at host http://git.corp.logiocean.com:3000.

I pushed an LFS-enabled repo, named logi to the gitea server. Everything worked fine until I tried git clone, which produced the above described error.

I pushed the same repo (at the same commit) to github and git clone from github, everything worked.

I have verified that the git commit hash on github and on my local gitea server are exactly the same.

I have also verified that the LFS file that failed to download are intact on the server. Here is how I verified it:

  • ssh into the server
  • go to the lfs data directory /home/git/data/lfs.
  • use the oid of the lfs file to locate the file.
  • use scp to copy the file in question to my own desktop and verified (using diff and other tools) that the size of the file and the content of the file are indeed intact.

I have searched for similar issues across this repo and in many other places. Have not found one with exactly the same syndrome.

Posting to hopefully help others who are having the same problem.

Any help would be much appreciated! Thanks in advance.
@zeripath
Copy link
Contributor

hmm... looking at your logs I see: /Users/ttaylorr/go/src/github.com/git-lfs/git-lfs/errors/types.go:188. Are you running Gitea on a Mac?

I wonder if the file that's being downloaded to be smudged by git-lfs has the wrong line-endings?

git lfs install --skip-smudge

should allow you to skip the smudge action and interrogate the smudge files to check how they are formatted - although they might actually already be in your checkout for you to take a look at.

@qzmfranklin
Copy link
Author

Gitea was running on a Ubuntu 18.04.1 server.

This issue happens for only a few files. I tried --skip-smudge and verified the content of the lfs object file. The file has exactly the same content as it should (verified by xxd and diff).

To get around this issue, I have removed those faulting files. Running out of ideas to try.

@zeripath
Copy link
Contributor

Are the faulting files proprietary? Would it be possible to post them so I could try them. Things that come to mind are:

  • Is there a buffer overrun/underrun problem? I.e. a file with the exact same length should also fail. These problems can be hard to find - but can be exploitable (!)
  • Is there a problem with something replacing part of the file. I.e. is something filtering out say the null bytes, or is the stream stopping too early due to null bytes.

The problem could be anywhere, but it's unlikely to be on your client end if you say it works for other providers. However it might be good to try a different client machine to see if it's restricted to a single client.

@stale
Copy link

stale bot commented Jan 5, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

@stale stale bot added the issue/stale label Jan 5, 2019
@yasuokav
Copy link
Contributor

yasuokav commented Jan 7, 2019 •
edited
Loading

I have the exact same issue. This is related to gzip.
When I disable ENABLE_GZIP in the config and the issue will be solved.
But when I enable the ENABLE_GZIP option, I won't be able to download some files via git-lfs or web (ERR_CONTENT_LENGTH_MISMATCH)

Similar issue: #4853

@stale stale bot removed the issue/stale label Jan 7, 2019
@lafriks lafriks added type/bug issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented labels Jan 7, 2019
@qzmfranklin
Copy link
Author

@yasuokav : Thanks for confirming and providing a workaround. This is infinitely helpful to us.

@zeripath
Copy link
Contributor

zeripath commented Jan 8, 2019

An example file would really help in debugging this. If the problem files are all the same length that would also help identify this.

I am concerned that this implies there is either a buffer overflow or underrun here - those tend to be exploitable, so it would be helpful to get this identified. I guess we need to look at where gzip is enabled and see if there's an problem there.

@yasuokav
Copy link
Contributor

yasuokav commented Jan 8, 2019

@zeripath
1.zip
SHA1(1.png)= b9a88e4d43568845689750acbec93de346dbc8a4
Size: 39,683 bytes
Maybe Gitea sent the content-length of original file size but it's a gzipped response. Is this the problem?

Gitea(Gzip enabled):

Content-Encoding: gzip
Content-Length: 39683

Apache(Gzip enabled):

Content-Encoding: gzip
Content-Length: 39459

@zeripath
Copy link
Contributor

zeripath commented Jan 8, 2019 •
edited
Loading

Ah ok. That makes things interesting.

It appears that gziper.Gziper is supposed to reset the content-length.

I wonder if the lfs.getContentHandler is somehow overriding this.

@zeripath
Copy link
Contributor

zeripath commented Jan 8, 2019

Certainly the range header code will not work in a gzip environment so even if this isn't relevant to the bug you're noticing it's definitely something.

@zeripath zeripath mentioned this issue Jan 13, 2019
Merged
@lafriks lafriks added this to the 1.8.0 milestone Jan 23, 2019
@go-gitea go-gitea locked and limited conversation to collaborators Nov 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented type/bug
Projects
None yet
Milestone
1.8.0
Development

Successfully merging a pull request may close this issue.

Pooled and buffered gzip implementation zeripath/gitea
4 participants
@lafriks @zeripath @yasuokav @qzmfranklin