Test views of LFS files

[kousu]

Follow up to #22195: that happened because there was no testing of how the UI
reacts to LFS files.

Until now Gitea has tested LFS via:

• the internal go API
• the external HTTP API
• the external git+http:// protocol
• the external git+ssh:// protocol

The first is small unit tests, and can create LFS structs in memory
as necessary. The latter all run git lfs track * and upload two random
binaries. ("code.gitea.io/gitea/tests/integration".lfsCommitAndPushTest)
In both cases, the LFS data are created by procedural code.

This is different: it adds declaractive LFS fixtures. That means a new
repo in gitea-repositories-meta/, a new folder for LFS objects in
gitea-lfs-meta/, and associated database entries in models/fixtures/.

Using this, it can easily add testing LFS via:

• the internal HTTP UI

Sources

• lfs.git was hand-crafted using git commands, then git pushed into an
  empty bare repo.

Its contents:

• CONTRIBUTING.md (LFS object 7b6b2c88dba9f760a1a58469b67fee2b698ef7e9399c4ca4f34a14ccbe39f623)
  was hand-written.
• subdir/README.md (LFS object 9d172e5c64b4f0024b9901ec6afe9ea052f3c9b6ff9f4b07956d8c48c86fca82)
  was also hand-written.
• jpeg.jpg (LFS object 0b8d8b5f15046343fd32f451df93acc2bdd9e6373be478b968e4cad6b6647351)
  was copied from tests/gitea-repositories-meta/user2/repo1.wiki.git/
• crypt.bin (LFS object 2eccdb43825d2a49d99d542daa20075cff1d97d9d2349a8977efe9c03661737c)
  was generated with dd if=/dev/urandom bs=1k count=2

[KN4CK3R]

I don't think this will work. You put the LFS files in the git repo filesystem lfs.git/lfs/objects but Gitea doesn't use this path. So it will not be possible to resolve the files. You need to upload the LFS files when running the test. There is already test code which does this. Maybe you can simply add your tests there?

[kousu]

Gitea doesn't use this path. So it will not be possible to resolve the files.

Right. I fooled myself by not testing a clean build. Gitea uses data/lfs/ instead of data/gitea-repositories/*/*.git/lfs/objects/. I guess something I did during dev copied in my objects.

You need to upload the LFS files when running the test. There is already test code which does this. Maybe you can simply add your tests there?

Do you have lfsCommitAndPushTest() in mind? Starting from

https://github.com/go-gitea/gitea/blob/bc82f6d82a60ab72f355edba68b6f6b8486be3dd/tests/integration/git_test.go#L147-L150

that test is basically

git clone $URL && cd $(basename $URL) # ~approximately
git lfs install
git lfs track "*"
dd if=/dev/urandom of="$(mktemp -u lfs-data-file-XXXXXXX)" count=1 bs=1K
dd if=/dev/urandom of="$(mktemp -u lfs-data-file-XXXXXXX)" count=128 bs=1M
git config user.name "User Two"
git config user.email "user2@example.com"
git commit -m "Testing commit @ $(date +%s)"
git push origin master

The .bin file I'm submitting could be replaced by the generated binary files, but I feel better when tests are fully deterministic (the existing test changes contents, and therefore LFS OIDs, each time). And I have no idea how I would handle the .md or .jpg files? base64 encode them in the source code?

Fixtures are there to be fixed and make it (relatively) easy to add data. Encoding fixtures as procedural code is harder than declaring them as data.

So I've come up with a different fix, let me know what you think: I set up a new folder of LFS fixtures just like the existing folder of git fixtures, and I load them into the tests the same way:

https://github.com/go-gitea/gitea/blob/50f89f5641e15e774087f274232766f6abddd91d/tests/test_utils.go#L171-L174

(CI is running now; I've successfully make test-sqlite and make test-mysql'd on my local copy so 🤞 it passes in CI too.)

[kousu]

MySQL is failing. Everything else is passing.

This was a head-scratcher until I realized it's not really MySQL's fault; test-mysql is also the only test that uses

gitea/tests/mysql.ini.tmpl

Lines 70 to 71 in b767182

	[storage]
	STORAGE_TYPE = minio

which implies LFS objects are stored on a third-party server. Where my fixtures are not. 😭 This is of course exactly what KN4CK3R was warning me about but I didn't make the connection.

I did run make test-mysql locally, but only by erasing that section of the config file.

Okay, that's okay.

Maybe I can use this the migration code?

gitea/tests/integration/api_repo_lfs_migrate_test.go

Lines 35 to 42 in b767182

	req := NewRequestWithJSON(t, "POST", "/api/v1/repos/migrate?token="+token, &api.MigrateRepoOptions{
	CloneAddr: path.Join(setting.RepoRootPath, "migration/lfs-test.git"),
	RepoOwnerID: user.ID,
	RepoName: "lfs-test-local",
	LFS: true,
	})
	resp := MakeRequest(t, req, NoExpectedStatus)
	assert.EqualValues(t, http.StatusCreated, resp.Code)

but that's more fragile than just directly loading.

Maybe instead of using CopyDir I can use

gitea/modules/storage/storage.go

Line 72 in b767182

func Copy(dstStorage ObjectStorage, dstPath string, srcStorage ObjectStorage, srcPath string) (int64, error) {

but same deal: if storage breaks then all the tests will break.

I'll have to try both but that's going to take a bunch more time. Too bad. I was hoping to tie this off before the holidays but now it seems like it's likely not going to get finished for a month. Oh well. Happy Solstice everyone!

[lunny]

MySQL is failing. Everything else is passing.

We can have a real git lfs push or we can use storage.Copy to copy the file to lfs.

[kousu]

use storage.Copy to copy the file to lfs.

https://github.com/go-gitea/gitea/blob/e79ac5be623d3c822d27c1385321af7a183a3d96/tests/test_utils.go#L197-L205 !

🌵 🌵 🍰

I'm optimistic about this one :)

[lunny]

@kousu please update the branch

[kousu]

It's ready :)

[kousu]

Oh no it's not! drat

[codecov-commenter]

Codecov Report

Merging #22196 (3fa54a8) into main (e81ccc4) will increase coverage by 0.01%.
The diff coverage is 58.72%.

❗ Current head 3fa54a8 differs from pull request most recent head 99d05b3. Consider uploading reports for the commit 99d05b3 to get more accurate results

@@            Coverage Diff             @@
##             main   #22196      +/-   ##
==========================================
+ Coverage   48.15%   48.16%   +0.01%     
==========================================
  Files        1037     1045       +8     
  Lines      141353   142421    +1068     
==========================================
+ Hits        68071    68602     +531     
- Misses      65145    65627     +482     
- Partials     8137     8192      +55     

Impacted Files	Coverage Δ
cmd/admin.go	0.00% <0.00%> (ø)
models/db/context.go	65.17% <ø> (ø)
models/db/engine.go	40.90% <ø> (ø)
models/db/index.go	36.25% <0.00%> (-7.69%)	⬇️
models/issues/issue_lock.go	0.00% <0.00%> (ø)
models/issues/issue_project.go	31.06% <0.00%> (ø)
models/issues/label.go	56.68% <0.00%> (ø)
models/issues/reaction.go	79.91% <0.00%> (ø)
models/org.go	45.33% <0.00%> (ø)
models/packages/package_cleanup_rule.go	41.17% <ø> (ø)
... and 225 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[kousu]

@lunny now it's really ready