Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repo mirroring doesn't mirror LFS data. #849

Closed
2 of 6 tasks
damomurf opened this issue Feb 6, 2017 · 16 comments · Fixed by #14726
Closed
2 of 6 tasks

Repo mirroring doesn't mirror LFS data. #849

damomurf opened this issue Feb 6, 2017 · 16 comments · Fixed by #14726
Labels
issue/bounty This issue has a bounty associated. Whoever opens a PR and gets it merged can claim the bounty. issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented type/feature Completely new functionality. Can only be merged if feature freeze is not active.
Milestone

Comments

@damomurf
Copy link

damomurf commented Feb 6, 2017

  • Gitea version (or commit ref): 1.0.0+220-g027591a3
  • Git version: 2.11.0
  • Operating system: Ubuntu 16.04 (Docker)
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
    • Not relevant
  • Log gist:

Bountysource

Description

In a clean install of Gitea I have attempted to mirror a Bitbucket server repository that I know has LFS enabled and stores LFS data. The LFS data appears to not being mirrored in Gitea. (The configured LFS data directory remains empty, even though the repository mirroring appears to be successful).

@andreynering andreynering added this to the 1.1.0 milestone Feb 6, 2017
@lunny
Copy link
Member

lunny commented Feb 15, 2017

This is the same reason with #732. Let's close this and discuss there.

@lunny lunny closed this as completed Feb 15, 2017
@lunny lunny removed this from the 1.1.0 milestone Feb 15, 2017
@fabian-z
Copy link
Contributor

I'm afraid I have to disagree: #732 is not related. Gitea cannot currently retrieve LFS objects from another server. This would basically require adding a complete LFS client on top of the current implementation.

@damomurf: Please see GitHub documentation regarding repository mirroring with LFS

@lunny
Copy link
Member

lunny commented Feb 24, 2017

Seems you are right.

@lunny lunny reopened this Feb 24, 2017
@lunny lunny added this to the 1.2.0 milestone Feb 24, 2017
@fabian-z
Copy link
Contributor

I'd suggest to tag this as a feature request. Current behaviour is basically expected imho and supporting LFS mirrors will be challenging in some regards:

  • Need LFS client implementation
    • CLI client not developed for interactive use
    • Reusing source would require rewriting to some extent
  • Using CLI client
    • Would use at least double the disk space for the LFS objects during mirror
    • Continuous mirror pulling would likely need double the disk space all the time
  • The operation will need to scan all files in every commit for LFS meta files, potentially taking very long for large repositories

Just out of interest, does somebody know whether GitHub support automated LFS object mirroring?

For future reference:
GitLab issue, git-lfs issue

@lunny lunny added type/feature Completely new functionality. Can only be merged if feature freeze is not active. and removed type/bug labels Feb 24, 2017
@lunny lunny modified the milestones: 1.x.x, 1.2.0 Feb 24, 2017
@lunny
Copy link
Member

lunny commented Feb 24, 2017

Maybe put a notice on migrating UI at first.

@fabian-z
Copy link
Contributor

I'll look into adding a note to the UI when LFS support is enabled.

@boskiv
Copy link

boskiv commented Mar 27, 2018

What is steps for mirroring LFS now? Is there some hook to make LFS pull with cmd ?
Or may be I need to setup some cron task ?

@stale
Copy link

stale bot commented Feb 4, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

@stale stale bot added the issue/stale label Feb 4, 2019
@lunny lunny added issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented and removed issue/stale labels Feb 6, 2019
@fabian-z
Copy link
Contributor

@boskiv
You will currently have to manually mirror the repository if you wish to transfer the LFS objects, too.
It may or may not be (easily) possible to setup a custom mirror using cronjobs and shell scripts, but you would need to plan for double the disk space usage (permanently) and possibly use a separate local user account.

Please also see the GitHub docs for manual steps.

@mloskot
Copy link

mloskot commented Apr 26, 2019

BTW, the GitHub docs is what currently Git LFS recommends as per git-lfs/git-lfs#2342 (comment)


@fabian-z Forgive me if I'm hijacking the issue, but could you explain your last comment in more details, in the context explained below?

I have a GitHub repo with files in Git LFS.
I'm looking for solution to have Gitea serve mirror that repo (just Git repo).

TL;TR: Is it possible?

Would something along these lines work?

  1. Mirror GitHub repo with LFS

    git clone --bare <GITHUB-URL> <GITHUB-DIR>
    git lfs fetch origin --all
    git push --mirror <GITEA-URL>
    git lfs push --all <GITEA-URL>
    
  2. Attach Gitea to serve the files pushed to <GITEA-URL>

  3. Frequently run

    cd <GITHUB-DIR>
    git fetch origin --all
    git lfs fetch origin --all
    git push --all <GITEA-URL>
    git lfs push --all <GITEA-URL>
    

@GlassedSilver
Copy link

Is there anyone kinda sorta working on it? Or is there anyone who figured out a way how to do this manually/scripted externally for the time being?

I'm using Gitea to keep backups of software I use or just find interesting, just in case... Obviously having the LFS objects as well is often very handy, many times necessary.

@zeripath
Copy link
Contributor

No one is working on this. A bounty on bountysource might spur someone to look at it.

From a quick glance we would need to implement an LFS client - possibly through vendoring git-lfs/git-lfs. There would need to be significant changes to the mirroring UI too.

@GlassedSilver
Copy link

@GlassedSilver
Copy link

Is it rude for me to ask for a maintainer of the repo to add the bounty tag and maybe declare the bounty value in the title?

Cheers and overall many many thanks for this lovely project to everyone who is working on it. It's of great use to me. :)

@techknowlogick techknowlogick added the issue/bounty This issue has a bounty associated. Whoever opens a PR and gets it merged can claim the bounty. label Jun 2, 2020
@techknowlogick
Copy link
Member

@GlassedSilver thanks for your support :) I've just applied the bounty label to make this easier to find.

@6543 6543 mentioned this issue Sep 3, 2020
24 tasks
@kdumontnu
Copy link
Contributor

kdumontnu commented Jan 26, 2021

I added to this bounty. Bountysource

Feel free to add the following code to the Issue:

[![Bountysource](https://api.bountysource.com/badge/issue?issue_id=41700547)](https://www.bountysource.com/issues/41700547-repo-mirroring-doesn-t-mirror-lfs-data)

@lunny lunny modified the milestones: 1.x.x, 1.15.0 Apr 16, 2021
@go-gitea go-gitea locked and limited conversation to collaborators May 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
issue/bounty This issue has a bounty associated. Whoever opens a PR and gets it merged can claim the bounty. issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented type/feature Completely new functionality. Can only be merged if feature freeze is not active.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants