Custom Backends?

[tsuraan]

Is there any plan to add custom backends into pygit2? What Rugged has (Rugged::Backend) looks really nice; it would be great to be able to do that in pygit2, but the entire pygit2 Repository API seems to depend on everything working on a local filesystem.

[jdavid]

Hello,

I don't have plans to implement it any time soon. Right now I am busy
enough just to review the pull-requests submitted by contributors. Next
in my priority list is to work on the documentation.

Please open an issue about this feature, eventually someone will implement it.
Oh! Replied by email without checking, this is already an issue!

[cholin]

As you might have noticed, Rugged has removed their implementation for custom backends. It seems to be super slow due context switches. If you want to use a custom backend you should do it in libgit2 (see https://github.com/libgit2/libgit2-backends)

[carlosmn]

Rugged does have some left-over backends support, but it was removed because it's simply too slow to implement the backend in a scripting language. You have to move the data from C to python, convert it and then do the whole dance in reverse. By the time the data makes it to whatever database you want to store the data, you've grown old and don't care about it any more.

What would be good is a way to connect the backend written in native code with the library directly, but be able to control that through python, so you don't have to write system code. How simple or doable that is, however, is a different matter.

[bendavis78]

What about native backend support? Would there be a way for me to use pygit2 with one of these existing backends? https://github.com/libgit2/libgit2-backends

[jdavid]

I have never looked into this.

We need some courageous adventurer to explore it... Share here if you try.

[jmorse]

Hi,

I've taken a look at this matter and prototyped some code linking up libgit2/libgit2-backends with pygit2. It'd be great to get some feedback on whether there's interest in merging it, whether I've missed something etc.

libgit2 neatly separates it's operations from how data is stored through storage backend API's, published in git2/sys/*.h. To use a custom backend with libgit2, one has to build a git_repository around odb / refdb objects configured to use the custom backend rather than the default filesystem backends. Once the git_repository is initialized, the backend is completely transparant to the libgit2 API. Thus, pygit2 would only need to build custom git_repository's to support custom backends.

In the work at jmorse/pygit2@6edb77f5 I've added support for creating pygit2 repo's straight from pre-made git_repository's wrapped in a PyCapsule. I've then put code in a different repo/package (jmorse/pygit2-backends@e1597808) that builds the mysql custom backend, creates or opens a storage backend in a database, constructs a git_repository and wraps it in a pygit2.Repository. Some code to exercise this is in 'testscript.py' in that repo.

Assumptions I've made:

• Making pygit2 depend on custom backends would be bad, hence the separate repo/package. (I'm no expert on python packaging, perhaps there's a better way).
• The user of this facility knows that they're not getting a fully fledged git repo out of this. Things like the index, working copy, reflog etc won't work with this technique.

Caveats:

• I've worked off libgit2 v0.19 and the corresponding pygit2 release. This is the latest release supported for my ubuntu 14.4 machine.
• Indirect / symbolic references don't work with the mysql backend in it's current state (and probably never will).
• I've paid no attention to supporting python3
• No support for custom configuration backends
• I've had to fix a number of things in the mysql backend, including writing the refdb facility from scratch. Until I get some real tests in place, it may be dodgy. I haven't yet submitted a pull request to the libgit2/libgit2-backends maintainers.

Opinions would be most appreicated.

[jdavid]

With PR #942, #948 and #982 merged I consider this done.
Sorry @jmorse I could not look at your work at the time.
Please open new issues if there's something left.

[webknjaz]

@jdavid it looks like the docs haven't been updated on the website. Do you not have auto-update from master?

[jdavid]

No, I update them manually with every release. I'll try to do the release this weekend.