-
Notifications
You must be signed in to change notification settings - Fork 280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Better Backend Support #528
base: master
Are you sure you want to change the base?
Conversation
@tpickett66 would you mind taking a look? This is obviously still WIP, but I think this should support all the your use cases once ready. |
Wow, I wasn't expecting such a big response on this idea. I'm excited to take a look first thing in the morning! |
Ok, I lied; I was too excited to wait. I have to say this is almost exactly what I was wanting to build when I started but at the time wasn't sure such a major rework on the backend system would be looked kindly upon by someone who just popped up. How can I help round this work out? I'd be happy to start filling in some of the various C -> Ruby bridge methods or build a reference backend with documentation or really any other task that would be useful. |
Hey @tpickett66 Glad to hear this is shaping out to do exactly what you were missing. I'll work on filling in most of the missing pieces here and there over the course of the week. Existing native backends like One thing that I wanted to let you know, and we also should mention in the docs for alternative backends (mostly for odb backends) is that they all have pretty abysmal performance when compared with the default filesystem based backends. This is mostly due to the amount of operations that need to happen when reading and writing objects into the odb. |
@arthurschreiber Thanks for the heads up on the poor performance on ODB backends, after digging into how the ODB works it totally makes sense that performance would be terrible for non-fs backends. Since this project is an experiment it won't be the end of the world if performance is terrible but hopefully we can get it to a level that's acceptable for the interactions we're needing. |
Great to see this being extended! Looks good. @arthurschreiber How do you plan to implement the support for backends implemented in ruby? I believe that support was originally included in Rugged (I can try and dig out an old commit I found last time I looked), but was later removed for performance reasons – there was too much context switching (too many indirections) between ruby and C on every read and write to make backends written in Ruby practical (e.g. for Redis, it would be something like That's also the reason why I've done the backends the way they are implemented, because they only get injected from Ruby code, but the reads and writes are all performed in native land as if you were using libgit and the backend library directly. Did you come up with a better solution for this? |
@charypar I'll factor out the Ruby backend implementations into a separate gem and add a big scary warning that'll say "DANGERZONE - DO NOT USE IN PRODUCTION". On another note, how did you even use the redis backend? I wanted to open a PR against your repo to switch it over to the new backend API and ran into the issue that the |
@arthurschreiber That's a bit strange! we used it in [https://github.com/bskyb/colonel] as a storage backend. The As an aside, I personally haven't used |
The way the shared connection thing is implemented right now can not really work with multiple backend instances. If you have two redis backends open, and close one, this will cause the connection to be freed and the second backend will become unusable. Also, the shared connection thing prevents two backends using two different redis hosts. |
@arthurschreiber I think the reason it doesn't support it is we never needed multiple backend instances at the same time. I couldn't think of a case when you'd need it and what the situation would be, so I skipped it (would be interested in hearing about some). If you can get it not to waste connections inside one backend instance and still allow multiple to exist, great, I'm all for it. 👍 |
I pulled out the code that allows defining backends in ruby into a separate repository, rugged-ruby. That will serve as a good example of how new backends can be exposed to rugged. |
de4191e
to
c937735
Compare
@arthurschreiber do you have any sort of idea of how much defining them in ruby affects performance yet? Just out of interest |
@charypar No idea, except that it'll probably be pretty bad. I can run some benchmarks against the "native" rugged-redis once everything is ready here. |
@arthurschreiber awesome, will keep an eye out for those 👍 |
4ec3211
to
248eae9
Compare
@tpickett66 Did you have any change to take a look over https://github.com/arthurschreiber/rugged-ruby? I started implementing a refdb and config db backend, and would appreciate any contributions. @charypar Is there anyone maintaining |
hi, any news on this? |
I'd really like to see this implemented too. |
Any chance we can update and/or merge this? I'd also like to see support for the in-memory backend, which would be super useful for testing. @carlosmn how could I help get this over the finish line? |
Initial backend support was added by @charypar as part of #410.
Unfortunately, the implementation is a bit convoluted: Every backend implementation needs to
include <rugged.h>
to get the definition ofrugged_backend
. Also, the way backends are implemented right now, they always need to provide arefdb
andodb
backend together, there's no way to use e.g. one backend type for the odb and another backend type for the refdb. Also, config backends were missing completely.This PR is aimed at fixing those shortcomings and make backend support first class.
It introduces a few new classes that wrap existing functionality from libgit2, provides abstract implementations of custom refdb, odb and config backends and exposes new APIs on the
Repository
object to make it easier to initialize repositories with custom backends.Rugged::Refdb
to wrapgit_refdb
functionality fromlibgit2
.Rugged::Refdb::Backend::FileSystem
to exposegit_refdb_backend_fs
and to serve as an example on how to wrapgit_refdb_backend
.AddRugged::Refdb::Backend::Custom
as an abstract base class that can be extended to provide new refdb backends that are defined using Ruby code.Rugged::Odb
to wrapgit_odb
functionality fromlibgit2
.Rugged::Odb::Backend::Loose
,Rugged::Odb::Backend::Packed
andRugged::Odb::Backend::Single
to exposegit_odb_backend_loose
,git_odb_backend_packed
andgit_odb_backend_single
and to serve as examples on how to wrapgit_odb_backend
.AddRugged::Odb::Backend::Custom
as an abstract base class that can be extended to provide new odb backends that are defined using Ruby code.AddRugged::Config::Backend::Custom
as an abstract base class that can be extended to provide new config backends that are defined using Ruby code.