-
Notifications
You must be signed in to change notification settings - Fork 541
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: Ability to configure VCS rewrites #372
Comments
I think you might be able to do this today using the If Point is, having to repeatedly add that information in PS - does your internal gitolite service really have more uptime 9's than github? 😁 |
It actually does, believe it or not :) Anyway, yes, we can (and do) accomplish this today by hand, by editing the glide.yaml file to point to our mirror. This all works well. The problem is that it's opt-in, so if somebody forgets to do it (and really, it's a pain to do manually anyway), then our build hosts will be pulling from github.com, which is considered a non-starter to our SRE team. What you've suggested, transparently rewriting the URLs in a custom glide, is the approach that seems easiest (nothing to maintain for you, we can write code specifically tailored to our needs internally). Before I start going down that road, I was curious whether this issue has come up before, and whether it's worth making generic for you. Happy to do it either way. It sounds like a custom glide on our build servers is the best way to go for now. |
I can! Mostly ribbing.
Glad we're on the same page about that :) And I realize that I think I misread your original proposal - I was assuming you were suggesting an additional stanza on a per-dep basis, but instead you were suggesting a single, additional config stanza that would perform a general rewrite on all dep URIs it runs into. My bad. If the concern is really just over ensuring that the build servers aren't reaching directly out to github, wouldn't the easier solution - and more fully within the SREs' control, I imagine - be to intercept that via DNS? I didn't mention that approach at first because I thought the goal was also to affect developers' environments, but if it's just build boxes you're concerned about... In any case, yes, the issue's come up in discussion, and there's probably at least one issue with a similar flavor somewhere in the queue. IMO, there is space for adding something like this. Since you're asking about this, though, and it does intersect a bit with something else, let me ask you: would the presence of a local cache of repositories address any of the issues here? If/when we get around to integrating the To be clear, these 'caches' are necessary whether or not you have the mirrored intermediaries. But I'm asking because one of the design choices I'm actively considering right now is how well to support "offline" use of these caches. |
DNS rewriting is interesting, but wouldn't be an out-of-the-box solution, in our case. It's not just hostname rewriting (regardless of the implications of SSH key verification), but the path attribute that comes after needs to have a I think the local cache would totally solve our use case. The only behavior I'm not sure would be defined for the filesystem cache, as opposed to the gitolite cache, is what happens when somebody force-pushes their master branch, and this orphans a ref that eventually gets garbage collected? Rare, and totally avoidable if people are being good citizens, but we really don't want a production build to ever fail due to changes happening outside of our control. Is there a timeline on the vsolver engine? Sounds like "TBD". I think in the interim we will patch our internal glide build to have this behavior, and then at some point we may be able to drop it. I'll close this for now. If somebody finds this issue at a future point in time and would like to see our code upstreamed, please re-open and I'll share. Thanks! |
Or, gasp, when they move a tag to a different commit. Yep, we have to handle the bad citizen cases, and we do. Though I've opened an issue to cover it in particular: sdboyer/gps#6 . Kind of less "issue" and more "dump of thoughts," but gotta start somewhere.
IMO, that's a significant responsibility of a package management tool.
We're trying to sort that out now. The code's coming along nicely - hopefully will be ready for integration in a matter of weeks. But there are a number of changes to glide's workflows and options that we need to sort out. @technosophos, @mattfarina and I need to talk that through. In the meantime, I wouldn't at all mind you opening an issue (and/or jumping on that one I linked earlier) detailing the expectations you'd around the behavior of a local cache.
Fingers crossed! |
@sectioneight this is actually a duplicate of #39. I identified early on that we need this. Having overriding aliases is definitely something we need to have for many enterprise environments and some places that require high control. Where I work we need the same level of control your SREs require. If you're going to craft this for your own needs would you want to try and contribute it? |
I believe @arschles is also working on this. If the git protocol were On Tue, Apr 5, 2016, 7:05 PM Matt Farina notifications@github.com wrote:
|
@technosophos restricting to http has its own weirdness, i think - it would require (potentially) ignoring, or at least rewriting, custom-set of course, i suppose that'd also depend on the implementation you have in mind :) |
@technosophos @sdboyer I think it's far easier than that. When a |
@mattfarina ok - where is that list coming from? such a list has also been on my thoughts, because (assuming that i'm thinking what you're thinking) i need to include it as part of the identity of a source URI in the solver. basically, we're talking a list of URLs that are all capable of locating the same URI, right? |
@technosophos @mattfarina I'm working with @sectioneight and may submit a PR with this functionality in the near future (rewrite rules)...how would you feel about supporting this in some kind of global config ( Then again, it's adding a new global config file, so I figured I'd ask specifically first. |
@crsmithdev That is sorta the direction I was thinking, as well. I guess we could do something like "load the global .gliderc, and if there's a local .gliderc (e.g. in same dir as glie.yaml), merge that with the global". But even starting with a global config gives us a strong starting point. |
I would love to see this. In our CI stack the infrastructure is shared between multiple projects with different config, writing to the $HOME is likely to cause concurrency and clean up issues. A file that can be git ignored and put into the project dir by the initialization hook would be great. |
Late to the party here, responding to @technosophos. I've been working on a proxy (https://github.com/arschles/goprox) that I just got minimally working. It serves the same functionality as the gitolite solution that @sectioneight originally mentioned, but backs each repo to S3. On the functionality side, I'm planning on adding some other small features (like aliasing, for those who don't use glide) and a server admin tool, but it's just about feature complete now on the end-user side. Anyway, I discussed with @technosophos a few weeks ago, and the topic of rewriting transitive dependencies came up. What's the thinking on this? As a concrete example, if I have |
Good question about transitive dependencies. We're currently solving this by patching masterminds/vcs internally, but there will likely need to be a PR to |
thanks for letting me know @sectioneight. fwiw, I'd want to rewrite the entire dependency tree by default. rewrites would be significantly less useful for me if I was only able to rewrite direct dependencies (i.e. at the top of the dep tree). The |
Yes, our end goal is the same: transparent rewrites including transitive dependencies |
i started writing a response here, but ended up not putting it in because...reasons. so i'll just note that i suspect that the first post- |
@sdboyer @sectioneight is there an issue/issues that I can follow related to that work? |
@arschles yep - initial I also jotted down some of my notes in a vsolver issue (sdboyer/gps#10), though I'm now revisiting that thinking a bit. The issue is probably a bit arcane, but if you want to discuss, I'm happy to clarify as needed. |
Thanks @sdboyer ! |
So I realize this is a bit of a special case, but wanted to run by you to see if you'd be open to accepting a PR.
At my place of employment, we are switching to Glide (from Godeps). At the same time, we'd like to move away from vendoring. Glide is great for this, and we get reproducible builds when GitHub is up, but want to guard against repo renames/GH instability.
Internally, we have a read-through cache/mirror of GitHub repos, so that if we pull from a special git URI (e.g.
git clone git@mirror:github/Masterminds/glide
) it will read the latest state from GitHub, but still return you the code if GitHub goes down (via gitolite's mirroring options).We'd like the ability to configure glide internally, such that if an engineer adds a github.com dependency to their glide.lock/yaml, it will get pulled through our mirror.
Alternatively, we are open to patching glide on the client side, such that any "glide get github.com/..." will result in the proper VCS settings in glide.yaml. This is slightly less preferable because then we have to distribute a patched glide.
I'm open to implementing this on an internal fork if you don't foresee others needing this functionality.
What I'm thinking is a
~/.glide.yaml
that has a block like:What do you think?
The text was updated successfully, but these errors were encountered: