Replace git cloner with go getter to support various target #2169

yujunz · 2020-02-02T04:19:18Z

Liujingfang1 · 2020-02-03T17:06:18Z

Originally we used go-getter to support the remote targets. When we integrated kustomize into kubectl, the go-getter and it dependencies blocked the integration. So we removed it. There is no plan for adding it back for now. You can use the go-getter plugin instead. Thank you.

yujunz · 2020-02-04T05:54:12Z

TIL

Shall we consider making the cloner configurable? i.e. instead of hardcoding it as ClonerUsingGitExec, allow executing arbitrary binary configured by user. This would not introduce any dependency on kustomize and is backward compatible.

yujunz · 2020-02-04T05:56:52Z

cc @howardjohn for visibility

pwittrock · 2020-02-24T17:19:04Z

Shall we consider making the cloner configurable? i.e. instead of hardcoding it as ClonerUsingGitExec, allow executing arbitrary binary configured by user. This would not introduce any dependency on kustomize and is backward compatible.

@yujunz I think this is what @Liujingfang1 was suggesting by using a generator plugin.

pwittrock · 2020-02-24T17:21:53Z

Using go-getter would be nice, we should revisit if this is possible. IIRC, the go-getter code pulled in a massive transitive dependency list. There may be a way to prune this through an upstream contribution. Or perhaps it won't be as controversial now.

yujunz · 2020-02-25T02:46:17Z

@yujunz I think this is what @Liujingfang1 was suggesting by using a generator plugin.

I did implement a generator plugin for go-getter. It was not so easy to configure and use.

The most user friendly way to solve the issue would be include go-getter package and solve the dependency issue in integration. How do we check it without merging this PR? @Liujingfang1

Customizable cloner still requires user to install the cloner (git, go-getter or anything) by themselves. A tradeoff for flexibility. If it looks good for all, I can make some time to implement it.

pwittrock · 2020-02-25T19:38:55Z

@yujunz I'm not sure I understand the proposal? Is it to create a new pluggable-cloner mechanism that would exec out to a cloner? Or something else?

Liujingfang1 · 2020-02-25T19:50:46Z

The most user friendly way to solve the issue would be include go-getter package and solve the dependency issue in integration. How do we check it without merging this PR? @Liujingfang1

You can take a look at the api/go.sum to check what packages have been added when you add a new dependency. For example, github.com/aws/aws-sdk-go is added.

yujunz · 2020-02-26T03:08:53Z

kustomize/api/internal/git/cloner.go

Line 34 in 5707962

cmd := exec.Command(

The existing cloner is a simply exec of gitProgram. So if we can provide a way to make it declarative then it should work with arbitrary binary. e.g. in kustomizeconfig.yaml

cloners:
- go-getter
- aws s3 cp -r

Then we shall try exec ${cloner} ${resource} ${dest} until success. @pwittrock

yujunz · 2020-02-26T03:17:55Z

You can take a look at the api/go.sum to check what packages have been added when you add a new dependency. For example, github.com/aws/aws-sdk-go is added.

IIUC, we want to be cloud agnostic. I think the most wanted source is https archive. The go standard package should be sufficient.

I can try to tail go-getter to make it embeddable in kustomize. Does that work for you? @Liujingfang1

Liujingfang1 · 2020-02-26T18:16:40Z

You can take a look at the api/go.sum to check what packages have been added when you add a new dependency. For example, github.com/aws/aws-sdk-go is added.

IIUC, we want to be cloud agnostic. I think the most wanted source is https archive. The go standard package should be sufficient.

I can try to tail go-getter to make it embeddable in kustomize. Does that work for you? @Liujingfang1

It would be great if you can do it by go standard packages.

yujunz · 2020-02-29T09:55:23Z

@Liujingfang1 I made a try to embed the go-getter package.

The changes to original go-getter are

removed s3 and gcs from detect and get
replaced module path of helper/url

Some additional notes

don't be scared about the 180 files, most of them are test data as it is
it still introduces several depedency
- they are all small helper module and unlikely break upstream integration
- we may rewrite some of them with standard package, but it may not worth doing so

Let me know if it works for you. cc @pwittrock

Liujingfang1

Can you add some tests that are not supported previously, but work with this change?

Among the 180 files, lots of them are not needed. Can you do a clean up? I'm also wondering if you can push this go-getter version to another branch of the upstream go-getter.

@monopole Any suggestions?

Liujingfang1 · 2020-03-04T00:15:34Z

api/go.mod

@@ -16,3 +23,5 @@ require (
 	k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a
 	sigs.k8s.io/yaml v1.1.0
 )
+
+replace sigs.k8s.io/kustomize/api/internal/getter/helper/url => ./helper/url


Why adding this replace? I don't think it's needed.

Liujingfang1 · 2020-03-04T00:19:38Z

api/loader/loader.go

-		return newLoaderAtGitClone(
-			repoSpec, fSys, nil, git.ClonerUsingGitExec)
+
+	root, errD := demandDirectoryRoot(fSys, target)


Liujingfang1 · 2020-03-04T00:20:28Z

api/loader/fileloader.go

-	if err == nil {
-		// Treat this as git repo clone request.
+
+	root, errD := func() (filesys.ConfirmedDir, error) {


The previous order is first to treat target as a remote target; when failed, treat it as a local target. The changes make the order reverted. Is there any reason for doing this?

In fact, getter is capable to handle local target so we cannot check it with the error return as repoSpec, err := git.NewRepoSpecFromUrl(path). However it may not behavior exactly the same, e.g. the containment violation.

Putting local target before getter is to ensure backward compatibility.

yujunz · 2020-03-04T08:34:01Z

Can you add some tests that are not supported previously, but work with this change?

You mean an integration test? We can update the examples to include them, which is also covered by automatic testing IIRC. For unit test, I think it is self contained. We may mock some call to ensure it reaches the right method.

Among the 180 files, lots of them are not needed. Can you do a clean up?

I can try my best to clean up a bit.

I'm also wondering if you can push this go-getter version to another branch of the upstream go-getter.

Not sure it can be accepted but we definitely can propose that.

Any recommended justification?

yujunz · 2020-03-04T12:12:12Z

Pushed an update with dependency on a forked go-getter.

It is quite tricky to handle go mod with branches or forks. replace does not work well. So I ended up in modifying the package name in code. See yujunz/go-getter@fc7e8cd

Do you think we can host a fork under kubernetes-sigs organization? That would make the patch in kustomize cleaner and easier to maintain. @Liujingfang1 @monopole

yujunz

Replies inline.

yujunz · 2020-03-06T01:52:54Z

Link upstream discussion about reducing dependency: hashicorp/go-getter#193

Liujingfang1 · 2020-03-06T23:30:16Z

Let's merge this for now. When we can get some updates from go-getter side for hashicorp/go-getter#193, we can change the dependency to the lite version of go-getter. Thank you for working on this. @yujunz As a follow up, can you add an integration test or an example test for this?

/lgtm
/approve

k8s-ci-robot · 2020-03-06T23:30:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Liujingfang1, yujunz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Liujingfang1]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…-getter" This reverts commit 0b1ad03, reversing changes made to 300dd10.

Revert "Merge pull request #2169 from yujunz/loader/go-getter"

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 2, 2020

k8s-ci-robot requested review from justinsb and Liujingfang1 February 2, 2020 04:19

yujunz force-pushed the loader/go-getter branch from b40efba to f3fef38 Compare February 29, 2020 08:29

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 29, 2020

yujunz force-pushed the loader/go-getter branch from d41a0d6 to 4b97fba Compare February 29, 2020 09:35

Liujingfang1 reviewed Mar 4, 2020

View reviewed changes

yujunz and others added 3 commits March 4, 2020 20:04

Replace git cloner with go getter to support various target

fae6644

Use a tailored go-getter to get rid of transitive depenencies

a24bf6a

Use a lite fork of go-getter

735cefa

yujunz force-pushed the loader/go-getter branch from 4b97fba to 735cefa Compare March 4, 2020 12:05

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Mar 4, 2020

yujunz mentioned this pull request Mar 4, 2020

Lite version with fewer dependency hashicorp/go-getter#236

Closed

yujunz commented Mar 5, 2020

View reviewed changes

yujunz requested a review from Liujingfang1 March 5, 2020 02:46

k8s-ci-robot assigned Liujingfang1 Mar 6, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 6, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 6, 2020

k8s-ci-robot merged commit 0b1ad03 into kubernetes-sigs:master Mar 6, 2020

yujunz deleted the loader/go-getter branch March 7, 2020 01:41

yujunz mentioned this pull request Mar 7, 2020

go mod tidy to fix version of github.com/yujunz/go-getter #2263

Closed

yujunz added a commit to yujunz/kustomize that referenced this pull request Mar 7, 2020

Revert "Merge pull request kubernetes-sigs#2169 from yujunz/loader/go…

711bab8

…-getter" This reverts commit 0b1ad03, reversing changes made to 300dd10.

This was referenced Mar 7, 2020

Revert "Merge pull request #2169 from yujunz/loader/go-getter" #2264

Merged

Allow loading file from http #2167

Merged

k8s-ci-robot added a commit that referenced this pull request Mar 9, 2020

Merge pull request #2264 from yujunz/revert-2169

6944cea

Revert "Merge pull request #2169 from yujunz/loader/go-getter"

yujunz mentioned this pull request Nov 18, 2020

kustomize 3.5.5 breaks installing ArgoCD from URL #2538

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace git cloner with go getter to support various target #2169

Replace git cloner with go getter to support various target #2169

yujunz commented Feb 2, 2020

Liujingfang1 commented Feb 3, 2020

yujunz commented Feb 4, 2020

yujunz commented Feb 4, 2020

pwittrock commented Feb 24, 2020

pwittrock commented Feb 24, 2020

yujunz commented Feb 25, 2020

pwittrock commented Feb 25, 2020

Liujingfang1 commented Feb 25, 2020

yujunz commented Feb 26, 2020

yujunz commented Feb 26, 2020

Liujingfang1 commented Feb 26, 2020

yujunz commented Feb 29, 2020

Liujingfang1 left a comment

Liujingfang1 Mar 4, 2020

yujunz Mar 5, 2020

Liujingfang1 Mar 4, 2020

yujunz Mar 5, 2020

Liujingfang1 Mar 4, 2020

yujunz Mar 5, 2020

yujunz commented Mar 4, 2020 •

edited

Loading

yujunz commented Mar 4, 2020

yujunz left a comment

yujunz commented Mar 6, 2020

Liujingfang1 commented Mar 6, 2020

k8s-ci-robot commented Mar 6, 2020

Replace git cloner with go getter to support various target #2169

Replace git cloner with go getter to support various target #2169

Conversation

yujunz commented Feb 2, 2020

Liujingfang1 commented Feb 3, 2020

yujunz commented Feb 4, 2020

yujunz commented Feb 4, 2020

pwittrock commented Feb 24, 2020

pwittrock commented Feb 24, 2020

yujunz commented Feb 25, 2020

pwittrock commented Feb 25, 2020

Liujingfang1 commented Feb 25, 2020

yujunz commented Feb 26, 2020

yujunz commented Feb 26, 2020

Liujingfang1 commented Feb 26, 2020

yujunz commented Feb 29, 2020

Liujingfang1 left a comment

Choose a reason for hiding this comment

Liujingfang1 Mar 4, 2020

Choose a reason for hiding this comment

yujunz Mar 5, 2020

Choose a reason for hiding this comment

Liujingfang1 Mar 4, 2020

Choose a reason for hiding this comment

yujunz Mar 5, 2020

Choose a reason for hiding this comment

Liujingfang1 Mar 4, 2020

Choose a reason for hiding this comment

yujunz Mar 5, 2020

Choose a reason for hiding this comment

yujunz commented Mar 4, 2020 • edited Loading

yujunz commented Mar 4, 2020

yujunz left a comment

Choose a reason for hiding this comment

yujunz commented Mar 6, 2020

Liujingfang1 commented Mar 6, 2020

k8s-ci-robot commented Mar 6, 2020

yujunz commented Mar 4, 2020 •

edited

Loading