🐛 Allow using the --from flag to get a template from a github release #7453

nunnatsa · 2022-10-25T14:05:42Z

Currently, clusterctl generate cluster's --from flag only supports yaml files from github code (blob). Trying to get a template asset from a release tag is failing.

This PR adds the option to use the --form also for release assets.

Fixes #7455

Signed-off-by: Nahshon Unna-Tsameret nunnatsa@redhat.com

alexander-demicev · 2022-10-26T08:27:32Z

cmd/clusterctl/client/cluster/template.go

@@ -181,35 +182,77 @@ func (t *templateClient) getGitHubFileContent(rURL *url.URL) ([]byte, error) {

 	// Extract all the info from url split.
 	owner := urlSplit[0]
-	repository := urlSplit[1]


just curious, what is the reason for renaming the variable?

The linter says: "Variable 'repository' collides with imported package name". Same for the client var below.

alexander-demicev · 2022-10-26T08:29:54Z

cmd/clusterctl/client/cluster/template.go

+		return getGithubFileContentFromCode(ghClient, rURL.Path, owner, repo, path, branch)
+
+	case "releases": // get a github release asset
+		tag := urlSplit[4]


does it make sense to extract it above with all other variables related to the URL?

Not sure. I tend not to do that, because the meaning of the 4th and 5th path components is different between these two cases.

Agree with not extracting them early. Those values hold different meaning under blob and releases cases.

alexander-demicev · 2022-10-26T09:31:45Z

cmd/clusterctl/client/cluster/template.go

+func getGithubAssetFromRelease(ghClient *github.Client, path string, owner string, repo string, tag string, assetName string) ([]byte, error) {
+	release, _, err := ghClient.Repositories.GetReleaseByTag(ctx, owner, repo, tag)
+	if err != nil {
+		return nil, handleGithubErr(err, "failed to get release '%s' from %s%s repository", tag, owner, repo)


Suggested change

return nil, handleGithubErr(err, "failed to get release '%s' from %s%s repository", tag, owner, repo)

return nil, handleGithubErr(err, "failed to get release '%s' from %s/%s repository", tag, owner, repo)

alexander-demicev · 2022-10-26T09:33:03Z

cmd/clusterctl/client/cluster/template.go

 	}
 	return content, nil
 }

+func getGithubAssetFromRelease(ghClient *github.Client, path string, owner string, repo string, tag string, assetName string) ([]byte, error) {
+	release, _, err := ghClient.Repositories.GetReleaseByTag(ctx, owner, repo, tag)


should we check for a non-nil release?

nunnatsa · 2022-10-28T07:42:40Z

@alexander-demicev - could you please review again, after the fixes?

ykakarap · 2022-11-01T00:16:58Z

cmd/clusterctl/client/cluster/template.go

+		return getGithubFileContentFromCode(ghClient, rURL.Path, owner, repo, path, branch)
+
+	case "releases": // get a github release asset
+		tag := urlSplit[4]


Agree with not extracting them early. Those values hold different meaning under blob and releases cases.

ykakarap · 2022-11-01T00:20:00Z

cmd/clusterctl/client/cluster/template.go

+	var rc io.ReadCloser
+	for _, asset := range release.Assets {
+		if asset.GetName() == assetName {
+			rc, _, err = ghClient.Repositories.DownloadReleaseAsset(ctx, owner, repo, asset.GetID(), ghClient.Client())


I believe you will also have to handle the redirects when working with release assets. Similar to how we handle the redirects here:
https://github.com/nunnatsa/cluster-api/blob/fd24c247c3ae9b32fed236f544088486ec6331cf/cmd/clusterctl/client/repository/repository_github.go#L331-L344

I believe you will also have to handle the redirects when working with release assets. Similar to how we handle the redirects here: https://github.com/nunnatsa/cluster-api/blob/fd24c247c3ae9b32fed236f544088486ec6331cf/cmd/clusterctl/client/repository/repository_github.go#L331-L344

Thanks @ykakarap. I'm not sure it's needed. According to the DownloadReleaseAsset documentation:

followRedirectsClient can be passed to download the asset from a redirected location. Passing http.DefaultClient is recommended unless special circumstances exist, but it's possible to pass any http.Client. If nil is passed the redirectURL will be returned instead.

I added unit test to check the redirect scenario. Also, This code is working with the real github, that actually responses with redirect. If you think it's a must, I'll add it, but it's not needed, I think.

Seem to work for redirected assets as well. @fabriziopandini Do you know why we are redirecting here: https://github.com/nunnatsa/cluster-api/blob/fd24c247c3ae9b32fed236f544088486ec6331cf/cmd/clusterctl/client/repository/repository_github.go#L331-L344?

Do you know of a case that is not covered by the standard DownloadReleaseAsset function?

@ykakarap we implemented this because this is one of the use case allowed by the return parameters of the GitHub API, but AFAIK we did not investigate why this parameter exists / in which case they support.

Unless we collect evidence this is unnecessary, I strongly suggest to handle redirect also here.

Thanks @fabriziopandini. According to the DownloadReleaseAsset documentation I copied above, the redirect is only returned if the followRedirectsClient parameter is empty. This is, I guess, if you want to implement the redirect logic. But if you pass the client, the function will do it for you. This is how I understand it.

kk, I double checked the DownloadReleaseAsset implementation and it seems that this is not necessary if we are passing a followRedirectsClient.
In a separated PR we can also eventually clean up also the bit of code @ykakarap was referencing to

Created a follow-up issue to track the clean up work. #7942

sbueringer · 2022-11-01T07:26:08Z

@ykakarap Can you please double check if this PR is in conflict with #7371 or if they are othogonal? (feature-wise)

nunnatsa · 2022-11-01T11:11:37Z

... if this PR is in conflict with #7371 or if they are othogonal? (feature-wise)

Thanks @sbueringer - I didn't know #7371, but I just read it. As far I can see, not only this PR is not conflicts with 7371, but it is even more important with it, because with 7371, blob github link will work, non-github link will work, but github release asset links will fail - and so #7455 is even more meaningful bug.

ykakarap · 2022-11-02T00:12:16Z

@ykakarap Can you please double check if this PR is in conflict with #7371 or if they are othogonal? (feature-wise)

They are not in conflict with each other. They are complimentary. I agree with @nunnatsa that having both 7371 and this PR would be beneficial as they would cover a broader set of cases.

sbueringer · 2022-11-02T08:22:57Z

Sounds good! Thx for checking

CecileRobertMichon · 2022-11-03T23:25:21Z

/cc @Jont828

ykakarap

Overall LGTM.

This will be a nice addition :)

ykakarap · 2022-11-15T03:39:57Z

cmd/clusterctl/client/cluster/template.go

@@ -181,35 +182,81 @@ func (t *templateClient) getGitHubFileContent(rURL *url.URL) ([]byte, error) {



I cannot select the correct line here but the error in l.179 should be adjusted as now we support a blob as well as a release asset.

Thanks @ykakarap - fixed.

ykakarap · 2022-11-15T04:43:57Z

cmd/clusterctl/client/cluster/template.go

+	var rc io.ReadCloser
+	for _, asset := range release.Assets {
+		if asset.GetName() == assetName {
+			rc, _, err = ghClient.Repositories.DownloadReleaseAsset(ctx, owner, repo, asset.GetID(), ghClient.Client())


Seem to work for redirected assets as well. @fabriziopandini Do you know why we are redirecting here: https://github.com/nunnatsa/cluster-api/blob/fd24c247c3ae9b32fed236f544088486ec6331cf/cmd/clusterctl/client/repository/repository_github.go#L331-L344?

Do you know of a case that is not covered by the standard DownloadReleaseAsset function?

nunnatsa · 2022-11-27T16:08:37Z

Hi @fabriziopandini. What do you think about the redirect question from @ykakarap, here? #7453 (comment)

fabriziopandini · 2023-01-02T13:39:38Z

Hi @fabriziopandini. What do you think about the redirect question from @ykakarap, here? #7453 (comment)

Sorry for the delay, answered in the thread above

Currently, `clusterctl generate cluster`'s `--from` flag only support yaml files from github code. Trying to get template from a rlease tag is failing. This PR adds the option to use the `--form` also for release assets. Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>

nunnatsa · 2023-01-02T14:26:58Z

rebased (fix conflicts)

Jont828

Thanks for your contribution, overall this make sense to me. I'll try to test this out locally when I get the chance, were you able to get this working by running a copy of clusterctl built off of this code?

nunnatsa · 2023-01-11T13:05:16Z

were you able to get this working by running a copy of clusterctl built off of this code?

yes

fabriziopandini · 2023-01-11T14:46:16Z

/lgtm
ready to approve as soon as we get a final feedback from @Jont828 and @ykakarap

k8s-ci-robot · 2023-01-11T14:46:23Z

LGTM label has been added.

Git tree hash: 018505cf99c94f0d5cc398e6b24e2b18ab452b3b

ykakarap · 2023-01-17T23:55:36Z

Did a final pass and this is good to go.

/lgtm

Great work! Thank you for adding the release asset support. 🚀

sbueringer · 2023-01-18T07:36:08Z

/approve
Given lgtms above

k8s-ci-robot · 2023-01-18T07:36:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 25, 2022

k8s-ci-robot requested review from stmcginnis and ykakarap October 25, 2022 14:05

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 25, 2022

nunnatsa changed the title ~~Allow using the --from to get template from a github release~~ 🐛 Allow using the --from to get template from a github release Oct 25, 2022

nunnatsa force-pushed the fix-from-url branch 4 times, most recently from b90b1fa to 4a7aae4 Compare October 25, 2022 14:41

nunnatsa changed the title ~~🐛 Allow using the --from to get template from a github release~~ 🐛 Allow using the --from flag to get a template from a github release Oct 25, 2022

alexander-demicev reviewed Oct 26, 2022

View reviewed changes

nunnatsa force-pushed the fix-from-url branch from 4a7aae4 to fd24c24 Compare October 26, 2022 09:46

nunnatsa mentioned this pull request Oct 31, 2022

📖 Add the quickstart details for KubeVirt #7444

Merged

ykakarap suggested changes Nov 1, 2022

View reviewed changes

nunnatsa force-pushed the fix-from-url branch from fd24c24 to 245acc9 Compare November 1, 2022 10:58

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 1, 2022

k8s-ci-robot requested a review from Jont828 November 3, 2022 23:25

ykakarap reviewed Nov 15, 2022

View reviewed changes

nunnatsa force-pushed the fix-from-url branch from 245acc9 to 3c66590 Compare November 15, 2022 05:57

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 2, 2023

nunnatsa force-pushed the fix-from-url branch from 3c66590 to 39e99d3 Compare January 2, 2023 14:26

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 2, 2023

Jont828 reviewed Jan 9, 2023

View reviewed changes

k8s-ci-robot assigned fabriziopandini Jan 11, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 11, 2023

k8s-ci-robot assigned ykakarap Jan 17, 2023

ykakarap mentioned this pull request Jan 18, 2023

clean-up github release asset download code in github repository client's downloadFilesFromRelease #7942

Closed

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 18, 2023

k8s-ci-robot merged commit 686961a into kubernetes-sigs:main Jan 18, 2023

k8s-ci-robot added this to the v1.4 milestone Jan 18, 2023

nunnatsa deleted the fix-from-url branch January 18, 2023 08:15

	return nil, handleGithubErr(err, "failed to get release '%s' from %s%s repository", tag, owner, repo)
	return nil, handleGithubErr(err, "failed to get release '%s' from %s/%s repository", tag, owner, repo)

		@@ -181,35 +182,81 @@ func (t templateClient) getGitHubFileContent(rURL url.URL) ([]byte, error) {

🐛 Allow using the --from flag to get a template from a github release #7453

🐛 Allow using the --from flag to get a template from a github release #7453

Conversation

nunnatsa commented Oct 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nunnatsa Oct 26, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nunnatsa commented Oct 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabriziopandini Jan 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbueringer commented Nov 1, 2022

nunnatsa commented Nov 1, 2022

ykakarap commented Nov 2, 2022

sbueringer commented Nov 2, 2022

CecileRobertMichon commented Nov 3, 2022

ykakarap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nunnatsa commented Nov 27, 2022

fabriziopandini commented Jan 2, 2023

nunnatsa commented Jan 2, 2023

Jont828 left a comment

Choose a reason for hiding this comment

nunnatsa commented Jan 11, 2023

fabriziopandini commented Jan 11, 2023

k8s-ci-robot commented Jan 11, 2023

ykakarap commented Jan 17, 2023

sbueringer commented Jan 18, 2023

k8s-ci-robot commented Jan 18, 2023

nunnatsa commented Oct 25, 2022 •

edited

Loading

nunnatsa Oct 26, 2022 •

edited

Loading

fabriziopandini Jan 2, 2023 •

edited

Loading