fetching alpha1 packages fails to fetch the package

[mikebz]

Describe your problem

It looks like when I fetch an old package I only get the Kptfile, but not anything else in that package. This makes it impossible for me to update the package locally. The work around I need to resort to then is either getting alpha1 kpt and juggling both of them, or using raw git.

Example:

$ kpt pkg get https://github.com/GoogleCloudPlatform/blueprints.git/catalog/bucket  
Package "bucket":
Fetching https://github.com/GoogleCloudPlatform/blueprints@main
From https://github.com/GoogleCloudPlatform/blueprints
 * branch            main       -> FETCH_HEAD
 * [new branch]      main       -> origin/main
Error: Kptfile at "https:/github.com/GoogleCloudPlatform/blueprints/catalog/bucket@main" has an old version ("v1alpha1") of the Kptfile schema.
Please update the package to the latest format by following https://kpt.dev/installation/migration. 
$ ls
bucket
$ cd bucket 
$ ls
Kptfile

What would I expect instead?

When fetching the old package the new binary should at the very least do what I would do with git - grab the current package and have full contents of it on my local machine. That way I can fork and migrate.

Steps to fork and migrate:

1. I do kpt pkg get of an old package
2. I get a warning which links to a place to download the old kpt
3. If I proceed I am obviously interested in migrating. I run the fix function written by @phanimarupaka
4. I am now able to save the updated package to a branch or another git repo.

Q: What happens if I am under impression that I will still be able to get updates upstream?
We should give an error that says that the upstream package is of the old format and the consumer should ask the producer to migrate the package and provide the link to the download of the old binary if they updated by mistake.

[morgante]

Personally, I would expect that I can fetch an alpha1 package. It should download the contents as they are from the repo and prompt me that I need to upgrade (by running kpt fn fix). This is better UX than forcing customers to leave kpt and figure out the right Git commands to get a copy of the package.

[phanimarupaka]

I think it would be confusing to just fetch a package using kpt pkg get and making the functionality equivalent to git clone. kpt pkg get does many more things than just fetching the package and users have different expectations from get command.

In v0.39 kpt pkg get does the following things

1. Fetches the package (= git clone)
2. Adds Kptfile to top level folder if it is not present
3. Updates the upstream section of Kptfile
4. Applies auto-setters

Now we are saying if the fetched package is of old version, then just exit after step 1. I think if would confuse users more when kpt pkg get behaves differently.

Instead, users are prompted to use, git clone which makes it very clear as to what to expect of that command. I think we should not make changes for now and wait for more feedback.

[morgante]

Why couldn't we fetch the package, run kpt fix, then add upstream info?

[phanimarupaka]

You mean kpt1 pkg get should fetch any package, run fix ? That would be confusing and doesn't fall under the contract of get command. We should have migration to be a clear step and users should know what to expect from it.

This is better UX than forcing customers to leave kpt => git is already part of kpt workflow for initializing and committing changes. So it is not a new thing to use git.

[mikebz]

I think other options are on the table as well. One of the ideas to explore is possibly to use the checkout with "detached head" which explicitly disables upstream rebase down the line: https://www.git-tower.com/learn/git/faq/detached-head-when-checkout-commit/

So if there is a command to get a detached pkg get command then people can fix it locally.

My fear is that without having 1-2 commands that easily help migration a large portion of the packages we are forcing the package consumer to do too much. What's losing here is kpt.

[morgante]

You mean kpt1 pkg get should fetch any package, run fix ? That would be confusing and doesn't fall under the contract of get command. We should have migration to be a clear step and users should know what to expect from it.

Fine, we don't need to run fix automatically but I think it is still reasonable to expect kpt pkg get to be able to fetch older packages.

This is better UX than forcing customers to leave kpt => git is already part of kpt workflow for initializing and committing changes. So it is not a new thing to use git.

It is new to require consumers to make their own checkout of the entire upstream repo.

[mikebz]

Another data point to consider here is the desire to support the following use cases:

1. Support OCI images in addition to git #2300 by @bgrant0607
2. dumb http servers not supported #2352 by @justinsb
3. local packages to aid in working in a monorepo without having to go through git roundtrips (https://kubernetes.slack.com/archives/C0155NSPJSZ/p1625160488048400)

All of those use cases would probably result in not being able to fully support a 3 way merge and I am not sure that customers would expect that. The higher order decision here is - do we support what the upstream storage system supports (lowest level being rsync) or do we limit the upstream storage and format because we want to ensure advanced merge scenarios.

[bgrant0607]

We should be able to support a 3-way merge with OCI, with an opinionated versioning model.

[mortent]

So we have discussed supporting backwards compatibility and automatic migration of kpt v1alpha1 packages. Given the number of changes to the package format and the functionality in kpt v1, this will require significant amount of work and it will not be a seamless migration. We just don't think the resulting experience will be worth the additional effort and complexity required. Given that we have published the v1 version of the package format, it should not change in any backwards incompatible ways unless there is a v2 version of the format at some point in the future.

As for supporting OCI, I expect we will be able to do that with 3-way merge. The current package schema is designed to allow for non-git storage backends.

Supporting the "dumb" git protocol will not cause any problems for the 3-way merge. It is still git, so we can fetch old revisions just like we currently do. It is just less efficient and not really widely used.

[morgante]

We don't necessarily need to support automatic migration.

I think the ask here is just to be able to run kpt pkg get and end up with a copy of the package locally which I can then begin to migrate.

What would be the significant complexity in that?

[kalhaarsavaj]

Hello team, this error seems to be persistent. We

The workaround is to download this file from github ASM Packages and clean up the resources manually by running kubectl delete -f asm/canonical-service/controller.yaml