Save install receipt #195

corneliusweig · 2019-06-02T23:49:36Z

I want to add more tests, however I would ask for some early feedback whether you like the direction this is taking.

Summary of changes

There is a new manifest kind "Receipt" which contains the details of the install, such as
- date of first install
- date of last update
- plugin manifest of last install/update
The krew info command now looks for index manifests, and receipts. If a receipt is found, the shown information is slightly augmented.
plugin instances are now passed around as pointers instead of full copies. Was there a particular reason for not using pointers?
a few code movements were necessary to remove import cycles
a new dependency on google/go-cmp was added and vendored. If you don't like this, I can also do this manually, but the test assertions won't be so pretty. Otherwise, this library should be used in a few more places.

Todos

There are some loose ends that should probably be taken up in other PRs, because this one is already quite large:

maybe add some migration logic, so that receipts are installed for all currently installed plugins (probably as a post-upgrade activity, so that it is guaranteed that the receipts are correct)
switch uninstall logic to use the receipts instead of the current index

Close #163

corneliusweig · 2019-06-02T23:50:51Z

/assign @ahmetb

codecov-io · 2019-06-02T23:52:38Z

Codecov Report

Merging #195 into master will decrease coverage by 0.21%.
The diff coverage is 46.34%.

@@            Coverage Diff            @@
##           master    #195      +/-   ##
=========================================
- Coverage   53.82%   53.6%   -0.22%     
=========================================
  Files          16      16              
  Lines         732     735       +3     
=========================================
  Hits          394     394              
- Misses        286     289       +3     
  Partials       52      52

Impacted Files	Coverage Δ
pkg/installation/install.go	`32% <0%> (ø)`	⬆️
pkg/installation/upgrade.go	`0% <0%> (ø)`	⬆️
pkg/environment/environment.go	`79.16% <100%> (ø)`	⬆️
pkg/info/info.go	`100% <100%> (ø)`	⬆️
pkg/index/indexscanner/scanner.go	`67.34% <50%> (ø)`	⬆️
pkg/receipt/receipt.go	`66.66% <66.66%> (ø)`	⬆️
cmd/validate-krew-manifest/main.go	`58.33% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b0255c...ff5b7e6. Read the comment docs.

ahmetb · 2019-06-03T03:08:57Z

This is a massive PR.
Please don’t expect me to review this very soon. I don’t think we discussed how the implementation would look like, so I have to find time to study it from the PR.

ahmetb · 2019-06-03T03:09:42Z

But I’m glad we’ll do this before implementing multi-indexes which would definitely require this change.

corneliusweig · 2019-06-03T10:15:50Z

Take your time. As said elsewhere, don't hesitate to request parts being split out if you think it doesn't fit here.
BTW, I think it looks larger than it actually is due to the addition of a new dependency.

Adding corneliusweig to owners. He has been consistently helping both with krew and krew-index repositories in terms of: - developing plugins himself - taking a stab at krew machinery with large scale code refactors - adding integration test suite to the project - adding more validation and test cases - increasing developer documentation Some of his notable work: - kubernetes-sigs#195 - kubernetes-sigs#183 - kubernetes-sigs#191 - kubernetes-sigs#201 - kubernetes-sigs#202 - kubernetes-sigs#203 - kubernetes-sigs#208 He is familiar with the codebase enough to officially review and approve code. Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>

Adding corneliusweig to owners. He has been consistently helping both with krew and krew-index repositories in terms of: - developing plugins himself - taking a stab at krew machinery with large scale code refactors - adding integration test suite to the project - adding more validation and test cases - increasing developer documentation Some of his notable work: - #195 - #183 - #191 - #201 - #202 - #203 - #208 He is familiar with the codebase enough to officially review and approve code. Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>

ahmetb · 2019-06-18T05:03:25Z

This is still on my list, I'll get to reviewing hopefully soon.

ahmetb · 2019-06-19T04:28:18Z

Before reading the code, I'd like to discuss some points (and this is precisely why we should design it before coding it):

There is a new manifest kind "Receipt" which contains the details of the install, such as

date of first install

date of last update

plugin manifest of last install/update

Why do we need a new kind? Can't we just copy the manifest over? (This simplifies a lot.) I don't think there's any point of storing these dates.

The krew info command now looks for index manifests, and receipts. If a receipt is found, the shown information is slightly augmented.

I don't know if users can understand this easily. This is a bit "magic" behavior, and will get more complicated when we support multiple indexes and there's name collisions. We should probably look at what other package managers do to show "more info" about a package, and does the behavior change when the package is installed.

plugin instances are now passed around as pointers instead of full copies. Was there a particular reason for not using pointers?

I think mostly because there aren't a lot of good reasons to use pointers.

We aren't worried about little memory overhead
copying reduces the effects of accidentally modifying the Plugin struct instance in a particular function
there's no way to get panics due to nil, if it's not a pointer.

So it doesn't add anything, consider undoing this.

a new dependency on google/go-cmp was added and vendored. If you don't like this, I can also do this manually, but the test assertions won't be so pretty. Otherwise, this library should be used in a few more places.

I think new test dependencies are ok, our tests have been pretty lean so far, but we can change that.

corneliusweig · 2019-06-20T10:57:22Z

First of all, thanks for your feedback. I think it's absolutely necessary to do that even if there is a working first revistion.
So there seem to be 3 areas that need to be addressed:

Receipt type
passing plugins as object vs. pointer to object
test dependencies
magic krew info

does not need more discussion.
From what I see in other k8s code, it is much more common to pass pointers to objects. However, this is independent of the main point of this PR. So I'll open an issue where we can discuss the pros and cons and drop the respective commit from here.

So I'll only focus on point 1 and 4:

1

The main reason why I introduced the new type was that it gives us more flexibility in the future. For example, when we have multiple indices and somebody introduces a name clash, we can record what the source index for the specific plugin was. The dates which I put into the receipt right now is just a nice-to-have but definitely not important. And there are for sure other important use-cases for saving metadata about the install process in the receipt, which we can't think of right now.
So I'm a bit torn apart between ease of maintenance (no extra type) and flexibility for future use-cases (extra type). I do admit that a single type simplifies the logic somewhat.

4

I don't agree that this appears magic to users. It should be quite obvious that info does not print the install date for a plugin which is not installed. On the other hand, it makes perfect sense to show that for an installed plugin. However, it may be a good idea to show all possible output fields and leave them blank if they don't apply.

ahmetb · 2019-06-20T18:24:40Z

I do admit that a single type simplifies the logic somewhat.

I'd like to hammer on this, it really does simplify, and I don't think there's much value of showing dates of installation. Most other package managers don't do this either.

The TOP reason to save install receipt is the ability to uninstall plugins that don't exit/never existed in the index.

I'm not entirely opposing the idea. It's just low-priority to me. We can introduce a status: field (k8s convention) in the Plugin object if you really want to save some dates.

I don't agree that this appears magic to users. It should be quite obvious that info does not print the install date for a plugin which is not installed. On the other hand, it makes perfect sense to show that for an installed plugin. However, it may be a good idea to show all possible output fields and leave them blank if they don't apply.

To me krew info showing Status: INSTALLED (vX.Y.Z) vs Status: NOT INSTALLED is good enough and P1, anything else is secondary. Would you agree?

corneliusweig · 2019-06-21T20:06:02Z

Yeah, let's keep it simple. If we really must add install metadata, we can add fields to the plugin manifest which should not be populated by plugin authors but only by krew. I don't think that install dates are important enough to mandate for a manifest change, so let's drop that.

pkg/index/platform_test.go

Saving receipts is less prone to failure than the whole plugin installation. When swapped, krew will have an inconsistent state in fewer occasions.

…r when plugin does not exist

ahmetb · 2019-06-29T21:43:09Z

I've just tested this, it's mostly good to go.

I have several questions, I'm curious when do you want to address them, as they're breaking changes:

Currently krew list doesn't rely on receipts to see what's installed, it scans $KREW_ROOT/bin directory.
Currently krew uninstall also scans $KREW_ROOT/bin directory to see if a plugin is installed. Should it rather look for a receipt?
Currently krew upgrade also scans $KREW_ROOT/bin directory which plugins are installed. It similarly infers the installed version of the plugin by looking at the symlink path of the executable instead of the receipt.

Should we do this breaking change later?

Seems like this PR does what it says ("save install receipt") as it saves the receipt on install and upgrades.

pkg/installation/install.go

corneliusweig · 2019-06-30T12:05:54Z

@ahmetb Thanks for another close look at this PR.

Storing the install receipt for upgrades should be part of this PR.

The other points you raised are all valid. But they are pretty independent, so I would rather not include them here. As stated in the first PR comment, this requires some migration of the krew folder, because we need to store the receipts for already installed plugins. This problem needs to be solved first before we can migrate the logic for list, uninstall, and upgrade.

EDIT: I opened #232 to discuss the migration plan.

ahmetb · 2019-07-02T05:21:27Z

/lgtm
/approve
Thanks for staying on top of this! 🍸

k8s-ci-robot · 2019-07-02T05:21:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahmetb, corneliusweig

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahmetb,corneliusweig]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Looks like we didn't address the code review comment at kubernetes-sigs#195 (comment). Copying the conversation from there: > This is better because: > > - actually installing a plugin (extract files, symlink binary) but then failing > to store the receipt is okay (at most you'd be leaking some files) > - claiming the plugin is installed (i.e. receipt exists) while it's not > installed is not okay. > - tens of things can go wrong in `install()`, much fewer things can go wrong > in `receipt.Store()`, so makes sense to do it last. Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>

* pkg/installation: reorder receipt save & install() Looks like we didn't address the code review comment at #195 (comment). Copying the conversation from there: > This is better because: > > - actually installing a plugin (extract files, symlink binary) but then failing > to store the receipt is okay (at most you'd be leaking some files) > - claiming the plugin is installed (i.e. receipt exists) while it's not > installed is not okay. > - tens of things can go wrong in `install()`, much fewer things can go wrong > in `receipt.Store()`, so makes sense to do it last. Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com> * pkg/installation: reorder receipt saving in uninstall() Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>

k8s-ci-robot requested review from ahmetb and juanvallejo June 2, 2019 23:49

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 2, 2019

k8s-ci-robot assigned ahmetb Jun 2, 2019

ahmetb mentioned this pull request Jun 3, 2019

Allow installation from remote manifests without downloading the bundle #193

Closed

corneliusweig mentioned this pull request Jun 3, 2019

Proposal: Use semver to determine plugin versions and upgrades #166

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 5, 2019

corneliusweig force-pushed the install-receipt branch from 7e42a3b to bb0f363 Compare June 5, 2019 23:08

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 5, 2019

corneliusweig force-pushed the install-receipt branch from bb0f363 to 42d601c Compare June 10, 2019 21:25

ahmetb mentioned this pull request Jun 14, 2019

Make corneliusweig a maintainer #214

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 18, 2019

corneliusweig force-pushed the install-receipt branch from 42d601c to 89486b3 Compare June 18, 2019 18:40

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 18, 2019

corneliusweig mentioned this pull request Jun 19, 2019

REQUEST: New membership for corneliusweig kubernetes/org#929

Closed

6 tasks

corneliusweig force-pushed the install-receipt branch from 89486b3 to eb428ac Compare June 21, 2019 22:33

corneliusweig commented Jun 21, 2019

View reviewed changes

pkg/index/platform_test.go Outdated Show resolved Hide resolved

corneliusweig added 10 commits June 26, 2019 21:54

Remove plugin receipt on uninstall

10038df

Load plugin manifest from receipts directory or the index

b62c54e

Do not expect specific folder layout in LoadPluginFileFromFS

419da84

Fix godoc for environment paths

52ec800

Review comments

361faa9

Swap plugin install and saving receipts

7e7cf0b

Saving receipts is less prone to failure than the whole plugin installation. When swapped, krew will have an inconsistent state in fewer occasions.

Add test to ensure LoadManifestFromReceiptOrIndex returns ENOENT erro…

275ab13

…r when plugin does not exist

Put receipts in folder 'receipts' instead of 'receipts/krew-index'

7ccb27b

Fix linter issues

68ea9a4

Rename ReceiptsPath -> InstallReceiptPath

6d78644

corneliusweig force-pushed the install-receipt branch from 055aa1c to 6d78644 Compare June 26, 2019 19:55

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 26, 2019

ahmetb reviewed Jun 29, 2019

View reviewed changes

pkg/installation/install.go Show resolved Hide resolved

corneliusweig mentioned this pull request Jun 30, 2019

Migrate krew home to add install receipts for installed plugins #232

Closed

Save the new install receipt when upgrading a plugin

ff5b7e6

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 30, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 2, 2019

k8s-ci-robot merged commit 1831b1b into kubernetes-sigs:master Jul 2, 2019

ahmetb mentioned this pull request Jul 2, 2019

pkg/installation: reorder receipt save & install() #237

Merged

corneliusweig deleted the install-receipt branch July 2, 2019 06:52

corneliusweig mentioned this pull request Jul 2, 2019

Move to go modules (vgo) #235

Merged

This was referenced Jul 17, 2019

Eliminate different yaml parsing/serializing dependencies #265

Closed

Refactor all commands to determine they use plugin receipts #277

Closed

corneliusweig mentioned this pull request Feb 12, 2020

Upgrade failed; but receipt still written when symlinking fails #502

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Save install receipt #195

Save install receipt #195

corneliusweig commented Jun 2, 2019

corneliusweig commented Jun 2, 2019

codecov-io commented Jun 2, 2019 •

edited

Loading

ahmetb commented Jun 3, 2019

ahmetb commented Jun 3, 2019

corneliusweig commented Jun 3, 2019 •

edited

Loading

ahmetb commented Jun 18, 2019

ahmetb commented Jun 19, 2019 •

edited

Loading

corneliusweig commented Jun 20, 2019 •

edited

Loading

ahmetb commented Jun 20, 2019

corneliusweig commented Jun 21, 2019

ahmetb commented Jun 29, 2019

corneliusweig commented Jun 30, 2019 •

edited

Loading

ahmetb commented Jul 2, 2019

k8s-ci-robot commented Jul 2, 2019

Save install receipt #195

Save install receipt #195

Conversation

corneliusweig commented Jun 2, 2019

Summary of changes

Todos

corneliusweig commented Jun 2, 2019

codecov-io commented Jun 2, 2019 • edited Loading

Codecov Report

ahmetb commented Jun 3, 2019

ahmetb commented Jun 3, 2019

corneliusweig commented Jun 3, 2019 • edited Loading

ahmetb commented Jun 18, 2019

ahmetb commented Jun 19, 2019 • edited Loading

corneliusweig commented Jun 20, 2019 • edited Loading

1

4

ahmetb commented Jun 20, 2019

corneliusweig commented Jun 21, 2019

ahmetb commented Jun 29, 2019

corneliusweig commented Jun 30, 2019 • edited Loading

ahmetb commented Jul 2, 2019

k8s-ci-robot commented Jul 2, 2019

codecov-io commented Jun 2, 2019 •

edited

Loading

corneliusweig commented Jun 3, 2019 •

edited

Loading

ahmetb commented Jun 19, 2019 •

edited

Loading

corneliusweig commented Jun 20, 2019 •

edited

Loading

corneliusweig commented Jun 30, 2019 •

edited

Loading