Make files in `spec.platforms[]` optional #261

corneliusweig · 2019-07-16T10:24:20Z

Many plugins use the pattern to copy all files from the archive into the installation directory. Those plugins always have to specify a file copy operation such as

files:
- from: "*"
  to: "."

This PR makes the files field completely optional and defaults to the above copy specifier. As discussed in #135, the default copy spec may install too many files. The plugin developer docs therefore explicitly show such an antipattern example.

Close #135

codecov-io · 2019-07-16T10:27:47Z

Codecov Report

Merging #261 into master will increase coverage by 0.67%.
The diff coverage is 87.5%.

@@            Coverage Diff             @@
##           master     #261      +/-   ##
==========================================
+ Coverage   54.98%   55.66%   +0.67%     
==========================================
  Files          19       19              
  Lines         902      918      +16     
==========================================
+ Hits          496      511      +15     
- Misses        354      355       +1     
  Partials       52       52

Impacted Files	Coverage Δ
pkg/installation/install.go	`38.98% <80%> (+1.81%)`	⬆️
pkg/index/validation/validate.go	`94.59% <89.47%> (+0.94%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 43b29c3...3e520e1. Read the comment docs.

ahmetb · 2019-07-16T18:49:13Z

docs/DEVELOPER_GUIDE.md

+  the root of your plugin, use the default:
+
+  ```yaml
+      files: []  # same as [{from: "*", to: "."}]


Hmm. I think empty list should not be treated as unspecified (nil) field.

Empty means do nothing, which a valid operation, but it shouldn't be allowed since it does not make sense.

Isn't the differentiation between empty and nil a programming language subtlety? From a user's perspective both cases mean "no rules given". And here I think we are complicating things if we make that distinction.

I'll move this to main thread.

ahmetb · 2019-07-16T18:49:49Z

docs/DEVELOPER_GUIDE.md

+      files: []  # same as [{from: "*", to: "."}]
+  ```
+
+  The resulting installation directory would eventually contain:


This would copy out binaries for both platforms to the installation directory on user’s machine, despite only one of them will be used.

ahmetb · 2019-07-16T18:51:11Z

pkg/index/validate.go

@@ -90,8 +90,13 @@ func (p Platform) Validate() error {
 	if p.Bin == "" {
 		return errors.New("bin has to be set")
 	}
-	if len(p.Files) == 0 {
-		return errors.New("can't have a plugin without specifying file operations")
+	for _, fo := range p.Files {


Let's extract this to a new method that we can unit-test, just like ValidatePlatform above.

ahmetb · 2019-07-16T18:52:33Z

pkg/index/validate.go

@@ -90,8 +90,13 @@ func (p Platform) Validate() error {
 	if p.Bin == "" {
 		return errors.New("bin has to be set")
 	}
-	if len(p.Files) == 0 {
-		return errors.New("can't have a plugin without specifying file operations")
+	for _, fo := range p.Files {


Let’s add a check that verifies nil is ok for p.Files (you should make this Files field a pointer like *[]FileOperaton first I think to distinguish between nil and empty []) AND validate that empty is not allowed and should not be defaulted.

ahmetb · 2019-07-16T18:53:53Z

pkg/installation/util.go

-	return version, uri, p.Files, p.Bin, nil
+	files := p.Files
+	if len(files) == 0 {
+		files = []index.FileOperation{{From: "*", To: "."}}


worth adding a glog.V(4) here saying file operation not specified, assuming %v

ahmetb · 2019-07-16T18:55:40Z

pkg/installation/util_test.go

+			wantBin:     "kubectl-foo",
+			wantErr:     false,
+		}, {
+			name: "Find Matching Platform with default files",


...with FileOperations unspecified or ...defaulted

ahmetb · 2019-07-16T18:56:34Z

Overall looks good. I think we just need to distinguish clearly between unspecified (null) and empty ([]) while allowing one and failing on another.

ahmetb · 2019-07-16T23:34:44Z

Isn't the differentiation between empty and nil a programming language subtlety?

Sadly, no. This is independent of programming language.
In one case, you're specifying "I have no elements" in another, it's unspecified altogether.

More obvious examples of this can be seen in Kubernetes API design. For example, in Kubernetes NetworkPolicy API, specifying ingress: [] as an empty array explicitly causes a policy to be created allowing no traffic. However, not specifying this field doesn't have that impact (it makes the NetworkPolicy object not an ingress policy at all).

Let's play safe in this case and (1) distinguish nil and default on that (2) validate error on empty array. Here's another reason to play safe: We can start with defaulting on just nil today and failijng on [], if it works out we can expand the definition. However, if we default on both nil and [] (and realize it's wrong API design), we can't easily change the behavior without breaking the API.

I prefer to err on being more restrictive and being more lenient only if there's a need.

corneliusweig · 2019-07-18T06:55:13Z

Ok, that forward compatibility argument is convincing. I'll address the nil vs empty issue, but I'll need a few days due to other obligations.

ahmetb · 2019-07-22T20:13:06Z

Haha sorry I caused merge conflicts everywhere. Now that we have proper validation on Platform, and testutil.NewPlugin() utilities, this PR should be easier to code.

corneliusweig · 2019-07-22T21:48:03Z

/hold

corneliusweig · 2019-07-22T22:03:21Z

/hold cancel

corneliusweig · 2019-07-22T22:09:32Z

Rebase is finally done.

I added a new validation case which forbids empty file operations now. The difference between empty and unspecified is quite subtle though (this comes from our yaml library):

      ...
      bin: foo-amd64
      files:  # present, but no values

is equivalent to

      ...
      bin: foo-amd64
#    files:  <- not present

Whereas this triggers the error case:

      ...
      bin: foo-amd64
      files: []

ahmetb · 2019-07-22T22:18:26Z

I think it's ok to treat unspecified and files: (with no values) the same.
On the other hand, files: [] explicitly says I don't want any values in there.

ahmetb · 2019-07-22T22:19:28Z

docs/DEVELOPER_GUIDE.md

-└── krew-foo-windows.exe
-```
+- To copy all files in the `bin/` directory of the extracted archive to
+  the root of your plugin, use the default:


hmm, to use the default, maybe they shouldn't need to specify files:?

Well, my thinking was to keep the two examples as similar as possible. But I agree that this looks odd. I changed the wording.

ahmetb · 2019-07-22T22:20:55Z

pkg/index/validation/validate.go

+	}
+	for _, op := range fops {
+		if op.From == "" {
+			return errors.New("from field has to be set")


can we say "from" and "to" (below) to distinguish field names from the rest of the error message. especially below it says to ... to be set so it's a bit confusing.

I enclosed field names in backticks. To be consistent, I did the same for other error messages.

pkg/index/validation/validate.go

ahmetb · 2019-07-22T22:22:48Z

pkg/index/validation/validate_test.go

@@ -190,8 +190,8 @@ func TestValidatePlatform(t *testing.T) {
 			wantErr:  true,
 		},
 		{
-			name:     "no file operations",
-			platform: testutil.NewPlatform().WithFiles(nil).V(),
+			name:     "empty file operations",


can you also add WithFiles(nil) case (wantErr:false)?

Those test cases are covered by Test_validateFiles. I only included this error case in this test, to check if files are validated at all. Do you think this is not enough?

ahmetb · 2019-07-22T22:24:23Z

pkg/installation/util.go

 	uri = p.URI
 	sha256sum = p.Sha256
+
+	files := p.Files
+	if len(files) == 0 {


I'm removing this method (getDownloadTarget) soon as part of #293.

I think we should consider a method like func applyDefaults(*index.Platform) to achieve defaulting on the FileOperations. Are you open to such a method, that's easily testable etc?

In principle, I like the idea. However, I didn't find a good spot for this. The best place is close to validation, but there we pass in a copy of the plugin object which loses any mutations. And I didn't want to put this into ReadPluginFile.

So I ended up, putting it in moveToInstallDir. Not ideal, though..

I think that's why we should call applyDefaults(&plugin) right before we do moveToInstallDir. Then we can do moveToInstallDir(..., plugin.Files).

Don't move inside moveToInstallDir as the name indicates, it's not expected to do defaulting.

Alright, I moved the logic one level up into install. Also, I added a unit test for applyDefaults.

If unspecified, default to `{From: "*", To: "."}`. When `files` is given, but empty, an error is produced instead. Also validate that files specification has both fields set.

- Improve readability of error messages.

pkg/installation/move.go

ahmetb · 2019-07-24T21:07:34Z

pkg/installation/install_test.go

+		},
+		{
+			name:     "no defaults for other fields",
+			platform: testutil.NewPlatform().WithBin("").WithOS("").WithSelector(nil).WithSHA256("").WithURI("").V(),


oh boy this test is gonna be a problem in the future.
because we will keep adding defaults to testutil.NewPlatform(), and we will keep adding new WithXxx.
let's keep it until it breaks.

ahmetb · 2019-07-24T21:08:42Z

/lgtm
/approve

k8s-ci-robot · 2019-07-24T21:08:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahmetb, corneliusweig

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahmetb,corneliusweig]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 16, 2019

k8s-ci-robot requested review from ahmetb and juanvallejo July 16, 2019 10:24

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 16, 2019

ahmetb reviewed Jul 16, 2019

View reviewed changes

corneliusweig force-pushed the w/optional-platform-spec branch from b9a58c3 to dfc1a2e Compare July 22, 2019 21:45

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 22, 2019

corneliusweig force-pushed the w/optional-platform-spec branch from dfc1a2e to 946b1bf Compare July 22, 2019 22:02

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 22, 2019

ahmetb requested changes Jul 22, 2019

View reviewed changes

Allow files specification to be unspecified in plugin manifests

79173d5

If unspecified, default to `{From: "*", To: "."}`. When `files` is given, but empty, an error is produced instead. Also validate that files specification has both fields set.

corneliusweig force-pushed the w/optional-platform-spec branch from 946b1bf to e9e84d0 Compare July 23, 2019 21:44

Review comments

f357807

- Improve readability of error messages.

corneliusweig force-pushed the w/optional-platform-spec branch from e9e84d0 to f357807 Compare July 23, 2019 21:45

ahmetb reviewed Jul 23, 2019

View reviewed changes

pkg/installation/move.go Outdated Show resolved Hide resolved

Move defaults for file operations into install method

3e520e1

ahmetb reviewed Jul 24, 2019

View reviewed changes

k8s-ci-robot assigned ahmetb Jul 24, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 24, 2019

k8s-ci-robot merged commit 8867a94 into kubernetes-sigs:master Jul 24, 2019

corneliusweig deleted the w/optional-platform-spec branch July 24, 2019 21:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make files in `spec.platforms[]` optional #261

Make files in `spec.platforms[]` optional #261

corneliusweig commented Jul 16, 2019

codecov-io commented Jul 16, 2019 •

edited

Loading

ahmetb Jul 16, 2019

corneliusweig Jul 16, 2019 •

edited

Loading

ahmetb Jul 16, 2019

ahmetb Jul 16, 2019 •

edited

Loading

ahmetb Jul 16, 2019

ahmetb Jul 16, 2019

ahmetb Jul 16, 2019

ahmetb Jul 16, 2019

ahmetb commented Jul 16, 2019

ahmetb commented Jul 16, 2019 •

edited

Loading

corneliusweig commented Jul 18, 2019

ahmetb commented Jul 22, 2019

corneliusweig commented Jul 22, 2019

corneliusweig commented Jul 22, 2019

corneliusweig commented Jul 22, 2019

ahmetb commented Jul 22, 2019

ahmetb Jul 22, 2019

corneliusweig Jul 23, 2019

ahmetb Jul 22, 2019

corneliusweig Jul 23, 2019

ahmetb Jul 22, 2019

corneliusweig Jul 23, 2019

ahmetb Jul 22, 2019

corneliusweig Jul 23, 2019

ahmetb Jul 23, 2019

corneliusweig Jul 24, 2019

ahmetb Jul 24, 2019

ahmetb commented Jul 24, 2019

k8s-ci-robot commented Jul 24, 2019

Make files in spec.platforms[] optional #261

Make files in spec.platforms[] optional #261

Conversation

corneliusweig commented Jul 16, 2019

codecov-io commented Jul 16, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

corneliusweig Jul 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmetb Jul 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmetb commented Jul 16, 2019

ahmetb commented Jul 16, 2019 • edited Loading

corneliusweig commented Jul 18, 2019

ahmetb commented Jul 22, 2019

corneliusweig commented Jul 22, 2019

corneliusweig commented Jul 22, 2019

corneliusweig commented Jul 22, 2019

ahmetb commented Jul 22, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmetb commented Jul 24, 2019

k8s-ci-robot commented Jul 24, 2019

Make files in `spec.platforms[]` optional #261

Make files in `spec.platforms[]` optional #261

codecov-io commented Jul 16, 2019 •

edited

Loading

corneliusweig Jul 16, 2019 •

edited

Loading

ahmetb Jul 16, 2019 •

edited

Loading

ahmetb commented Jul 16, 2019 •

edited

Loading