Image relocation #671

glyn · 2019-03-20T11:30:30Z

See duffle relocate -h for documentation.

Adds the originalImage and originalDigest fields to images in both the image map and invocation images on the assumption that cnabio/cnab-spec#157 will be implemented.

Deferred:

relocation of invocation images (duffle relocate to relocate invocation images #706)
relocation of thick bundles (duffle relocate for thick bundles #705)

Fixes #668

cmd/duffle/relocate.go

radu-matei · 2019-04-11T16:36:24Z

The linter seems to be unhappy:

golangci-lint run --config ./golangci.yml
cmd/duffle/root.go:50: File is not `goimports`-ed (goimports)
        newRelocateCmd(outLog),
pkg/bundle/bundle.go:68: File is not `goimports`-ed (goimports)
	ImageType      string         `json:"imageType" mapstructure:"imageType"`
	Image          string         `json:"image" mapstructure:"image"`
	OriginalImage  string         `json:"originalImage" mapstructure:"originalImage"`
	Digest         string         `json:"digest,omitempty" mapstructure:"digest"`
	Size           uint64         `json:"size,omitempty" mapstructure:"size"`
	Platform       *ImagePlatform `json:"platform,omitempty" mapstructure:"platform"`
	MediaType      string         `json:"mediaType,omitempty" mapstructure:"mediaType"`
make: *** [lint] Error 1
Makefile:74: recipe for target 'lint' failed

glyn · 2019-04-11T16:39:58Z

@radu-matei wrote:

The linter seems to be unhappy:

Oops. Fixed. Apologies.

pkg/bundle/bundle.go

radu-matei · 2019-04-15T14:50:33Z

cmd/duffle/relocate.go

+`
+
+type relocateCmd struct {
+	// args


We don't usually have comments to separate args, flags and other things here, but don't really have anything against them -- and might actually start using them in other places as well.

radu-matei · 2019-04-15T14:52:08Z

Gopkg.toml

@@ -53,3 +53,7 @@
 [[constraint]]
  name = "github.com/docker/go"
  version = "1.5.1-1"
+
+[[constraint]]


Looking at https://github.com/pivotal/image-relocation -- would it make sense to have an initial release we can pinpoint here?

Not necessarily blocking this PR though.

Possibly, although that would make most sense if go-containerregistry was also released, which it hasn't been so far. Let's defer for now.

@glyn - For helm3 the push/pull utilizes https://github.com/deislabs/oras which is based on containerd @bacongobbler do you think there could be a release of oras for CNAB if needed?

Thanks. The plan is to merge this PR with its implementation in terms of go-containerregistry. The implementation can be changed in the future as appropriate.

radu-matei · 2019-04-15T15:10:50Z

Passing an inexistent registry prefix errors out in a rather non-intuitive way:

$ duffle relocate cmd/duffle/testdata/relocate/bundle.json relocatedbundle-2 --repository-prefix example.com/user --input-bundle-is-file
Error: unrecognized HTTP status: 404 Not Found

silvin-lubecki · 2019-04-16T08:14:55Z

Hello Glynn, as we discussed in the issue, I think we should try to improve cnab-to-oci Fixup instead.

If I understand it correctly, the only blocking point you saw in cnab-to-oci is that we mount/copy all the images in the same repository, and in your case you need to push all the images to their "own" repository? My guess is that this can be easily implemented in the current Fixup command, it is just a matter of where we push the images, but all the logic of pushing is already there.
@simonferquel and I would be pleased to help you write this PR against cnab-to-oci 👍

A second argument for not merging this PR as-is, is that it adds a dependency on github.com/google/go-containerregistry, which I think will "duplicate" with the containerd library used to push/pull bundles to a registry. @radu-matei any thoughts on that point?

glyn · 2019-04-16T10:04:24Z

@silvin-lubecki Let's take the discussion over to the issue as I think it's more appropriate there.

glyn · 2019-04-16T13:16:42Z

cnabio/cnab-go#4 needs merging and we need to upgrade master of this repo to include that before we can resolve the pkg/bundle/bundle.go conflict in this PR.

glyn · 2019-04-16T13:30:13Z

Passing an inexistent registry prefix errors out in a rather non-intuitive way:

$ duffle relocate cmd/duffle/testdata/relocate/bundle.json relocatedbundle-2 --repository-prefix example.com/user --input-bundle-is-file
Error: unrecognized HTTP status: 404 Not Found

I'll address this in the image-relocation repo: vmware-archive/image-relocation#6

radu-matei · 2019-04-16T15:00:29Z

@silvin-lubecki

A few weeks back we brought this up in the CNAB meeting, and it was decided that, at least for now, we want to have both the push and relocate commands, as they serve different use cases, both valid for the end users.

My opinion is that we'd rather have this piece of functionality merged, and work to converging the two approaches after, than waiting for it -- I'm really interested in the approaches you raised, and am working on the push/pull functionality, so let's continue the discussion about using cnab-to-oci for this as well.

But in the meantime, I think we should have this merged -- thoughts?

silvin-lubecki · 2019-04-16T15:32:35Z

My opinion is that we'd rather have this piece of functionality merged, and work to converging the two approaches after, than waiting for it

I'm totally fine with this 👍. Thanks to @glyn we have a clear functional scope, and if it is merged now we can improve cnab-to-oci without rushing it.

radu-matei · 2019-04-16T19:28:09Z

Ok, happy we're all back on the same page here.

@glyn -- once the conflicts are fixed, I think this first iteration is good to go.
Thanks a lot!

glyn · 2019-04-17T09:24:30Z

Passing an inexistent registry prefix errors out in a rather non-intuitive way:
$ duffle relocate cmd/duffle/testdata/relocate/bundle.json relocatedbundle-2 --repository-prefix example.com/user --input-bundle-is-file
Error: unrecognized HTTP status: 404 Not Found
I'll address this in the image-relocation repo: pivotal/image-relocation#6

This is now fixed. The error is now:

Error: failed to write image example.com/user/deislabs-duffle-50aa5cc4ebb040ac696a9753d1695298@sha256:4d41eeb38fb14266b7c0461ef1ef0b2f8c05f41cd544987a259a9d92cdad2540: unrecognized HTTP status: 404 Not Found

cmd/duffle/relocate.go

silvin-lubecki · 2019-04-17T09:45:49Z

cmd/duffle/relocate.go

+		repositoryValue := cmd.Flag(flagName).Value.String()
+
+		if strings.HasSuffix(repositoryValue, "/") || strings.Contains(repositoryValue, "//") {
+			return fmt.Errorf("invalid repository: %s", repositoryValue)


As you have here a clear error case, maybe you can improve the error message and tell what is the exact issue in the repository name?

silvin-lubecki · 2019-04-17T09:46:00Z

cmd/duffle/relocate.go

+		for i, part := range strings.Split(repositoryValue, "/") {
+			if i != 0 {
+				if strings.ContainsAny(part, ":@\" ") {
+					return fmt.Errorf("invalid repository: %s", repositoryValue)


silvin-lubecki · 2019-04-17T09:54:02Z

cmd/duffle/relocate.go

+		PreRunE: func(cmd *cobra.Command, args []string) error {
+			// validate --repository-prefix if it is set, otherwise allow flag omission to be diagnosed as such
+			if cmd.Flags().Changed("repository-prefix") {
+				if err := flagValidRepository("repository-prefix")(cmd); err != nil {


Does this function really need to take this string argument and return a functor? As it is only used once, it can be simplified. You can also always check the repository prefix value set:

PreRunE: func(cmd *cobra.Command, args []string) error { if err := validateRepository(relocate. repoPrefix); err != nil{ return err }

with bellow, replacing func flagValidRepository(flagName string) flagsValidator {:

func validateRepository(repository string) error { if strings.HasSuffix(repository, "/") || strings.Contains(repository, "//") { return fmt.Errorf(...) ...

I got rid of the functor as suggested. (Functors made more sense in the project where this code originated since there was a lot of flag validation going on.)

Note that PreRunE deliberately falls through when the repository prefix is not specified so that we get the cobra error message for a missing required flag (for consistency). I've changed the comment to make this clearer.

glyn · 2019-04-17T11:01:57Z

@silvin-lubecki Thanks for your comments. Changes force pushed.

cmd/duffle/relocate_test.go

See `duffle relocate -h` for documentation. Adds the originalImage field to images in both the image map and invocation images. Fixes cnabio#668

silvin-lubecki

Thanks @glyn, the overall code looks good to me 👍

radu-matei

LGTM.
Thanks a lot for your patience here, @glyn -- this looks great for a first iteration.

glyn force-pushed the 668-image-relocation branch 4 times, most recently from 2ba0c67 to 0b8cb55 Compare March 25, 2019 12:34

radu-matei reviewed Mar 26, 2019

View reviewed changes

cmd/duffle/relocate.go Outdated Show resolved Hide resolved

glyn force-pushed the 668-image-relocation branch from 0b8cb55 to 9d631d7 Compare March 26, 2019 13:28

glyn force-pushed the 668-image-relocation branch 8 times, most recently from 758427e to dbad895 Compare April 11, 2019 10:05

glyn changed the title ~~[WIP] Image relocation~~ Image relocation Apr 11, 2019

glyn force-pushed the 668-image-relocation branch from dbad895 to 1a328c5 Compare April 11, 2019 10:13

glyn marked this pull request as ready for review April 11, 2019 10:25

silvin-lubecki reviewed Apr 11, 2019

View reviewed changes

cmd/duffle/relocate.go Outdated Show resolved Hide resolved

glyn mentioned this pull request Apr 11, 2019

Exploit common code to preserve tags and digests in duffle relocate #708

Closed

glyn force-pushed the 668-image-relocation branch 3 times, most recently from 60e5416 to 59432c8 Compare April 11, 2019 16:33

glyn force-pushed the 668-image-relocation branch from 59432c8 to 912bea9 Compare April 11, 2019 16:39

glyn force-pushed the 668-image-relocation branch from 912bea9 to 44b874b Compare April 11, 2019 16:47

radu-matei reviewed Apr 15, 2019

View reviewed changes

pkg/bundle/bundle.go Outdated Show resolved Hide resolved

radu-matei reviewed Apr 15, 2019

View reviewed changes

glyn mentioned this pull request Apr 16, 2019

Support image relocation for docker and oci image types #668

Closed

This was referenced Apr 16, 2019

InvocationImage support of relocations/thick bundle cnabio/cnab-spec#162

Closed

Thick bundle: clarify how invocation images are "air-gapped-aware" cnabio/cnab-spec#142

Closed

glyn force-pushed the 668-image-relocation branch 2 times, most recently from f0d1b64 to 8e8c3c2 Compare April 17, 2019 09:18