Add 'umoci copy' subcommand

[jdolitsky]

• Support for pulling oci:// URL into local store
• Support for pushing from local store to registry
• Downloads blobs only when not found in local store
• Currently only supports OCI v1 manifest media type
• Does not support docker:// URLs or Docker manifests
• Does not yet support chunked blob uploading

Resolves #345

---

Demo

Get binaries for zot and umoci (modified), add to PATH

mkdir demo && cd demo/
mkdir bin/
git clone --depth 1 --branch v1.1.12 https://github.com/anuvu/zot.git
(cd zot && make binary && mv bin/zot ../bin/)
git clone --depth 1 --branch copy-subcommand https://github.com/bloodorangeio/umoci.git
(cd umoci && make && mv ./umoci ../bin/)
export PATH="${PWD}/bin:${PATH}"

Start zot server (port 8080)

echo "{\"storage\":{\"rootDirectory\":\"${PWD}/storage\"}}" > zot.yaml
zot serve ./zot.yaml

Use skopeo to fetch Docker manifest, convert to OCI, and push to zot

skopeo copy -f oci --dest-tls-verify=false \
  docker://opensuse/amd64:42.2 docker://127.0.0.1:8080/opensuse/amd64:42.2

umoci copy: remote to local

umoci --log=debug copy --plain-http oci://localhost:8080/opensuse/amd64:42.2 opensuse:42.2

output:

   • Registry address: http://localhost:8080, namespace: opensuse/amd64, tag: 42.2
   • Checking if manifest available in registry
   • Registry reports manifest with digest sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • Downloading manifest from registry
   • actual manifest digest matches expected digest (sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed)
   • blob saved to local store, digest: sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed, size: 349
   • Manifest layer list contains 1 item(s)
   • Copying layer 1/1 with digest sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8
   • actual layer digest matches expected digest (sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8)
   • blob saved to local store, digest: sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8, size: 48850323
   • Copying config with digest sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff
   • actual config digest matches expected digest (sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff)
   • blob saved to local store, digest: sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff, size: 720
   • Saving reference '42.2' to index in opensuse

inspect CAS:

$ tree opensuse
opensuse
├── blobs
│   └── sha256
│       ├── 2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff
│       ├── 7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
│       └── f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8
├── index.json
└── oci-layout
$ cat opensuse/index.json | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed",
      "size": 349,
      "annotations": {
        "org.opencontainers.image.ref.name": "42.2"
      }
    }
  ]
}

umoci copy: local to remote

umoci --log=debug copy --plain-http opensuse:42.2 oci://localhost:8080/othersuse/amd64:othertag

output:

   • Registry address: http://localhost:8080, namespace: othersusex/amd64, tag: othertag
   • -> ws.recurse             digest=sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • <- ws.recurse             digest=sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • casext.ResolveReference(42.2) got these descriptors refs=[{[{application/vnd.oci.image.manifest.v1+json sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed 349 [] map[org.opencontainers.image.ref.name:42.2] <nil>}]}]
   • Reference '42.2' found in index, points to manifest sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • Manifest successfully loaded from local store
   • Manifest layer list contains 1 item(s)
   • Uploading layer 1/1 with digest sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8 from local store
   • Uploading config sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff
   • Successfully copied to remote localhost:8080

If you add the --trace-requests flag, raw HTTP output will be displayed:

umoci --log=debug copy --plain-http --trace-requests oci://localhost:8080/opensuse/amd64:42.2 opensuse:42.2

output:

   • Registry address: http://localhost:8080, namespace: opensuse/amd64, tag: 42.2
   • Checking if manifest available in registry
2021/02/08 15:58:15.750531 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
HEAD  /v2/opensuse/amd64/manifests/42.2  HTTP/1.1
HOST   : localhost:8080
HEADERS:
	Accept: application/vnd.oci.image.manifest.v1+json
	User-Agent: umoci 0.4.6+dev~gitd2f6056482314a44724ac2151622843881a18c45
BODY   :
***** NO CONTENT *****
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200 OK
RECEIVED AT  : 2021-02-08T15:58:15.749825-05:00
TIME DURATION: 4.758014ms
HEADERS      :
	Content-Length: 0
	Date: Mon, 08 Feb 2021 20:58:15 GMT
	Docker-Content-Digest: sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
BODY         :

==============================================================================
   • Registry reports manifest with digest sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
 ...

[tych0]

As far as I can tell, resp.Body() here is: https://pkg.go.dev/github.com/go-resty/resty/v2#Response.Body which will end up reading the whole layer into memory. Seems like you want RawBody() here?

https://pkg.go.dev/github.com/go-resty/resty/v2#Response.RawBody

[tych0]

Oh, derp. I meant to put this on the layer one below. For this one I think it's unavoidable: you definitely want to read the manifest bytes into memory.

[tych0]

One small nit on this, but otherwise it looks like it might be useful to us. How close do you feel this is to merging?

[cyphar]

@tych0

One small nit on this, but otherwise it looks like it might be useful to us. How close do you feel this is to merging?

I've been chatting to @jdolitsky about this for a little bit -- my current plan is for me to rework this PR so that it's implemented as a cas.Engine backend once I get back from vacation next week.

[tych0]

my current plan is for me to rework this PR so that it's implemented as a cas.Engine backend once I get back from vacation next week.

Excellent, thanks!

[tych0]

What's the status of this? I'd like to get some features into containers/image soon-ish, but it would be better to have them here, if possible :)

[cyphar]

I haven't touched this in a while. I can try to revive my PoC using the CAS API -- what exactly did you want to get into containers/image? Switching to umoci copy would probably be somewhat non-trivial even if I did get it working soon.

[tych0]

Mostly I don't want it to compress things automatically that aren't gzip :)

[cyphar]

I feel the the only slight complication is going to be chunked uploads and downloads which this PoC didn't implement. I started working on a PoC of this, it shouldn't be that bad on paper but implementing a generic umoci copy API might be slightly tricky and I think the CasEngineExt API needs to be reworked somewhat in order to be able to handle arbitrary CAS implementations to wrap).

[jdolitsky]

@cyphar - shall I close this for now?