Support push/pull images from Kaniko image builder

[omusaev]

I have successfully installed trow on my kube cluster, built an image externally and pushed it to the registry. But when I try to pull it from the registry (I'm using kaniko to build images inside the cluster) I get this error:

Downloading base image trow.kube-public/blablabla
error building image: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+json", see google/go-containerregistry#377

[amouat]

Thanks Oleg. Could you share the image you're pushing/pulling?

I suspect Trow is failing to parse the manifest as v2 for some reason.

[amouat]

Just to clarify, I suspect what's happening is Trow is being sent a v2 manifest but is rejecting it for some reason and falling back to v1. I can't fix this issue with an example image or manifest.

[omusaev]

Sorry for the delay.
A few more details:

• I tried pushing and pulling the same image from the official registry image (https://hub.docker.com/_/registry). Works fine.
• I'm using kaniko to build images based on images from the registry, so the error is from kaniko, but I think it doesn't really matter because it's getting the v1 manifest for some reason
• here are a few screenshots (not sure if it helps, though :) ):
  push from outside the cluster
  
  docker version
  
  kaniko log (inside the cluster)
  

Thanks!

[amouat]

So the base image being pulled by kaniko in the last log is the same image you pushed in the first step (the greyed out text is identical)?

Interesting. I think I'll have to test with kaniko to find this.

[omusaev]

Yes, it was the same image, sorry for the grey areas :)

[omusaev]

New info:
I have deployed helm chart for docker registry (https://github.com/helm/charts/tree/master/stable/docker-registry) and configured the certs manually. Everything works fine (same image, same kaniko build, etc.).

So, without deep researching, I would say that something wrong with the registry trow uses :)

[amouat]

Trow is a registry :)

And I'm pretty sure it is a bug in Trow. I think it's failing to parse the image manifest and falling back to v1.

To help me debug this it would be great if you could share an image that causes the error.

[amouat]

I'm going to rename this issue as after looking at the image and playing with Kaniko, it seems the main problem is that Trow doesn't respond in the expected way to some API call by kaniko. I'm not convinced there is a v1/v2 manifest problem (although that might also be a problem).

[amouat]

So, this was a weird one.

Kaniko behaves a bit strangely with pushes (the http logs don't make a lot of sense to me), and this has exposed a bug in some of Trow's underlying libraries. It seems Rocket ships with an old version of hyper, which under certain circumstances will cut off transfers. The mitigation is to turn off keep-alive, which I've done on master. Long-term this will be fixed when Rocket upgrades the version of hyper, or Trow moves to a different framework.

The code in master should now work correctly for kaniko and shortly the Docker Hub containersol/trow:default and latest images will contain the fix.