cluster up fails to push to the registry on Docker for Mac CE 17.12.0 and newer

[csrwng]

Docker for Mac CE 17.12.0 introduced an internal proxy for Docker that prevents pushes into the local OpenShift registry when running a cluster with cluster up

You also get a warning when starting cluster up like:

WARNING: An HTTP proxy (docker.for.mac.http.internal:3128) is configured for the Docker daemon, but you did not specify one for cluster up
WARNING: An HTTPS proxy (docker.for.mac.http.internal:3129) is configured for the Docker daemon, but you did not specify one for cluster up
WARNING: A proxy is configured for Docker, however 172.30.1.1 is not included in its NO_PROXY list.
   172.30.1.1 needs to be included in the Docker daemon's NO_PROXY environment variable so pushes to the local OpenShift registry can succeed.

For now, the workaround is to downgrade Docker to a version prior to 17.12.0. You can find download links to previous releases in their release note page:
https://docs.docker.com/docker-for-mac/release-notes/

Open Issues:
docker/for-mac#2506
docker/for-mac#2470
docker/for-mac#2467
docker/for-mac#2386

[csrwng]

@mfojtik @deads2k
/cc @mpbarrett

[mfojtik]

I would hold on fixing this until @deads2k finish the internal refactoring of cluster up.

[riuvshin]

same here, is there any other workaround for this? I don't want to downgrade...
adding 172.30.1.1 to no_proxy in docker prefs does not help
docker Version 18.02.0-ce-mac53 (22617)

[alberttwong]

I haven't be able to find a work around.. No proxy doesn't seem to work.

[mikz]

Docker Community Edition 18.03.0-ce-mac59 2018-03-26
Fix for the HTTP/S transparent proxy when using “localhost” names (e.g. host.docker.internal).

This does not fix the issue. So don't upgrade.

[cmcconnell1]

Confirming that I'm still seeing this with
2018-04-30 18.03.1-ce-mac65

[Valien]

So decided to try this again and still broken with docker version 18.03.1-ce. At least it's nice to know it's a verified issue/bug. Bummed there is no resolution besides using an older version of docker though.

[cmcconnell1]

Apologies, I previously automated workarounds for our team shortly after my posts above and then I forgot about this issue. I'm new to Minishift/Openshift (coming from deis workflow && looking at OS Origin to provide a solid PaaS solution), and not sure if this would work for everyone.

TL;DR:
Here is my initial POC (with docker for mac minishift workaround) for a local minishift cluster:
minishift-local-poc
Its automated and allows our distributed teams to quickly config and deploy our platform within a few minutes.

Details

• For our use case, we're working locally with minishift, pulling from remote private AWS ECR repos, tagging and pushing our images to the remote (Openshift / Minishift registry--AFAIK, minishift still considers this to be a remote) docker registry.

• I had found another blog post from last year where they needed to use secrets else they were getting the permission denied errors (as was I) so set those up, and automated that process as well. I'm not sure if that's the correct "Openshift/Minishift way," but it works for us so I went with it.

• My solution configures a minishift cluster with requisite configs such that future starts can just be minishift start and it will use the default profile configurations specified in the script. Then the parent script spawns child script/processes, which each configure requisite apps for our platform in separate minishift project/namespaces--with each app pulling/tagging their requisite docker images from private AWS ECR repo and pushing into the Minishift docker registry--i.e.:

docker push 172.30.1.1:5000/myapp-local/myapp

• Another bit of advice from my experience was that the virtualbox driver just worked--('seem to recall some other drivers missing plugins, configurations, etc.) so went with VB driver and that = profit for me. JIC, my script will install the requisite minishift plugins, enable them, and also set admin creds--this should only be used in local/dev env's as it is not secure. So it will configure the driver to be virtualbox as the default, but you can easily change that as you wish.

• Notes
  I'm currently Not running local docker, but using Minishift docker context--will that work for you?
  This parent script sets the Docker context to the Minishift Registry and thus, I'm not even running Docker at all on my OSX laptop/workstation

  • Note: see the above noted git repo for the minishift deployment script which pushes, to Minishift docker registry without any local docker for mac running.

$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

AIR, you Can, have local docker running with this script without issues too, since the script will set the requisite context.

DOCKER CONTEXT
# switch to minishift context
eval $(minishift docker-env)
#eval $(minikube docker-env) # if we were using minikube

env | grep -i docker
DOCKER_HOST=tcp://192.168.99.100:2376
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/Users/cmcc/.minishift/certs

# switch back to your machines context do:
eval $(docker-machine env -u)

I hope this helps others and would love to hear how folks are automating their docker images in Openshift Origin in H/A production environments, as I'm currently rolling my own scripts ATM to facilitate app creation, deployment, rollback, etc.
I'm curious if the masses would recommend embracing using Openshift templates? This seems very cool to be able to export any/all requisite NS/Project artifacts and codify/promote/manage them, etc.

-cmcc

[mikz]

@cmcconnell1 I just settled on oc cluster up --create-machine.

[patelbharat]

I am seeing this issue on docker version 18.06.1-ce-mac73 as well. Had to downgrade to 17.09 as a work around

[fg78nc]

@patelbharat You can just do oc cluster up --create-machine.

[mikz]

@fg78nc that is gone in 3.10.

[bakayolo]

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     true

$ oc version
oc v3.10.0+dd10d17
kubernetes v1.10.0+b81c8f8
features: Basic-Auth

Server https://127.0.0.1:8443
openshift v3.10.0+e3465d0-44
kubernetes v1.10.0+b81c8f8

Still same issue.

$ oc cluster up --create-machine
Error: unknown flag: --create-machine

[bakayolo]

Confirmed working with

docker version
Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:22:25 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:28:28 2017
 OS/Arch:      linux/amd64
 Experimental: true

[rebeccaSimmonds19]

I still have this problem with docker q8.06.01-ce-mac73 and 3.10. Only updated as i thought this was fixed.

[adambkaplan]

FYI OpenShift 4.0 won't run "bare" on macOS, period. Minishift is the only viable path to running a local cluster on macOS at this point.

@csrwng once minishift supports 4.0 we can probably close this issue.

[mikz]

@adambkaplan minishift with hyperkit runner is basically the same as Docker for macOS. And that works. So, why it would not run on plain Docker for mac? Just because of some proxy setting?

[leetrout]

@adambkaplan Is your comment summing this up as a docker for mac issue and not an OpenShift issue at this point (due to the proxy settings being ignored by Docker for Mac)?

Could you elaborate by what you mean by "bare" on Mac OS? You are just reiterating some form of virtualization is required to provide a Linux kernel?

[adambkaplan]

@leetrout @mikz for OpenShift 3.11, the barrier is the outstanding docker for mac issue, which has been there for quite some time. Running "bare" via oc cluster up (which is what I meant) will fail if you're using Docker for Mac > 17.09.

OpenShift 4.0 (which is currently alpha.0) is a whole different ballgame. We don't have support for running local clusters on a Mac at present.

[riuvshin]

@adambkaplan what about Linux? would it still be possible to run oc cluster up ?
that is a significant change that will break a lot of CI testing pipelines where products are tested on local openshift clusters. For testing purposes there is no way to run whole proper OS cluster and in the same time you can't run minishift with VM backend on CI infras due to nested virtualization issues.

[Asgoret]

@Asgoret I used to downgrade docker for mac to get this worked...

@riuvshin I upgrade Docker for macOS today....so which version i work?

[Asgoret]

@alberttwong one question. If oc cluster up won't work on macOS is that mean, that documentation will be corrected for minishift in short time? At this moment there is no actual documentation for large part of OKD in minishift. For example ASB installer documentation shows 404 error and minishift addons doesn't updates for 7 months or more time.

[hakdogan]

In addition to the thought of @Asgoret, I think that also the OKD documentation is not enough. There is no explanation for Mac users about this problem. I used to downgrade because there are no other options.

[riuvshin]

@Asgoret I was using this version https://download.docker.com/mac/stable/18433/Docker.dmg
this is 17.06.0-ce-mac18

[rb2010]

still an issue with docker for mac and oc cluster up:
Docker version 18.09.1, build 4c52b90
will try to downgrade and check if it works.

[benweizhu]

Any updates?

[benweizhu]

I downgrade to Docker version 17.09.1-ce, build 19e2cf6, it works~

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[infinitewarp]

/remove-lifecycle stale

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[infinitewarp]

/remove-lifecycle rotten

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[infinitewarp]

/remove-lifecycle stale

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.