new build flag --docker-host does not work as expected

[wburningham]

Summary

This could be my misunderstanding of how the --docker-host flag should be used, so let me know if the error is on my end.

I've been using the --publish flag for the past months and have been waiting for #988 so I could build my containers using a remote docker daemon without publishing. Now that #988 is merged I can't seem to get the --docker-host flag to work.

My builds exit early with:

09:59:49  ===> ANALYZING
09:59:49  ERROR: failed to get previous image: Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running?
09:59:49  ERROR: failed to build: executing lifecycle: failed with status code: 1

---

Reproduction

Steps

1. set up a remote docker dameon
2. set DOCKER_HOST=tcp://127.0.0.1:2375
3. make sure docker info works
4. download the tip of main for pack and build
5. make sure pack report shows a commit that includes the new build flag --docker-host
6. run ./pack build --publish demo/app -v --builder paketobuildpacks/builder:tiny (The builder doesn't matter in this case since the error happens in the analyze phase trying to use docker)
7. notice how everything works with the --publish flag
8. run ./pack build --docker-host=inherit demo/app -v --builder paketobuildpacks/builder:tiny (The builder doesn't matter in this case since the error happens in the analyze phase trying to use docker)
9. notice the ERROR: failed to get previous image: Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running? error in the ANALYZE phase.
10. run ./pack build --docker-host=tcp://127.0.0.1:2375 demo/app -v --builder paketobuildpacks/builder:tiny (The builder doesn't matter in this case since the error happens in the analyze phase trying to use docker)
11. notice the ERROR: failed to get previous image: Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running? error in the ANALYZE phase.

Current behavior

• Builds using the --publish flag work
• Builds not using the --publish flag and using the --docker-host=inherit flag don't work
• Builds not using the --publish flag and using the --docker-host=tcp://127.0.0.1:2375 flag don't work

Expected behavior

• Builds not using the --publish flag and using the --docker-host=inherit flag do work
• Builds not using the --publish flag and using the --docker-host=tcp://127.0.0.1:2375 flag do work

---

Environment

DOCKER_HOST=tcp://127.0.0.1:2375

pack info

09:59:46  Pack:
09:59:46    Version:  0.0.0+git-0c84b4a
09:59:46    OS/Arch:  linux/amd64
09:59:46  
09:59:46  Default Lifecycle Version:  0.10.2
09:59:46  
09:59:46  Supported Platform APIs:  0.3, 0.4
09:59:46  
09:59:46  Config:
09:59:46  (no config file found at /root/.pack/config.toml)

docker info

09:59:47  Client:
09:59:47   Context:    default
09:59:47   Debug Mode: false
09:59:47   Plugins:
09:59:47    app: Docker App (Docker Inc., v0.9.1-beta3)
09:59:47    buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
09:59:47  
09:59:47  Server:
09:59:47   Containers: 345
09:59:47    Running: 4
09:59:47    Paused: 0
09:59:47    Stopped: 341
09:59:47   Images: 2198
09:59:47   Server Version: 20.10.3
09:59:47   Storage Driver: overlay2
09:59:47    Backing Filesystem: xfs
09:59:47    Supports d_type: true
09:59:47    Native Overlay Diff: true
09:59:47   Logging Driver: json-file
09:59:47   Cgroup Driver: cgroupfs
09:59:47   Cgroup Version: 1
09:59:47   Plugins:
09:59:47    Volume: local
09:59:47    Network: bridge host ipvlan macvlan null overlay
09:59:47    Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
09:59:47   Swarm: inactive
09:59:47   Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
09:59:47   Default Runtime: runc
09:59:47   Init Binary: docker-init
09:59:47   containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
09:59:47   runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
09:59:47   init version: de40ad0
09:59:47   Security Options:
09:59:47    seccomp
09:59:47     Profile: default
09:59:47   Kernel Version: 3.10.0-1160.15.2.el7.x86_64
09:59:47   Operating System: CentOS Linux 7 (Core)
09:59:47   OSType: linux
09:59:47   Architecture: x86_64
09:59:47   CPUs: 32
09:59:47   Total Memory: 125.7GiB
09:59:47   Name: jail4-house.eng.qops.net
09:59:47   ID: 2RSL:KOJM:VI3R:VHXW:2YDN:CWLL:W6JS:HY5T:GNOW:BM5U:QBDX:E6SY
09:59:47   Docker Root Dir: /var/lib/docker
09:59:47   Debug Mode: false
09:59:47   Registry: https://index.docker.io/v1/
09:59:47   Labels:
09:59:47   Experimental: false
09:59:47   Insecure Registries:
09:59:47    127.0.0.0/8
09:59:47   Live Restore Enabled: false
09:59:47  
09:59:47  WARNING: API is accessible on http://127.0.0.1:2375 without encryption.
09:59:47           Access to the remote API is equivalent to root access on the host. Refer
09:59:47           to the 'Docker daemon attack surface' section in the documentation for
09:59:47           more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
09:59:47  WARNING: bridge-nf-call-iptables is disabled
09:59:47  WARNING: bridge-nf-call-ip6tables is disabled

[wburningham]

@matejvasek (tagging you since you wrote the PR), am I using the --docker-host flag correctly?

[matejvasek]

@wburningham when using TCP you must ensure the address is visible from the build container. For globally visible hostnames/IP adresses this is not an issue, it's little bit more tricky with localhost.

On Linux you can do that by using --network=host then host's 127.0.0.1 will be visible.
./pack build --network=host --docker-host=tcp://127.0.0.1:2375 demo/app -v --builder paketobuildpacks/builder:tiny

Another option would be pass your docker bridge address it could look like 172.17.0.1, you could check it via ip a.

13: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:3b:c0:29:3e brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:3bff:fec0:293e/64 scope link 
       valid_lft forever preferred_lft forever

./pack build --docker-host=tcp://172.17.0.1:2375 demo/app -v --builder paketobuildpacks/builder:tiny

https://nickjanetakis.com/blog/docker-tip-65-get-your-docker-hosts-ip-address-from-in-a-container

[matejvasek]

Builds using the --publish flag work

I think I should have made --publish and --docker-host mutually exclusive.
wdyt @dfreilich @micahyoung ?

[wburningham]

@matejvasek so does the --publish flag automatically set the --network=host option?

[matejvasek]

@matejvasek so does the --publish flag automatically set the --network=host option?

No, it doesn't.

The --publish option is unrelated to the --network option, it's related to the --docker-host option. The thing is that it doesn't makes sense to use --publish and --docker-host together since --publish makes build container to push docker registry (like docker.io or quay.io) not to docker daemon. If you use --publish then --docker-host is ignored.

[matejvasek]

I think this works as designed. The only inconvenience is that by default from build container point of view 127.0.0.1 is not the same thing as 127.0.0.1 from host point of view. What do you think @dfreilich @micahyoung ?

[matejvasek]

Builds using the --publish flag work

I guess this works because the pack cli will upload the image to docker.io and they are downloading it.
Those upload/download may be unnecessary. @dfreilich correct me if I am wrong.

[micahyoung]

@matejvasek that lines up with my understanding - that any --docker-host=tcp://<host address> must be independently accessible to both to pack and the build container it creates on the daemon. Also, that adding --docker-host=tcp://127.0.0.1:2375 --network=host should potentially provide that connectivity (at least for a Linux Docker daemon).

And, yes --publish and --docker-host would be mutually exclusive since the build container's DOCKER_HOST would be ignored by lifecycle when it writes to the registry instead of the daemon.

I think this issue does point out how some underlying complexity of pack leaks through to the users with --docker-host. I wonder, could a warning or maybe some connectivity validation of the host value be useful? Maybe we could start with a string check for tcp://127.0.0.*:* or tcp://localhost:* then warn to use --network=host. Or maybe some other change to the UX?

(Updated for clarity)

[jdalbis]

I'm following this thread with interest as we're considering using pack in a CI/CD environment using jenkins and a remote docker daemon. Using the --docker-host parameter, I can get past the ===> ANALYZING problems, but only when using an insecure remote docker daemon.
Unfortunately, when using a docker daemon secured with --tlsverify and --tls*certs, the analyze phase now fails with a ERROR: failed to get previous image: Error response from daemon: Client sent an HTTP request to an HTTPS server. as it seems it does not respect the DOCKER_CERT_PATH and DOCKER_TLS_VERIFY environment variables either.

[micahyoung]

@jdalbis If I understand your scenario correctly, it sounds like your use-case may already be supported with current pack, without the --docker-host flag. You should be able to do this today:

export DOCKER_HOST=tcp://remote-host:2376
export DOCKER_CERT_PATH=/my-certs    #location of your tls* certs
export DOCKER_TLS_VERIFY=1

pack build example.com/my-app:latest ...    # normal params

With this setup, pack creates its build container on the remote Docker daemon over the TLS-encrypted API. It then runs all the buildpacks there, builds your app, and depending on whether you use --publish or not, it will write the image to a registry or to that daemon (over a locally bind-mounted socket in the container).

I put together a post from a couple months back that goes into some more depth: https://medium.com/buildpacks/pack-with-a-remote-docker-daemon-41aab804b839

But the --docker-host flag, right now, is really only to override how that last "locally bind-mounted socket in the container" part works. It wouldn't change how pack uses the DOCKER_* env vars to create the container in the first place, so I'm pretty sure you may be able to do what want today.

[micahyoung]

@matejvasek This leads me to wonder if there's a mismatch between the name of the --docker-host flag and it's implemented behavior. I wonder if it would make sense to change the flag name for now.

Maybe something like --lifecycle-docker-host or --lifecycle-socket but keep all the implemented behavior. That could make it easier to distinguish from pack's DOCKER_HOST, and probably makes it clearer what "inherit" is applying pack's settings to the lifecycle container.

Would you be ok with changing the name, and if so, do you have thoughts on a new name?

Perhaps (and separately from this PR) we should even add a separate flag for setting pack's daemon host, instead of through the DOCKER_* env vars I described in the previous comment. It could behave much like docker-CLI's --host, which is potentially a familiar pattern for doing this.

[jromero]

We should discuss this in a sub-team sync when @micahyoung is present.

[jromero]

Waiting on platform sub-team RFC from @micahyoung on this.

[jjbustamante]

We can fix this one, by solving #1338, I am going to close it in favor of the latest one. Please reopen this ticket if someone has strong feels about it