overlay2 storage driver not working on zfs dataset

[kkettinger]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

overlay2 storage driver working on zfs dataset. I don't want to use the zfs storage driver.

Actual behavior

docker start fails:

Jun 21 15:24:43 docker dockerd[1950]: time="2022-06-21T15:24:43.260443954Z" level=info msg="Starting up"
Jun 21 15:24:43 docker dockerd[1950]: time="2022-06-21T15:24:43.261302811Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jun 21 15:24:43 docker dockerd[1950]: time="2022-06-21T15:24:43.261316992Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jun 21 15:24:43 docker dockerd[1950]: time="2022-06-21T15:24:43.261335632Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jun 21 15:24:43 docker dockerd[1950]: time="2022-06-21T15:24:43.261376473Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jun 21 15:24:43 docker dockerd[1950]: time="2022-06-21T15:24:43.262237650Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jun 21 15:24:43 docker dockerd[1950]: time="2022-06-21T15:24:43.262248631Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jun 21 15:24:43 docker dockerd[1950]: time="2022-06-21T15:24:43.262259561Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jun 21 15:24:43 docker dockerd[1950]: time="2022-06-21T15:24:43.262269631Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jun 21 15:24:43 docker dockerd[1950]: time="2022-06-21T15:24:43.264022397Z" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
Jun 21 15:24:43 docker dockerd[1950]: failed to start daemon: error initializing graphdriver: driver not supported
Jun 21 15:24:43 docker systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jun 21 15:24:43 docker systemd[1]: docker.service: Failed with result 'exit-code'.
Jun 21 15:24:43 docker systemd[1]: Failed to start Docker Application Container Engine.

Content of /etc/docker/daemon.json:

{
        "data-root": "/data/docker",
        "storage-driver": "overlay2"
}

zfs list:

root@docker /data # zfs list
NAME                                                                               USED  AVAIL     REFER  MOUNTPOINT
pool1                                                                             2.12M  10.5T       96K  /pool1
pool1/data                                                                         776K  10.5T      232K  /data

Steps to reproduce the behavior

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:03:17 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., v2.6.0)

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

Linux docker 5.10.0-15-amd64 #1 SMP Debian 5.10.120-1 (2022-06-09) x86_64 GNU/Linux

[thaJeztah]

I don't think overlayfs works on top of zfs; and you'll likely see an error about that in the daemon logs; see the prerequisites at https://docs.docker.com/storage/storagedriver/overlayfs-driver/#prerequisites

So you may need to select some different storage location if you need to use overlayfs

closing this ticket, as this is expected, but feel free to continue the conversation

[benoitjpnet]

@thaJeztah ZFS is now supporting overlayfs: openzfs/zfs#9414