Volume synchronization bug on M1 Mac

[mbrowne]

• I have tried with the latest version of Docker Desktop
• I have tried disabling enabled experimental features
• I have uploaded Diagnostics
• Diagnostics ID:

Expected behavior

Consistent synchronization of file contents inside and outside the container

Actual behavior

When mounting a volume in an M1 Mac, file synchronization doesn't work reliably. Sometimes the Docker container still has an old copy of the file after saving a new version of the file on the host.

This seems to happen when checking out a different branch using git on the host (maybe too many file changes for it to handle at once? But the repo I'm working with isn't particularly big...)

Information

macOS version: 12.2.1
M1 Pro (MacBook Pro (16-inch, 2021)
Docker Desktop Version: 4.5.0

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

[PASS] DD0027: is there available disk space on the host?
[SKIP] DD0028: is there available VM disk space?
[PASS] DD0031: does the Docker API work?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0011: are the LinuxKit services running?
[FAIL] DD0016: is the LinuxKit VM running? vm is not running: vm has not started
[PASS] DD0001: is the application running?
[PASS] DD0018: does the host support virtualization?
[FAIL] DD0017: can a VM be started? vm has not started: vm has not started
[FAIL] DD0015: are the binary symlinks installed? /usr/local/bin/kubectl is not a symbolic link
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0007: is the backend responding?
[PASS] DD0014: are the backend processes running?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[PASS] DD0012: is the VM networking working?
[PASS] DD0032: do Docker networks overlap with host IPs?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0019: is the com.docker.vmnetd process responding?
[PASS] DD0033: does the host have Internet access?

Steps to reproduce the behavior

docker run -it --rm \
    -v "$(pwd)":/infrastructure-live \
    artnetworldwide/automation-dockerimage-ci:3.1.0

Then check out a different branch in git (e.g. run git checkout some-other-branch on the host)

Change a file in the current working directory and observe that the file hasn't been updated inside the container.

[mbrowne]

In case it's relevant, here's the Dockerfile for the image where I experienced the issue (I haven't tested any other images at this point):

FROM python:3.7.4-buster AS python

# -----------------------------
# Set environment variables
# -----------------------------

# Configure web servers to bind to port 80 when present
ENV ASPNETCORE_URLS=http://+:80
# Enable detection of running in a container
ENV DOTNET_RUNNING_IN_CONTAINER=true
# Enable correct mode for dotnet watch (only mode supported in a container)
ENV DOTNET_USE_POLLING_FILE_WATCHER=true
# Skip extraction of XML docs - generally not useful within an image/container - helps performance
ENV NUGET_XMLDOC_MODE=skip
# Environment variables for .NET tools
ENV PATH=/root/.dotnet/tools:$PATH
# Path settings for pip --user folder
ENV PATH=/root/.local/bin/:$PATH
# Environment variables for pipenv/python
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8
# Environment variables for GO
ENV GOLANG_VERSION=1.13
ENV GOPATH=/go
ENV PATH=$GOPATH/bin:/usr/local/go/bin:$PATH

COPY ./install-software.sh /root/install-software.sh
RUN /root/install-software.sh

CMD ["bash"]

[RSickenberg]

Same issue with 4.6.0 and the new VitrioFS

[emilecaron]

Same issue with 4.6.0 fresh install (VitrioFS disabled). Downgrading to 4.5.0 fixes the issue.

[boywijnmaalen]

not sure how this new VirtioFS feature works but I've seen similar behaviour with webdav mounted volumes (unsure if it is of any help)

[christianseel]

Might also be related to #5795

[twss]

I’ve noticed I get a lot of ‘git loose object` corruptions when trying to commit from a container when using VitrioFS on an Intel Mac. Turning off VitrioFS allowed me to commit.

[boywijnmaalen]

@twss I've seen the exact same behaviour. when I switch to the mounted directory on the host (because I get the error you mentioned in the container) I'm able to commit my code.

[tchalupnik]

Same issue with 4.8.2

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[mbrowne]

/remove-lifecycle stale

[b-hayes]

There is def an issue here.
I was pulling my hair out for 2 months wondering why mysql containers would not run for me when they worked fine for everyone else including older m1 air users.
Mysql would imeeedialty exit, if i stopped using a volume mount it would be fine but I need the volume mount.

Eventually I tried changing the image from mysql/mysql-server:latest to mysql:8.0-oracle and all my problem went away and I promply got back to work.

But now today I just tried to run a fresh http container and found volume mounts would simply not work.

Using the command from the docker hub documentation:

docker run -dit --name my-apache-app -p 8080:80 -v "$PWD":/usr/local/apache2/htdocs/ httpd:2.4

I was only ever able to get the default index.html provided by the image. I jump into the contianer and do an ls and see that none of my files are there.

After pulling my hair our for 2 hours I walked over to my old PC and ran the exact same command on Ubuntu WSLv2 and it works. Created a test folder with a file and it appears and the default index.html is nowhere to be seen.

So yeah def an issue with volume mounts going on but seems to be working for some images and not others. ( all images tested support arm64).

[yaroslavyaroslav]

Just for other folks, who searching quick-fix: the problem in VitrioFS related only. So switching the virtualisation driver to any other makes it work.

[docker-robott]

There hasn't been any activity on this issue for a long time.
If the problem is still relevant, mark the issue as fresh with a /remove-lifecycle stale comment.
If not, this issue will be closed in 30 days.

Prevent issues from auto-closing with a /lifecycle frozen comment.

/lifecycle stale

[b-hayes]

/remove-lifecycle stale

[docker-robot]

There hasn't been any activity on this issue for a long time.
If the problem is still relevant, mark the issue as fresh with a /remove-lifecycle stale comment.
If not, this issue will be closed in 30 days.

Prevent issues from auto-closing with a /lifecycle frozen comment.

/lifecycle stale

[27medkamal]

/remove-lifecycle stale

[garyhopkins]

/remove-lifecycle stale

[zhixinwen]

I had similar issue when we try to build rust code in container. We mount the /target dir from host to container, where all the compiled code would be stored in. The linker would complain about not able to link the compiled .o code because they cannot be found. Disabling VitrioFS fixes this.

One thing I am confused is that everything is happening in container for my case. We compile and build source code from scratch in the container. How would sync issue between host and container cause such a problem? The container cannot read what it writes?

[Arpanbhagat5]

Facing same issue on Apple Mac M2 pro + Docker for Desktop 4.22.1
No matter which image I deply on docker container, the updates to none of the file ever work.
I am tired of having to rebuild for every change that I want to test.
I keep switching to my older mac with intel chip to not lose on productivity.
Has anyone found any solution to this yet?

[b-hayes]

@Arpanbhagat5 I found specific images seem to work more often than others. I've been using FROM php:8.0-apache without any problems so far. I had a lot of issues with MYSQL and only the oracle version works for me FROM mysql:8.0-oracle

[Arpanbhagat5]

@b-hayes Thanks for sharing some details on this.
Actually I was able to get around the sync issue. It required me to re-install Docker Desktop client, make sure VirtioFS was enabled. I had the same settings before as well though.
Anyway I tested with multiple images now. The volumes are synced without any issues.

[dslatkin]

I had similar issue when we try to build rust code in container. We mount the /target dir from host to container, where all the compiled code would be stored in. The linker would complain about not able to link the compiled .o code because they cannot be found. Disabling VitrioFS fixes this.

One thing I am confused is that everything is happening in container for my case. We compile and build source code from scratch in the container. How would sync issue between host and container cause such a problem? The container cannot read what it writes?

Out of curiosity @zhixinwen, were the error messages you were experiencing similar to those here?

/usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.2siq4apgpszziqx5.rcgu.o: No such file or directory

[zhixinwen]

Yes Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Daniel Slatkin ***@***.***> Sent: Wednesday, October 11, 2023 8:38:37 PM To: docker/for-mac ***@***.***> Cc: Zhixin Wen ***@***.***>; Mention ***@***.***> Subject: Re: [docker/for-mac] Volume synchronization bug on M1 Mac (Issue #6219) I had similar issue when we try to build rust code in container. We mount the /target dir from host to container, where all the compiled code would be stored in. The linker would complain about not able to link the compiled .o code because they cannot be found. Disabling VitrioFS fixes this. One thing I am confused is that everything is happening in container for my case. We compile and build source code from scratch in the container. How would sync issue between host and container cause such a problem? The container cannot read what it writes? Out of curiosity @zhixinwen<https://github.com/zhixinwen>, were the error messages you were experiencing similar to those here<rust-lang/docker-rust#161 (comment)>? /usr/bin/ld: cannot find /failing-builds/target/debug/deps/failing_builds-64e53c5ba8e26b41.2siq4apgpszziqx5.rcgu.o: No such file or directory — Reply to this email directly, view it on GitHub<#6219 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACN2JKOQS6ACNBPYL5BPKLDX65Q33ANCNFSM5QDU5MFA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[enragedginger]

I'm running into this same issue
MacBook Pro M2 Max Ventura 13.5
Docker Desktop 4.24.2 (but was also happening on 4.23.x).

FWIW, my use case differs slightly. I'm writing a file to a shared drive from a JVM app running on the host OS. After the app finishes writing the file and closing it, it tells postgres (running in a container) to do a bulk import of the file. Postgres errors with a message indicating the file is incomplete / corrupted. However, if I simply tell Postgres to try importing the file again, it passes. Switching back to grpc fuse resolve my issue for now, but I'd really like to keep using virtio FS

[dslatkin]

To echo other commentators here, I've also had similar errors with git "loose object" corruption. In my case it was fetching from about 70 remotes using GNU parallel while effectively using unlimited number of jobs with --jobs 0 so I believe it uses hyperthreading or something similar to switch between tasks.

I've also had mind-bending issues where I would git worktree add some-branch some/dir and git worktree add another-branch another/dir and I would find that somehow some-branch would be checked out on another/dir. Very confusing.

I disable VirtioFS, all the problems go away.

I will call out I think this core issue may still be perfectly reproducible using the rust image as I've described here (and confirmed by @zhixinwen):

rust-lang/docker-rust#161 (comment)

[dslatkin]

I just tested this issue (probably the same as #6270) again in 4.27.2 using steps from rust-lang/docker-rust#161 and sad to report this issue still persists, preventing rust builds working in bind-mounted file systems.

[ajoy39]

Still having this issue. Files updated on my local file system are not updating in the container.

[brunorozendo]

I'm in the same situation as @ajoy39

[rodrigogs]

Is it still a problem?

[miklosbagi]

Yes, I can still see this happening in 4.30.0 (149282) and 4.31.0 (153195) on m1 mac (Sanoma 14.2 (23C64)).

My use case: I'm adding new entries to a sqlite database that I would expect to be available inside the container. It is not, I have to restart the container for the changes to be recognized. This is part of an autotest pack that works fine on linux.

I'm happy to build a minified example out of this usecase if that helps with repro.

[dslatkin]

Apple (just yesterday) got back to me about my issue opened via Feedback Assistant about VIRTIO affecting Rust builds in Docker in rust-lang/docker-rust#161. They said a potential fix was identified and asked that I test again using MacOS 15 beta.

It seems the issue might be fixed? For anyone wanting to test the fix, this worked for me:

1. Upgrade to MacOS 15 beta
2. Enable VirtioFS
3. Update Docker
4. Test your issue

Hoping to hear this addresses other issues like this as well. 🙂

[dootix-alberto]

Hoping to hear this addresses other issues like this as well. 🙂
For me, it's better than before, but it's not 100% stable.

For example, when I perform a git operation, like checkout or merge, things get messed up.