Permissions on for files mounted with Mutagen-based caching

[rastasheep]

• I have tried with the latest version of my channel (Stable or Edge)

Information

• macOS Version: 10.15.4
• Docker for Mac: 2.3.1.0 (45408) edge

I'm using docker-compose to mount files into container which is run with root user (official node image with node user).
When I set up Mutagen based caching, my non root user doesn't have permissions to edit mounted files. Without caching enabled mounted files are owned by that non root user.

Here is service definition from docker-compose:

app:
    build:
      context: ./packages/web-client/
      dockerfile: Dockerfile.dev
    working_dir: /application
    user: node
    volumes:
      - .:/application:cached
      - ./script/dev/entrypoint.sh:/home/node/entrypoint.sh:cached

File permissions without caching

File permissions with Mutagen-based caching

[kducharm]

Also confirmed with an Alpine container:

Test:
docker run -ti -v ~/cache-enabled-path:/app --user sync --rm alpine
vs
docker run -ti -v ~/non-cache-enabled-path:/app --user sync --rm alpine

Looking in /app on the cache-enabled path is not respecting the passed in user.

[djs55]

Thanks for the report! I've got an experimental build you could try: https://desktop-stage.docker.com/mac/edge/45418/Docker.dmg -- it has 2 changes:

1. by default it sets the permissions to 0777 and 0666 which should allow any user (in a container) to write
2. it re-enables support for ~/.mutagen.yml which can be used to override these defaults if necessary

Let me know if that helps at all.

[chezsmithy]

Confirmed this fixes the problem! @djs55 could you please release a signed copy of this release so I can continue testing without disabling gatekeeper?

[djs55]

@chezsmithy I've notarized another testing build: https://desktop-stage.docker.com/mac/edge/45429/Docker.dmg . This one should have the permissions change and the ~/.mutagen.yml support in it.

[rastasheep]

Thank you so much @djs55 for quick response.
I can confirm that build 45429 fixes problem for me.

Random question, i've noticed that while caching/indexing is in the progress, i cannot start containers which want to mount stuff from those locations (make sense). Are there plans for improving visibility into that?

Also is there any good place where i can find more information how Mutagen-based caching works, what's different and is it better than before other than https://docs.docker.com/docker-for-mac/mutagen-caching/ ?

[alexhafner]

@rastasheep - there's a lot on Mutagens website https://mutagen.io/documentation/introduction - including their initial Docker implementation https://mutagen.io/documentation/transports/docker . Will be interesting to learn how that was absorbed into Docker for Mac.

[rjensen26]

I have been using Mutagen for a little bit and my initial reaction is fairly impressed.

[rastasheep]

After some time using it on real project, i can notice great improvements on yarn install times .. maybe slower edit propagation into container, but that's minor.

[alexhafner]

Agreed, same here for database migrations with heavy file access, probably 3x faster than previously

[jasonwilliams]

Im still having permissions issues with 2.3.1.0 (45429)
do i need to remove the cache and rebuild it?
When i toggle to "Not Cached" everything works fine.

Edit: I deleted the cache, removed the containers, switched the cache back on then rebuilt containers and now it works fine.

[back-2-95]

45429 worked for me with Drupal 8 (PHP, Symfony3) project.

When I synced data from live environment, I got following kind errors from Drush (cli tool for Drupal) related to every folder in rsynced paths:

> rsync: chgrp "/app/public/sites/default/files/styles/hero_lg" failed: Operation not permitted (1)

Drush uses rsync to sync uploaded files and it seems to have some permission problems. I need to test it further.

[chezsmithy]

45429 worked for me with Drupal 8 (PHP, Symfony3) project.

When I synced data from live environment, I got following kind errors from Drush (cli tool for Drupal) related to every folder in rsynced paths:

> rsync: chgrp "/app/public/sites/default/files/styles/hero_lg" failed: Operation not permitted (1)

Drush uses rsync to sync uploaded files and it seems to have some permission problems. I need to test it further.

@back-2-95 did you download the updated notarized build from the comments above? This build fixed my file permission issues.

[jasonwilliams]

@djs55 could you let us know when your fix goes into the Edge release ?

[chezsmithy]

@chezsmithy I've notarized another testing build: https://desktop-stage.docker.com/mac/edge/45429/Docker.dmg . This one should have the permissions change and the ~/.mutagen.yml support in it.

@jasonwilliams here is an updated edge release which is notarized. I am running this release without issue now.

[back-2-95]

@chezsmithy to your question 2 days ago: yes, I used updated notarized build (45429)

[back-2-95]

I wonder what this means and how can I see what is wrong.

It's a site which I tested with and it worked. Now it's like this.

[jasonwilliams]

@djs55 has your fix gone into the mainline (edge) build, or do we still need to download your patch to fix the permissions issues?

@chezsmithy I've notarized another testing build: https://desktop-stage.docker.com/mac/edge/45429/Docker.dmg . This one should have the permissions change and the ~/.mutagen.yml support in it.

@jasonwilliams here is an updated edge release which is notarized. I am running this release without issue now.

@chezsmithy That's not what i meant, I meant do we still need to download the specific notarized build or can we use the edge release now

[rjensen26]

@back-2-95 , I actually got that a couple of times as well.. I had that issue when I was mounting a lot of file, I also may have ran out of disk space for docker as well.

If I remember right how I resolved it was to:

• Remove it from the "File Sharing"
• Restart Docker
• Verify Disk usage was not full or nearing full
• Clear volumes associated to that project
• Make sure that the docker-compose.yml or any other supporting docker-compose.* files had :delegated and not :cached

I do wish we could see what the actual error was..

[KazWolfe]

It would be nice if we could somehow specify permission and other sync-related configurations for each mount as well. Something like this would allow us a lot of power in setting owning users/groups as well as the countless other number of options Mutagen lets us tweak.

Even if we have a per-project mutagen.yml configuration we can use, this would help a lot with fixing this and potentially other configuration issues like #4794.

[jasonwilliams]

For those using the mutagen.yml to fix the permissions issue, what does the format of your mutagen.yml file look like?

[Valkyrie00]

For those using the mutagen.yml to fix the permissions issue, what does the format of your mutagen.yml file look like?

@jasonwilliams I use this structure, i hope you find it useful.

sync:
  defaults:
    permissions:
      defaultOwner: "id:1000"
      defaultGroup: "id:82"

If you don't know your group or user id, try to enter in your container and type easily id in your console for look this information.

[jasonwilliams]

Since switching to the latest edge 2.3.4.0 and using the mutagen.yml ive not had much luck at all.
It tries to cache my whole project folder rather than the specific folders i've given it (im guessing this is coming from docker-composer), plus it keeps restarting end saying "Halted".

I've tried clearing volumes and purging data etc.

I will have to go back to @djs55 specific build or back to Stable :(

Edit: it looks like switching to :cached instead of :delegated made things more stable. Im also using the build from #4590 (comment)

[stephen-turner]

Hi everyone, and thank you for all your feedback. I'm going to close this ticket because in the latest Edge release we have removed Mutagen for the moment while we reconsider a lot of the UX issues, such as those mentioned above. Instead we have made some big file sharing performance improvements by removing osxfs and using gRPC-FUSE instead (by default). The full explanation is at #1592 (comment), so please read that for more details.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked