Mount volume with rw permissions #131

aleksey925 · 2022-12-24T14:33:25Z

Describe the bug
It is not possible to create a container with a mounted volume.

Steps to reproduce

Create file docker-compose.yml with below content:

services:

  db:
    image: postgres:14-alpine
    restart: on-failure:10
    environment:
      - POSTGRES_USER=admin
      - POSTGRES_PASSWORD=admin
      - POSTGRES_DB=db_name
    volumes:
      - ./.volumes/db:/var/lib/postgresql/data
    ports:
      - '5432:5432'

Execute command finch compose up

Expected behavior
Container started successful

Logs

WARN[0000] Ignoring: service db: restart="on-failure:10" (unknown) 
WARN[0000] Ignoring: volume: Bind: [CreateHostPath]     
INFO[0000] Ensuring image postgres:14-alpine            
INFO[0000] Creating container runtime-config_db_1       
FATA[0000] error while creating container runtime-config_db_1: exit status 1 
FATA[0000] exit status 1

Finch version: v0.1.1
limactl version: fc3f2a9
qemu version: 7.1.0

The text was updated successfully, but these errors were encountered:

ningziwen · 2022-12-25T02:14:11Z

Duplicate with #107

aleksey925 · 2022-12-25T08:50:49Z

@ningziwen It's not duplicate. I tried to place docker-compose.yml in the root of the user directory. It did not help.

ningziwen · 2022-12-26T06:26:00Z

The mount path also needs to be in home directory. What is the absolute path of "./.volumes/db" in your compose yaml? Is it in home directory? @aleksey925

aleksey925 · 2022-12-26T20:05:07Z

@ningziwen

Is it in home directory?

Yes.
The path to the folder is ~/.volumes/db.

monirul · 2023-01-03T20:18:44Z

Hi @aleksey925,
I have reproduced this error.
This error occurs when the folder .volumes/db does not exist. It works for me when i created the folder .volumes/db folder.

My folder structure looks like:

~/finch-issue
|- docker-compose.yml
|- .volumes/db

I run the compose command from ~/finch-issue folder:

finch compose up

The output looks like:

WARN[0000] Ignoring: service db: restart="on-failure:10" (unknown) 
WARN[0000] Ignoring: volume: Bind: [CreateHostPath]     
INFO[0000] Ensuring image postgres:14-alpine            
INFO[0000] Creating container git-issue-131_db_1        
INFO[0000] Attaching to logs                            
db_1 |chown: /var/lib/postgresql/data: Permission denied
INFO[0000] Container "git-issue-131_db_1" exited        
INFO[0000] All the containers have exited               
INFO[0000] Stopping containers (forcibly)               
INFO[0000] Stopping container git-issue-131_db_1

To fix the db_1 |chown: /var/lib/postgresql/data: Permission denied error pls refer to this github issue in postgres
docker-library/postgres#116

To fix the permission issue I used following compose yml:

services:
  db:
    image: postgres:14-alpine
    restart: on-failure:10
    environment:
      - POSTGRES_USER=admin
      - POSTGRES_PASSWORD=admin
      - POSTGRES_DB=db_name
      - PGDATA=/var/lib/postgresql/pgdata
    volumes:
      - ./.volumes/db:/var/lib/postgresql/data:wr
    ports:
      - '5432:5432'

Output

WARN[0000] Ignoring: service db: restart="on-failure:10" (unknown) 
WARN[0000] Ignoring: volume: Bind: [CreateHostPath]     
INFO[0000] Ensuring image postgres:14-alpine            
INFO[0000] Creating container git-issue-131_db_1        
INFO[0000] Attaching to logs                            
db_1 |The files belonging to this database system will be owned by user "postgres".
db_1 |This user must also own the server process.
db_1 |
db_1 |The database cluster will be initialized with locale "en_US.utf8".
db_1 |The default database encoding has accordingly been set to "UTF8".
db_1 |The default text search configuration will be set to "english".
db_1 |
db_1 |Data page checksums are disabled.
db_1 |
db_1 |fixing permissions on existing directory /var/lib/postgresql/pgdata ... ok
db_1 |creating subdirectories ... ok
db_1 |selecting dynamic shared memory implementation ... posix
db_1 |selecting default max_connections ... 100
db_1 |selecting default shared_buffers ... 128MB
db_1 |selecting default time zone ... UTC
db_1 |creating configuration files ... ok
db_1 |running bootstrap script ... ok
db_1 |sh: locale: not found
db_1 |2023-01-03 20:11:19.853 UTC [31] WARNING:  no usable system locales were found
db_1 |performing post-bootstrap initialization ... ok
db_1 |syncing data to disk ... ok
db_1 |
db_1 |initdb: warning: enabling "trust" authentication for local connections
db_1 |
db_1 |Success. You can now start the database server using:
db_1 |You can change this by editing pg_hba.conf or using the option -A, or
db_1 |
db_1 |--auth-local and --auth-host, the next time you run initdb.
db_1 |    pg_ctl -D /var/lib/postgresql/pgdata -l logfile start
db_1 |
db_1 |waiting for server to start....2023-01-03 20:11:24.248 UTC [37] LOG:  starting PostgreSQL 14.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
db_1 |2023-01-03 20:11:24.252 UTC [37] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 |2023-01-03 20:11:24.266 UTC [38] LOG:  database system was shut down at 2023-01-03 20:11:21 UTC
db_1 |2023-01-03 20:11:24.280 UTC [37] LOG:  database system is ready to accept connections
db_1 | done
db_1 |server started
db_1 |CREATE DATABASE
db_1 |
db_1 |
db_1 |/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 |2023-01-03 20:11:25.652 UTC [37] LOG:  received fast shutdown request
db_1 |waiting for server to shut down....2023-01-03 20:11:25.656 UTC [37] LOG:  aborting any active transactions
db_1 |2023-01-03 20:11:25.661 UTC [37] LOG:  background worker "logical replication launcher" (PID 44) exited with exit code 1
db_1 |2023-01-03 20:11:25.674 UTC [39] LOG:  shutting down
db_1 |2023-01-03 20:11:25.701 UTC [37] LOG:  database system is shut down
db_1 | done
db_1 |server stopped
db_1 |
db_1 |PostgreSQL init process complete; ready for start up.
db_1 |
db_1 |2023-01-03 20:11:25.924 UTC [1] LOG:  starting PostgreSQL 14.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
db_1 |2023-01-03 20:11:25.924 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1 |2023-01-03 20:11:25.924 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1 |2023-01-03 20:11:25.930 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 |2023-01-03 20:11:25.939 UTC [52] LOG:  database system was shut down at 2023-01-03 20:11:25 UTC
db_1 |2023-01-03 20:11:25.959 UTC [1] LOG:  database system is ready to accept connections

aleksey925 · 2023-01-05T19:58:42Z

Hi @monirul, I tried to replicate this. The container started, but the database files are still missing on my local machine. Unfortunately, this is not a solution and the problem is somewhere deeper.

aleksey925 · 2023-01-12T08:32:49Z

Hi @monirul. Do you have any news?

monirul · 2023-01-13T11:29:18Z

Hi @aleksey925,
I am still investigating to find out the root cause. I will get back to you asap.

There was a typo in the yml i provided.

services:
  db:
    image: postgres:14-alpine
    restart: on-failure:10
    environment:
      - POSTGRES_USER=admin
      - POSTGRES_PASSWORD=admin
      - POSTGRES_DB=db_name
      - PGDATA=/var/lib/postgresql/db/pgdata
    volumes:
      - ./.volumes/db:/var/lib/postgresql/db:rw
    ports:
      - '5432:5432'

However, for this yml, the compose up fails with an error db_1 |chown: /var/lib/postgresql/db: Permission denied.

For executing the compose command, Finch passes the compose command to nerdctl. This error is returned from the nerdctl. I am investigating why it is failed. I will get back to you as soon as I get the root cause.

Thanks,
Monirul.

monirul · 2023-01-17T02:06:32Z

Found the root cause of this issue. Thanks @djdongjin for helping me with the investigation.

Finch uses lima for virtualization. It seems this error is a known limitation of the sshfs server in lima. sshfs server runs without the root privilege, due to that, chown command fails with permission error - chown: /var/lib/postgresql/db: Permission denied . Same error occurs in rancher as well. You can also reproduce it by shelling into the vm and run chown command.

Similar lima issues:
lima-vm/lima#231
rancher-sandbox/rancher-desktop#1209

Here is a lima-issue where lima maintainer suggested to use volume create as workaround.
lima-vm/lima#505

I manage to run the postgres using...

$ finch volume create pgdata

compose yaml file

services:
  db:
    image: postgres:14-alpine
    environment:
      - POSTGRES_USER=admin_user
      - POSTGRES_PASSWORD=admin12345
      - POSTGRES_DB=db_name
      - PGDATA=/var/lib/postgresql/db/pgdata
    volumes:
      - pgdata:/var/lib/postgresql/db:rw
    ports:
      - '5432:5432'
volumes:
  pgdata:

then run following command:

$ finch compose up
INFO[0000] Creating volume git-issue-131_pgdata
INFO[0000] Ensuring image postgres:14-alpine
INFO[0000] Re-creating container git-issue-131_db_1
INFO[0000] Attaching to logs
...
db_1                |2023-01-17 01:48:35.429 UTC [1] LOG:  starting PostgreSQL 14.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
db_1                |2023-01-17 01:48:35.429 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1                |2023-01-17 01:48:35.429 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1                |2023-01-17 01:48:35.436 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1                |2023-01-17 01:48:35.441 UTC [53] LOG:  database system was shut down at 2023-01-17 01:48:35 UTC
db_1                |2023-01-17 01:48:35.449 UTC [1] LOG:  database system is ready to accept connections

Btw, As per lima road map, currently lima supports 9p as experimental feature and plans to switch to 9p instead of reverse-sshfs as default mount driver in v1.0(tentative).

pendo324 · 2023-01-25T19:08:10Z

Confirmed that the workaround that @monirul posted works. I did not have to run a separate volume command either, just finch compose up with the compose file example that he provided.

For a permanent long term fix, we are tracking moving off of reverse sshfs file sharing to 9p and virtiofs (when available, so Monterrey and above). We were hoping that the latest QEMU version (7.2) would resolve the performance issues of 9p, but it seems like it still needs some more investigation (see the latest comments in lima-vm/lima#971).

I think we can leave this issue open until the long term fix is viable. Please reach out if Monirul's comment/workaround still doesn't solve your issue for now.

djdongjin · 2023-01-25T19:12:49Z

I did not have to run a separate volume command either, just finch compose up with the compose file example that he provided.

Yeah it's expected. Volumes in compose.yaml will be created automatically.

volumes:
  pgdata:

prithviramesh · 2023-01-25T19:17:20Z

how do you pre-populate pgdata if you want to mount data into postgres before starting the db?

I posted almost the same issue here: #81 (comment)

We use a bind mount to get data into postgres during docker compose. finch compose up fails due to the chown error you mentioned above.

If i create a volume as part of the file, but I need data in that volume before starting postgres - what would I do?

pendo324 · 2023-01-25T20:47:43Z

how do you pre-populate pgdata if you want to mount data into postgres before starting the db?

I posted almost the same issue here: #81 (comment)

We use a bind mount to get data into postgres during docker compose. finch compose up fails due to the chown error you mentioned above.

If i create a volume as part of the file, but I need data in that volume before starting postgres - what would I do?

This is not exactly the same issue, as your issue is regarding using data from the macOS system inside the container. Please see the reply to your issue for more information on the specific use case you asked about, thanks.

prithviramesh · 2023-01-25T21:14:12Z

Ah got it - thanks a bunch

ningziwen · 2023-04-03T22:07:54Z

Closing because of no response. Feel free to reopen. More discussions can continue in #81

aleksey925 added the bug Something isn't working label Dec 24, 2022

KevinLiAWS assigned ningziwen Dec 26, 2022

ningziwen assigned monirul and unassigned ningziwen Dec 26, 2022

pendo324 mentioned this issue Jan 25, 2023

finch run -v is failed #81

Closed

ningziwen closed this as completed Apr 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mount volume with rw permissions #131

Mount volume with rw permissions #131

aleksey925 commented Dec 24, 2022 •

edited

Loading

ningziwen commented Dec 25, 2022

aleksey925 commented Dec 25, 2022

ningziwen commented Dec 26, 2022

aleksey925 commented Dec 26, 2022 •

edited

Loading

monirul commented Jan 3, 2023 •

edited

Loading

aleksey925 commented Jan 5, 2023

aleksey925 commented Jan 12, 2023

monirul commented Jan 13, 2023 •

edited

Loading

monirul commented Jan 17, 2023 •

edited

Loading

pendo324 commented Jan 25, 2023 •

edited

Loading

djdongjin commented Jan 25, 2023

prithviramesh commented Jan 25, 2023

pendo324 commented Jan 25, 2023 •

edited

Loading

prithviramesh commented Jan 25, 2023

ningziwen commented Apr 3, 2023

Mount volume with rw permissions #131

Mount volume with rw permissions #131

Comments

aleksey925 commented Dec 24, 2022 • edited Loading

ningziwen commented Dec 25, 2022

aleksey925 commented Dec 25, 2022

ningziwen commented Dec 26, 2022

aleksey925 commented Dec 26, 2022 • edited Loading

monirul commented Jan 3, 2023 • edited Loading

aleksey925 commented Jan 5, 2023

aleksey925 commented Jan 12, 2023

monirul commented Jan 13, 2023 • edited Loading

monirul commented Jan 17, 2023 • edited Loading

pendo324 commented Jan 25, 2023 • edited Loading

djdongjin commented Jan 25, 2023

prithviramesh commented Jan 25, 2023

pendo324 commented Jan 25, 2023 • edited Loading

prithviramesh commented Jan 25, 2023

ningziwen commented Apr 3, 2023

aleksey925 commented Dec 24, 2022 •

edited

Loading

aleksey925 commented Dec 26, 2022 •

edited

Loading

monirul commented Jan 3, 2023 •

edited

Loading

monirul commented Jan 13, 2023 •

edited

Loading

monirul commented Jan 17, 2023 •

edited

Loading

pendo324 commented Jan 25, 2023 •

edited

Loading

pendo324 commented Jan 25, 2023 •

edited

Loading