Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ipfs in container creates files with root as owner #8725

Closed
3 tasks done
akhilman opened this issue Feb 9, 2022 · 15 comments · Fixed by #9992
Closed
3 tasks done

Ipfs in container creates files with root as owner #8725

akhilman opened this issue Feb 9, 2022 · 15 comments · Fixed by #9992
Labels
kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization

Comments

@akhilman
Copy link

akhilman commented Feb 9, 2022

Checklist

Installation method

third-party binary

Version

ipfs version 0.12.0-rc1
Found IPFS fs-repo at /data/ipfs
go-ipfs version: 0.12.0-rc1-8a88718
Repo version: 12
System version: amd64/linux
Golang version: go1.16.12

Config

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "http://192.168.16.90:5001",
        "http://192.168.16.90:5080",
        "http://servak.local:5001",
        "http://servak.local:5080"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip4/0.0.0.0/tcp/4002/ws",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "DNS": {
    "Resolvers": null
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "90GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true,
      "Interval": 10
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": false,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "ShardingEnabled": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooWQxEEfUh5HBgjA8bikndUFs9JnA62qAF7w7wvs4wYrtXP"
  },
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": null,
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "12h",
    "Strategy": "all"
  },
  "Routing": {
    "Type": "dht"
  },
  "Swarm": {
    "AddrFilters": null,
    "AutoRelay": true,
    "ConnMgr": {
      "GracePeriod": "20s",
      "HighWater": 600,
      "LowWater": 300,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

Since the go-ipfs v0.11 container from docker.io/ipfs/go-ipfs, files with root as owner are sometimes created and then ipfs cannot access these files.
Previously I had this problem with repo.lock and now with the database file.

go-ipfs v0.10 and earlier did not have this problem.

When this problem first appeared, after updating to v0.11, I rolled back to v0.10 and the problem went away.

root@servak:~# /usr/bin/podman run --replace -it --name ipfs-ipfs --label io.containers.autoupdate=image -v /srv/ipfs:/data/ipfs docker.io/ipfs/go-ipfs:v0.12.0-rc1
ce7b4ae10528dcd08042c8a70854dbdffe98f4dc3d0830983aed2fe59408ad34
Changing user to ipfs
ipfs version 0.12.0-rc1
Found IPFS fs-repo at /data/ipfs
Initializing daemon...
go-ipfs version: 0.12.0-rc1-8a88718
Repo version: 12
System version: amd64/linux
Golang version: go1.16.12

Error: open /data/ipfs/datastore/285386.ldb: permission denied

and root owned file itself

-rw-r--r-- 1 root root 29282 Feb  9 19:13 /srv/ipfs/datastore/285386.ldb

Repo.lock created with root as owner:

root@servak:~# ls -l /srv/ipfs/                                                                                                                                                                                      
total 120                                                                                                                                                                                                            
-rw-r--r--    1 ildar users 51571 Feb  3 08:06 11-to-12-cids.txt                                                                                                                                                     
drwxr-xr-x 1027 ildar users 20480 Feb  9 19:13 blocks                                                                                                                                                                
-rw-------    1 ildar users  3969 Feb  3 08:44 config                                                     
drwxr-xr-x    5 ildar root   4096 Apr 28  2021 _data                                                                                                                                                                 
drwxr-xr-x    2 ildar users 12288 Feb  9 19:13 datastore                                                                                                                                                             
-rw-------    1 ildar users   190 Apr 28  2021 datastore_spec                                             
drwx------    2 ildar users  4096 Apr 28  2021 keystore                                                   
drwx------    2 ildar root  16384 Apr 28  2021 lost+found                                                                                                                                                            
-rw-r--r--    1 root  root      0 Feb  9 19:13 repo.lock                                                                                                                                                             
-rw-r--r--    1 ildar users     3 Feb  3 08:06 version          
@akhilman akhilman added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels Feb 9, 2022
@welcome
Copy link

welcome bot commented Feb 9, 2022

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

  • "Priority" labels will show how urgent this is for the team.
  • "Status" labels will show if this is ready to be worked on, blocked, or in progress.
  • "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

@akhilman
Copy link
Author

akhilman commented Feb 9, 2022

podman top output:

USER   PID   PPID   %CPU    ELAPSED          TTY     TIME   COMMAND
root   1     0      0.000   1m0.60930892s    pts/0   0s     /sbin/tini -- /usr/local/bin/start_ipfs daemon --migrate=true --agent-version-suffix=docker 
ipfs   6     1      6.600   1m0.609464201s   pts/0   4s     ipfs daemon --migrate=true --agent-version-suffix=docker 

@akhilman
Copy link
Author

akhilman commented Feb 9, 2022

I will reopen this issue if get more information when exactly ipfs creates files owned by root.

@akhilman akhilman closed this as completed Feb 9, 2022
@jdannenberg
Copy link

I have the same issue (up to now only with repo.lock) running ipfs kubo in container with version 0.15.0.

@akhilman
Copy link
Author

I have the same issue (up to now only with repo.lock) running ipfs kubo in container with version 0.15.0.

I had solved this issue by adding --user=ipfs to podman run arguments. This problem may be solved for everybody by adding USER ipfs to Dockerfile.

@akhilman
Copy link
Author

Reopening this issue because it affects not only my.

@akhilman akhilman reopened this Sep 14, 2022
@jdannenberg
Copy link

jdannenberg commented Sep 14, 2022

Thank your for your answers! Really appreciated. I get why the fix might (not suggesting that it does not, but I do not understand the exact why) work, since I found this before I found this issue https://github.com/ipfs/kubo/blob/master/bin/container_daemon#L7-L13 and I have been of the opinion that it should work.

@akhilman
Copy link
Author

Thank your for your answers! Really appreciated. I get why the fix might (not suggesting that it does not, but I do not understand the exact why) work, since I found this before I found this issue https://github.com/ipfs/kubo/blob/master/bin/container_daemon#L7-L13 and I have been of the opinion that it should work.

OK. This is strange. Now I also don't understand why there are files owned by root.

@jdannenberg
Copy link

And btw.: Not setting user in Dockerfile, but changing it in the entrypoint script seems unusual. However, one benefit is that the cli process is still able to read files not belonging to the ipfs user. So, if I use user: ipfs my ipfs add does not work anymore.

@arseniiarsenii
Copy link

This also affected me. Last week node created a file owned by root, which was repo.lock. Today it was /datastore/000380.ldb. For now I set container to privileged mode in my docker-compose which is far from optimal.

As of now my docker-compose.yml entry fro this service is as follows

  ipfsnode:
    restart: always
    image: ipfs/go-ipfs:v0.14.0
    privileged: true
    volumes:
      - "/hdd/ipfs2/ipfs_staging:/export"
      - "/hdd/ipfs2/ipfs_data:/data/ipfs"
    healthcheck:
      interval: 5s
    ports:
      - "0.0.0.0:5021:5001"
      - "0.0.0.0:8080:8080"

Hope to see a solution soon, thanks

@r3c4ll
Copy link

r3c4ll commented May 25, 2023

Hi, I'm having still the same issue with the current version. I'm running the Kubo container through docker compose with a non-root user but it still create a ../home/ipfs directory and files owned by root.

Any advise for these folders to be created with the user I'm executing docker compose as the owner?

@dennis-tra
Copy link
Collaborator

I'm just adding my +1 here. I'm running a Kubo container on AWS ECS with Fargate and occasionally (not always) get the error:

Error: lock /data/ipfs/repo.lock: permission denied

The logs:

2023-06-19T00:22:01.101Z	Changing user to ipfs
2023-06-19T00:22:01.215Z	ipfs version 0.18.1
2023-06-19T00:22:01.282Z	generating ED25519 keypair...done
2023-06-19T00:22:01.286Z	peer identity: 12D3KooWE4jx5JD2C9CPzjHBNnBERUX85zEWvrE9Sw6CCUhASE7w
2023-06-19T00:22:01.286Z	initializing IPFS node at /data/ipfs
2023-06-19T00:22:01.464Z	to get started, enter:
2023-06-19T00:22:01.464Z	ipfs cat /ipfs/QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc/readme
2023-06-19T00:22:01.589Z	Error: lock /data/ipfs/repo.lock: permission denied

This happens in ~5-10% of the Kubo boots. Setting the privileged flag to true is not possible on Fargate and I wouldn't want to do that.

@dennis-tra
Copy link
Collaborator

For everyone who's watching this. We believe to have fixed the issue.

The problem seems to have been that the health check command ran before the daemon process started. Commonly, the health check is run as root and initializes the repository. The daemon, which is run as a non-privileged user by default, doesn't have permission to interact with that.

This was fixed in #9992 by passing a flag to the health check command that prevents it from initializing the repository.

@koxon
Copy link

koxon commented Jan 1, 2024

I still have the issue.

I've been running ipfs in a container on AWS Fargate.

ipfs version 0.25.0
Golang version: go1.21.5
Kubo version: 0.25.0-413a52d

Error: lock /data/ipfs/repo.lock: permission denied

I'm also trying to delete the repo.lock when the container starts but I struggle to extend the ENTRYPOINT and CMD to run a custom script that deletes this file.

I can't login into the container to delete the file by hand.
I want to automate the deletion so it works when the container restart after being killed by Fargate.

Any idea how I can automatically delete this file as the root user?

@koxon
Copy link

koxon commented Jan 2, 2024

I've made a fix for this repo.lock issue. See #2457
The fix uses a custom Dockerfile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants