Issue: Unable to Mount Azure Containers with Blobfuse2 - "too many open files" Error

[T0o-m]

Hi,

I am experiencing an issue where I need to mount 193 Azure containers using Blobfuse2, but I am unable to mount more than 118 containers. When attempting to mount additional containers, I receive the following error:

Error : failed to create watcher : too many open files

To resolve this, I have attempted the following changes to system configuration files:

1. Edited /etc/security/limits.conf by adding:

   *         hard    nofile      500000
   *         soft    nofile      500000
   root      hard    nofile      500000
   root      soft    nofile      500000
   

2. Edited /etc/pam.d/common-session by adding:

    session required pam_limits.so
   

3. Edited /etc/sysctl.conf by adding:

   fs.file-max = 2097152
   

Despite these changes, the problem persists.

Could you please provide guidance on how to resolve this issue or suggest any additional configurations that might help?

Thank you for your assistance.

[souravgupta-msft]

Hi @T0o-m. Can you please share the config file and mount command you are using.
Error : failed to create watcher : too many open files
From this error, it looks like health-monitor is enabled. If yes, please disable it and retry mounting.

[T0o-m]

Hi @souravgupta-msft ,

Thank you for your response. Here is the configuration file and the mount command I am using:

Config:

health_monitor:
  enable-monitoring: false

logging:
  type: syslog
  level: log_debug

components:
  - libfuse
  - file_cache
  - attr_cache
  - azstorage

libfuse:
  attribute-expiration-sec: 120
  entry-expiration-sec: 120
  negative-entry-expiration-sec: 240

file_cache:
  path: path
  timeout-sec: 120
  max-size-mb: 4096

attr_cache:
  timeout-sec: 7200

azstorage:
  type: adls
  account-name: account-name
  account-key: account-key
  endpoint: endpoint
  mode: key
  container: container-name

Mount Command:

blobfuse2 mount /mount/path/ --config-file=/config/path/config.yml --tmp-path=/path -o allow_other

Despite having enable-monitoring set to false in the health_monitor section of the config, I am still encountering the "failed to create watcher: too many open files" error.

For additional context, I am running this on a virtual machine hosted on Azure, with the following specifications:

• OS: Debian 11
• vCPUs: 4
• Memory: 16 GiB

Thank you for your assistance.

[souravgupta-msft]

blobfuse2 mount /mount/path/ --config-file=/config/path/config.yml --tmp-path=/path -o allow_other

This command will mount only a single container container-name as per the config. Are you running the mount command for all your containers one by one?

I would also like to mention that each blobfuse2 mount runs as a separate process. Your given VM spec is not sufficient for mounting 193 containers.