mount failed on ubuntu 22.04(aks 1.25.2) #1015

cvvz · 2023-01-04T07:51:45Z

Which version of blobfuse was used?

blobfuse2 version 2.0.1

Which OS distribution and version are you using?

Ubuntu 22.04

If relevant, please share your mount command.

blobfuse2 mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a45a1958a856c3aa56d78a5b485feb218d61673d0ed773a2367adaa6bf90fc4c/globalmount -o allow_other --file-cache-timeout-in-seconds=120 --use-attr-cache=true --cancel-list-on-mount-seconds=10 -o attr_timeout=120 -o entry_timeout=120 -o negative_timeout=120 --log-level=LOG_WARNING --cache-size-mb=1000 --container-name=pvc-45e0990f-bd6b-4cbe-baed-2e927981c7db --pre-mount-validate=true --use-https=true --empty-dir-check=false --tmp-path=/mnt/mc_weizhichen_weizhichen125_eastus#fuse74833b66893742a894d#pvc-45e0990f-bd6b-4cbe-baed-2e927981c7db##default#

What was the issue encountered?

Can't write to the file in the mountpoint, error:

root@aks-nodepool1-16062421-vmss000001:/# ls /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a45a1958a856c3aa56d78a5b485feb218d61673d0ed773a2367adaa6bf90fc4c/globalmount
test
root@aks-nodepool1-16062421-vmss000001:/# echo "123" >> /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a45a1958a856c3aa56d78a5b485feb218d61673d0ed773a2367adaa6bf90fc4c/globalmount/test
bash: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a45a1958a856c3aa56d78a5b485feb218d61673d0ed773a2367adaa6bf90fc4c/globalmount/test: Invalid argument

Have you found a mitigation/solution?

Mount using the same mount options can succeed on Ubuntu 18.04(aks < 1.25)

Please share logs if available.

Jan 4 07:03:49 aks-nodepool1-16062421-vmss000001 blobfuse2[47882]: LOG_ERR [libfuse_handler.go (633)]: Libfuse::libfuse_open : Flag (8401) not supported to open test when write back cache is on. Pass --disable-writeback-cache=true or --ignore-open-flags=true via CLI

The text was updated successfully, but these errors were encountered:

souravgupta-msft · 2023-01-04T09:29:32Z

Hi @cvvz,
As per the log you have to add --disable-writeback-cache=true or --ignore-open-flags=true in your CLI. This is expected behaviour. Please try it out and let us know if it fixes the issue.

andyzhangx · 2023-01-05T02:57:35Z

Hi @cvvz, As per the log you have to add --disable-writeback-cache=true or --ignore-open-flags=true in your CLI. This is expected behaviour. Please try it out and let us know if it fixes the issue.

@souravgupta-msft could you make one of these two flags as default mount option in v2? if customer is upgrading from aks 1.24 (with blobfuse v1) to 1.25(only blobfuse v2 is supported on Ubuntu 22.04), this would break.

As an agreement with blobfuse team, upgrade from v1 to v2 should be compatible, and customer does not need to add or remove any mount option to make it work.

andyzhangx · 2023-01-05T02:58:49Z

cc @vibhansa-msft this issue is blocking customer upgrading from AKS Ubuntu 18.04 node to 22.04 node, that's a critical bug since it makes blobfuse2 mount option not compatible with v1.

andyzhangx · 2023-01-05T03:03:58Z

and since AKS 1.25 is using ubuntu 22.04, and only v2 is supported, I am worried about the compatibility if customer upgraded from AKS 1.24 (with Ubuntu 18.04 blobfuse v1) to 1.25, is it possible to provide blobfuse v1 support on 22.04?

vibhansa-msft · 2023-01-05T03:59:37Z

@andyzhangx : I understand your concern here. Let me discuss this internally with team and get back to you.

andyzhangx · 2023-01-05T08:34:13Z

I have tried, set --ignore-open-flags=true won't work, only append --disable-writeback-cache=true mount option works.
shall we make this option as default in v2?

souravgupta-msft · 2023-01-05T09:01:11Z

@andyzhangx, I tried the operation locally with --ignore-open-flags=true CLI option. It is working for me.
Can you share the logs when you have --ignore-open-flags=true?

andyzhangx · 2023-01-05T09:24:07Z

--ignore-open-flags=true

@souravgupta-msft with only --ignore-open-flags=true, it failed with following errors:

e. Current Version=2.0.0-preview.3, Latest Version=2.0.1
Jan  5 09:21:29 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_INFO [root.go (145)]: *** blobfuse2: A new version (2.0.1) is available. Consider upgrading to latest version for bug-fixes & new features. ***
Jan  5 09:21:29 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_WARNING [mount.go (359)]: unsupported v1 CLI parameter: pre-mount-validate is always true in blobfuse2.
Jan  5 09:21:29 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_CRIT [mount.go (390)]: Starting Blobfuse2 Mount : 2.0.0-preview.3 on (Ubuntu 22.04.1 LTS)
Jan  5 09:21:29 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_CRIT [mount.go (391)]: Logging level set to : LOG_WARNING
Jan  5 09:21:30 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_ERR [libfuse_handler.go (628)]: Libfuse::libfuse_open : Flag (8401) not supported to open outfile when write back cache is on. Pass --disable-writeback-cache or --ignore-open-flag via CLI
Jan  5 09:21:30 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_ERR [libfuse_handler.go (628)]: Libfuse::libfuse_open : Flag (8401) not supported to open outfile when write back cache is on. Pass --disable-writeback-cache or --ignore-open-flag via CLI
Jan  5 09:21:44 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_ERR [libfuse_handler.go (628)]: Libfuse::libfuse_open : Flag (8401) not supported to open outfile when write back cache is on. Pass --disable-writeback-cache or --ignore-open-flag via CLI
Jan  5 09:22:11 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_ERR [libfuse_handler.go (628)]: Libfuse::libfuse_open : Flag (8401) not supported to open outfile when write back cache is on. Pass --disable-writeback-cache or --ignore-open-flag via CLI
Jan  5 09:22:29 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_ERR [cache_policy.go (96)]: cachePolicy::getCacheUsage : error running du [exec: "du": executable file not found in $PATH]
Jan  5 09:22:53 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_ERR [libfuse_handler.go (628)]: Libfuse::libfuse_open : Flag (8401) not supported to open outfile when write back cache is on. Pass --disable-writeback-cache or --ignore-open-flag via CLI

souravgupta-msft · 2023-01-05T09:44:35Z

Jan 5 09:21:29 aks-agentpool-38143190-vmss000000 blobfuse2[2388861]: LOG_CRIT [mount.go (390)]: Starting Blobfuse2 Mount : 2.0.0-preview.3 on (Ubuntu 22.04.1 LTS)

You are using older version of blobfuse2. Please update to blobfuse2-2.0.1 and retry.

andyzhangx · 2023-01-05T10:10:44Z

thanks, after upgrading to 2.0.1, even without -ignore-open-flags=true, mount succeeds now. Pls close this issue, thanks.
I will publish a new blob csi driver version with 2.0.1 version.

Jan  5 10:06:39 aks-agentpool-38143190-vmss000000 blobfuse2[2500068]: LOG_CRIT [mount.go (364)]: Starting Blobfuse2 Mount : 2.0.1 on [Ubuntu 22.04.1 LTS]
Jan  5 10:06:39 aks-agentpool-38143190-vmss000000 blobfuse2[2500068]: LOG_CRIT [mount.go (365)]: Logging level set to : LOG_WARNING
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500495]: LOG_WARNING [mount.go (344)]: mount: unsupported v1 CLI parameter: pre-mount-validate is always true in blobfuse2.
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500495]: LOG_CRIT [mount.go (364)]: Starting Blobfuse2 Mount : 2.0.1 on [Ubuntu 22.04.1 LTS]
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500495]: LOG_CRIT [mount.go (365)]: Logging level set to : LOG_WARNING
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500495]: LOG_ERR [file_cache.go (239)]: FileCache: config error [tmp-path does not exist. attempting to create tmp-path.]
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500505]: LOG_WARNING [mount.go (344)]: mount: unsupported v1 CLI parameter: pre-mount-validate is always true in blobfuse2.
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500505]: LOG_CRIT [mount.go (364)]: Starting Blobfuse2 Mount : 2.0.1 on [Ubuntu 22.04.1 LTS]
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500505]: LOG_CRIT [mount.go (365)]: Logging level set to : LOG_WARNING

souravgupta-msft · 2023-01-06T05:27:27Z

@andyzhangx, can you please check if the following append operation (originally reported) also works post successful mount after updating to 2.0.1.
echo "123" >> /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a45a1958a856c3aa56d78a5b485feb218d61673d0ed773a2367adaa6bf90fc4c/globalmount/test

andyzhangx · 2023-01-06T10:02:26Z

@andyzhangx, can you please check if the following append operation (originally reported) also works post successful mount after updating to 2.0.1. echo "123" >> /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a45a1958a856c3aa56d78a5b485feb218d61673d0ed773a2367adaa6bf90fc4c/globalmount/test

@souravgupta-msft it's working now, thanks.

# cat test
123

andyzhangx · 2023-01-11T06:02:37Z

thanks, after upgrading to 2.0.1, even without -ignore-open-flags=true, mount succeeds now. Pls close this issue, thanks. I will publish a new blob csi driver version with 2.0.1 version.

Jan  5 10:06:39 aks-agentpool-38143190-vmss000000 blobfuse2[2500068]: LOG_CRIT [mount.go (364)]: Starting Blobfuse2 Mount : 2.0.1 on [Ubuntu 22.04.1 LTS]
Jan  5 10:06:39 aks-agentpool-38143190-vmss000000 blobfuse2[2500068]: LOG_CRIT [mount.go (365)]: Logging level set to : LOG_WARNING
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500495]: LOG_WARNING [mount.go (344)]: mount: unsupported v1 CLI parameter: pre-mount-validate is always true in blobfuse2.
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500495]: LOG_CRIT [mount.go (364)]: Starting Blobfuse2 Mount : 2.0.1 on [Ubuntu 22.04.1 LTS]
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500495]: LOG_CRIT [mount.go (365)]: Logging level set to : LOG_WARNING
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500495]: LOG_ERR [file_cache.go (239)]: FileCache: config error [tmp-path does not exist. attempting to create tmp-path.]
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500505]: LOG_WARNING [mount.go (344)]: mount: unsupported v1 CLI parameter: pre-mount-validate is always true in blobfuse2.
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500505]: LOG_CRIT [mount.go (364)]: Starting Blobfuse2 Mount : 2.0.1 on [Ubuntu 22.04.1 LTS]
Jan  5 10:06:47 aks-agentpool-38143190-vmss000000 blobfuse2[2500505]: LOG_CRIT [mount.go (365)]: Logging level set to : LOG_WARNING

@souravgupta-msft sorry, I may get it wrong last time. It's still failing on Ubuntu 22.04 by default, could you set --ignore-open-flags=true as default in next blobfuse2 release, this is a blocking issue if user upgrades to Ubuntu 22.04, user has to set --ignore-open-flags=true mount options manually to workaround this issue.

related error logs

Jan 11 05:49:48 aks-agentpool-75094174-vmss000000 blobfuse2[277411]: LOG_WARNING [mount.go (344)]: mount: unsupported v1 CLI parameter: pre-mount-validate is always true in blobfuse2.
Jan 11 05:49:48 aks-agentpool-75094174-vmss000000 blobfuse2[277411]: LOG_CRIT [mount.go (364)]: Starting Blobfuse2 Mount : 2.0.1 on [Ubuntu 22.04.1 LTS]
Jan 11 05:49:48 aks-agentpool-75094174-vmss000000 blobfuse2[277411]: LOG_CRIT [mount.go (365)]: Logging level set to : LOG_WARNING
Jan 11 05:49:48 aks-agentpool-75094174-vmss000000 blobfuse2[277411]: LOG_ERR [file_cache.go (239)]: FileCache: config error [tmp-path does not exist. attempting to create tmp-path.]
Jan 11 05:49:48 aks-agentpool-75094174-vmss000000 blobfuse2[277417]: LOG_WARNING [mount.go (344)]: mount: unsupported v1 CLI parameter: pre-mount-validate is always true in blobfuse2.
Jan 11 05:49:48 aks-agentpool-75094174-vmss000000 blobfuse2[277417]: LOG_CRIT [mount.go (364)]: Starting Blobfuse2 Mount : 2.0.1 on [Ubuntu 22.04.1 LTS]
Jan 11 05:49:48 aks-agentpool-75094174-vmss000000 blobfuse2[277417]: LOG_CRIT [mount.go (365)]: Logging level set to : LOG_WARNING
Jan 11 05:49:54 aks-agentpool-75094174-vmss000000 blobfuse2[277417]: LOG_ERR [libfuse_handler.go (633)]: Libfuse::libfuse_open : Flag (8401) not supported to open outfile when write back cache is on. Pass --disable-writeback-cache=true or --ignore-open-flags=true via CLI
Jan 11 05:49:56 aks-agentpool-75094174-vmss000000 blobfuse2[277417]: LOG_ERR [libfuse_handler.go (633)]: Libfuse::libfuse_open : Flag (8401) not supported to open outfile when write back cache is on. Pass --disable-writeback-cache=true or --ignore-open-flags=true via CLI
Jan 11 05:50:08 aks-agentpool-75094174-vmss000000 blobfuse2[277417]: LOG_ERR [cache_policy.go (96)]: cachePolicy::getCacheUsage : error running du [exec: "du": executable file not found in $PATH]
Jan 11 05:50:08 aks-agentpool-75094174-vmss000000 blobfuse2[277417]: LOG_ERR [cache_policy.go (96)]: cachePolicy::getCacheUsage : error running du [exec: "du": executable file not found in $PATH]
Jan 11 05:50:11 aks-agentpool-75094174-vmss000000 blobfuse2[277417]: LOG_ERR [libfuse_handler.go (633)]: Libfuse::libfuse_open : Flag (8401) not supported to open outfile when write back cache is on. Pass --disable-writeback-cache=true or --ignore-open-flags=true via CLI

vibhansa-msft · 2023-01-11T06:40:20Z

We will change "ignore-open-flag" to be default true.

vibhansa-msft added the V2 label Jan 4, 2023

vibhansa-msft added this to the V2-2.0.2 milestone Jan 4, 2023

souravgupta-msft closed this as completed Jan 4, 2023

souravgupta-msft reopened this Jan 5, 2023

souravgupta-msft self-assigned this Jan 5, 2023

souravgupta-msft closed this as completed Jan 5, 2023

vibhansa-msft reopened this Jan 11, 2023

vibhansa-msft added the blocker label Jan 11, 2023

souravgupta-msft mentioned this issue Jan 11, 2023

Set ignore-open-flags to true by default #1029

Merged

souravgupta-msft closed this as completed in #1029 Jan 21, 2023

cvvz mentioned this issue Feb 23, 2023

fix: mount failed on ubuntu 22.04(aks 1.25.2) kubernetes-sigs/blob-csi-driver#827

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mount failed on ubuntu 22.04(aks 1.25.2) #1015

mount failed on ubuntu 22.04(aks 1.25.2) #1015

cvvz commented Jan 4, 2023 •

edited

Loading

souravgupta-msft commented Jan 4, 2023

andyzhangx commented Jan 5, 2023

andyzhangx commented Jan 5, 2023 •

edited

Loading

andyzhangx commented Jan 5, 2023

vibhansa-msft commented Jan 5, 2023

andyzhangx commented Jan 5, 2023

souravgupta-msft commented Jan 5, 2023

andyzhangx commented Jan 5, 2023

souravgupta-msft commented Jan 5, 2023

andyzhangx commented Jan 5, 2023

souravgupta-msft commented Jan 6, 2023

andyzhangx commented Jan 6, 2023

andyzhangx commented Jan 11, 2023

vibhansa-msft commented Jan 11, 2023

mount failed on ubuntu 22.04(aks 1.25.2) #1015

mount failed on ubuntu 22.04(aks 1.25.2) #1015

Comments

cvvz commented Jan 4, 2023 • edited Loading

Which version of blobfuse was used?

Which OS distribution and version are you using?

If relevant, please share your mount command.

What was the issue encountered?

Have you found a mitigation/solution?

Please share logs if available.

souravgupta-msft commented Jan 4, 2023

andyzhangx commented Jan 5, 2023

andyzhangx commented Jan 5, 2023 • edited Loading

andyzhangx commented Jan 5, 2023

vibhansa-msft commented Jan 5, 2023

andyzhangx commented Jan 5, 2023

souravgupta-msft commented Jan 5, 2023

andyzhangx commented Jan 5, 2023

souravgupta-msft commented Jan 5, 2023

andyzhangx commented Jan 5, 2023

souravgupta-msft commented Jan 6, 2023

andyzhangx commented Jan 6, 2023

andyzhangx commented Jan 11, 2023

vibhansa-msft commented Jan 11, 2023

cvvz commented Jan 4, 2023 •

edited

Loading

andyzhangx commented Jan 5, 2023 •

edited

Loading