panic creating temp dir for logs when running blobfuse2 non-interactively as a user without a $HOME

[iMHLv2]

Which version of blobfuse was used?

# apt-cache policy blobfuse2
blobfuse2:
  Installed: 2.0.1
  Candidate: 2.0.1
  Version table:
 *** 2.0.1 500
        500 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 Packages
        100 /var/lib/dpkg/status

Which OS distribution and version are you using?

# cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

If relevant, please share your mount command.

Not relevant (yet). Ultimately we want to mount things, but just getting the command to run with --help in this scenario would suffice.

What was the issue encountered?

For some use cases, we'd like to use blobfuse2 non-interactively - for example from a systemd service running under a service account. In some cases, the service account may not have a home directory. Steps to reproduce:

# groupadd test
# useradd -g test --no-create-home -s /bin/false --home-dir /does-not-exist test

The next command merely simulates the problem, but in a slightly different way. In the systemd case, $HOME will be set to /does-not-exist. In the example case, it will be set to /root since I'm invoking via sudo while logged in as root. Either way, the folder either does not exist or it exists and isn't writable by the test user.

# sudo -u test blobfuse2 --help
panic: Unable to create blobfuse2 temp directory for logs

goroutine 1 [running]:
github.com/Azure/azure-storage-fuse/v2/common/log.init.0()
	/mnt/vss/_work/1/s/azure-storage-fuse/common/log/logger.go:233 +0x1ab

That all makes sense so far. The confusing part occurs when trying to redirect the directory. The REAMDE.md mentions:

--default-working-dir: The default working directory to store log files and other blobfuse2 related information.

That didn't work:

# sudo -u test blobfuse2 --default-working-dir /tmp
panic: Unable to create blobfuse2 temp directory for logs

goroutine 1 [running]:
github.com/Azure/azure-storage-fuse/v2/common/log.init.0()
	/mnt/vss/_work/1/s/azure-storage-fuse/common/log/logger.go:233 +0x1ab

The --default-working-dir also doesn't seem to be a recognized option when run as root:

# sudo blobfuse2 --default-working-dir /tmp
unknown flag: --default-working-dir

There is a similar looking option called --log-file-path which looks like this:

# sudo blobfuse2 mount --help | grep log-file
      --log-file-path string         Configures the path for log files. Default is $HOME/.blobfuse2/blobfuse2.log (default "$HOME/.blobfuse2/blobfuse2.log")

However that doesn't work either:

# sudo -u test blobfuse2 --log-file-path /tmp/log.txt
panic: Unable to create blobfuse2 temp directory for logs

goroutine 1 [running]:
github.com/Azure/azure-storage-fuse/v2/common/log.init.0()
	/mnt/vss/_work/1/s/azure-storage-fuse/common/log/logger.go:233 +0x1ab

The tool seems to still be using the default of $HOME/.blobfuse2 which is /root/.blobfuse2 in the example.

# strace -f sudo -u test blobfuse2 --log-file-path /tmp/log.txt
...
[pid  1089] newfstatat(AT_FDCWD, "/root/.blobfuse2",  <unfinished ...>
[pid  1090] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid  1089] <... newfstatat resumed> 0xc0000a5218, 0) = -1 EACCES (Permission denied)
[pid  1089] newfstatat(AT_FDCWD, "/root", {st_mode=S_IFDIR|0700, st_size=4096, ...}, 0) = 0
[pid  1090] <... nanosleep resumed> NULL) = 0
[pid  1089] mkdirat(AT_FDCWD, "/root/.blobfuse2", 0777 <unfinished ...>
[pid  1090] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid  1089] <... mkdirat resumed> )     = -1 EACCES (Permission denied)
[pid  1089] newfstatat(AT_FDCWD, "/root/.blobfuse2", 0xc0000a53b8, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied)
...

Have you found a mitigation/solution?

We can obviously create a home directory for the service account with the proper permissions, but that wasn't necessary for blobfuse v1. If possible, we'd like to avoid creating the directory for this tool's use exclusively.

Please share logs if available.

See above, thank you!

[vibhansa-msft]

$HOME/.blobfuse2 is a directory that blobfuse2 creates to store of the house keeping data. Default log path in case of base logging is also the same. Base issue here is user is not having any home directory set which is leading to failure in creation of. blobfuse2 directory under home. For now, having a home directory is the workaround, till we fix this issue from our end.
Amy particular reason you have to not create home directory for the given user.

[vibhansa-msft]

We looked back at the codebase and there are other house-keeping things put in $HOME/.blobfuse2 directory. This also includes a pid based lock file and few initial logs. As this thing is initialized even before the cli parsing begins, default-working-dir that you are trying to set is of no-use. If the user mounting blobfuse2 has access to the $HOME path blobfuse2 will be able to create the directory and proceed. In case when $HOME is not set, $HOME returns back "/" which means user shall have permission to create a directory /.blobfuse2 in your case to run successfully.
I see the easier way out here is to have a home directory for the user who is going to mount blobfuse2. Changing the code-flow in this regard will be a breaking change from our side (as existing behavior will change) which we do not intend to do unless there is a reasonable use-case affecting multiple users.

[iMHLv2]

@vibhansa-msft Thanks for the reply, and I agree with your diagnosis (non-existent or non-writable $HOME). To explain a little more about our use case, we have a set of REST APIs for users to request mounting or unmounting blobs. The client is a web browser or an automated platform such as a SOAR or EDR. The APIs communicate to a systemd service (via database) which performs the operations by invoking the blobfuse tool. For security "defense-in-depth" the APIs and systemd service run as a non-root, neutered user account (i.e., no password set, no shell, no home directory).

As previously mentioned, this interaction worked fine with blobfuse v1 and it is also compatible with aws s3 and GCP, etc. In several years, we haven't run into any other actions/tools used or invoked by the APIs or systemd service that required $HOME to exist for this service account, so there has never been a reason to create the directory.

Here are a few arguments (the friendly kind!) to consider regarding making a change or not:

• It may indeed be a breaking change from blobfuse2-2.0.1 to a future version of blobfuse2, but it's already a behavior change for anyone upgrading from v1. It looks like you guys have taken a lot of care to make sure command-line options from v1 are either compatible with v2 or silently ignored, but this behavior change produces a panic.

• It sounds like the current version always writes to $HOME and if a user specifies --default-working-dir, then temp files end up spread across two folders. Or are the files created under $HOME moved into a single location after parsing the command-line options? In either case, IMO it would be cleaner to parse the command-line options first and use the user-specified directory from the start.

At the very least, I'd recommend updating the documentation to say that $HOME must exist and be writable for blobfuse v2 to work. It might also be good to modify the --help output for --default-working-dir and --log-file-path to mention that those are less defaults (i.e., "Default is $HOME/.blobfuse2/blobfuse2.log") and more static and unconditional.

Thanks for considering the change!

[vibhansa-msft]

Thanks for your suggestions and inputs. I will look into it and see if we can fix it up in next release. If change has some risks, we will go ahead and update the document saying $HOME is must with v2.

[vibhansa-msft]

I have a potential fix for the problem. Will it be possible for you to check-out the branch and build locally?

[brosahay]

I have a potential fix for the problem. Will it be possible for you to check-out the branch and build locally?

I can help

[vibhansa-msft]

@brosahay : you can try on this branch: https://github.com/Azure/azure-storage-fuse/tree/vibhansa/v2/default_work_dir_fix

[dannyfritz3]

@vibhansa-msft Sorry for the late response. I was able to build your branch locally and it worked using the --default-working-dir flag now with the $HOME directory not present.

[vibhansa-msft]

@dannyfritz3 : Great so my fix works well it seems for your use case. Will try to get this change in for the next release.

[dannyfritz3]

@vibhansa-msft Thank you for the swift assistance!

[scottporter]

We have been use blobfuse to actually bring in a home directory that we were storing on a blob storage container. This requirement of blobfuse2 to store files in the home directory obviously breaks what we were doing, keeping us from upgrading. Looking forward to examining the fix.

[vibhansa-msft]

@scottporter : there is still time before this gets officially released. Change is merged into main branch. Can you checkout main branch build locally and try this change out.

[scottporter]

@vibhansa-msft I tried building, but got this error:

build github.com/Azure/azure-storage-fuse/v2: cannot load io/fs: malformed module path "io/fs": missing dot in first path element

[scottporter]

Is there a binary with the fix I could check? I wasn't able to build on our infrastructure, but still willing to test the fix if there is an easy way to do that now.

[vibhansa-msft]

There is no need to build now, latest 2.0.2 release has the fix. Kindly upgrade and try again.